Task scheduling method and system, electronic equipment and storage medium

The task scheduling method optimizes resource allocation in machine learning training by analyzing task demands and matching them with optimal compute nodes, addressing inefficiencies in distributed training and improving execution efficiency.

CN120315879APending Publication Date: 2025-07-15CHINA TELECOM CORP LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202510406401.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-04-01
Publication Date
2025-07-15

AI Technical Summary

Technical Problem

In the prior art, in machine learning task training, the resources of a single computing device are limited, resulting in too long training time, and the resource allocation flexibility of distributed training mode is low, resource utilization and task execution efficiency are low.

Method used

By analyzing the to-process tasks, the computing resources, data storage and network bandwidth requirements information are determined, the resource registration library is used for matching and weighted calculations, the optimal computing node is selected for task allocation, and dynamic scheduling and resource adjustment are performed.

Benefits of technology

It improves resource utilization and task execution efficiency, realizes unified management of multiple clusters and multi-resource groups, and optimizes resource allocation and task scheduling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120315879A_ABST
    Figure CN120315879A_ABST
Patent Text Reader

Abstract

The invention discloses a task scheduling method and system, electronic equipment and a storage medium, and the method comprises the steps: analyzing an obtained to-be-processed task, and determining task resource demand information corresponding to the to-be-processed task; performing resource matching with a resource registration library according to the analyzed task resource demand information to obtain a matching score corresponding to the resource; performing weighting operation according to a preset weight and the matching scores to obtain final matching scores of different computing resources; sorting the resources of the resource registration library according to the final matching score, determining a target computing node, and allocating the to-be-processed task to the target computing node for processing; a to-be-processed task is analyzed and calculated to determine a resource demand required by the task, a proper computing resource is matched in a resource library according to the resource demand, a computing node processing task is determined, and the resource utilization rate is increased; tasks are processed by allocating appropriate resources, and the task execution efficiency is improved. The embodiment of the invention can be widely applied to the technical field of cloud computing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular, to a task scheduling method, system, electronic device, and storage medium. Background Art

[0002] In traditional machine learning task training, a single-machine training mode is usually adopted, where all training plans are executed by a single computing device; as the model complexity and training data scale in training tasks continue to increase, the computing resources of a single computing device are limited and the training time is too long; therefore, the prior art adopts a distributed training mode, where training tasks are split and assigned to multiple computing devices for simultaneous execution; however, the prior art only distributes training tasks to multiple computing devices for calculation, reducing the load of a single computing device, with low flexibility in task assignment and resource configuration, resulting in low resource utilization and low task execution efficiency. Summary of the Invention

[0003] The main objective of the embodiments of the present invention is to provide a task scheduling method, system, electronic device, and storage medium, which can improve resource utilization and task execution efficiency.

[0004] To achieve the above objective, on the one hand, an embodiment of the present invention provides a task scheduling method, which includes:

[0005] Analyze the obtained task to be processed to determine task resource requirement information; wherein, the task resource requirement information includes computing resource requirement information, data storage requirement information, and network bandwidth requirement information;

[0006] Match the task resource requirement information with a preset resource registration library to determine a first matching score set; perform weighted calculation according to a preset weight set and the first matching score set to determine a target matching score;

[0007] Sort the preset resource registration library according to the target matching score to determine a target computing node; allocate the task to be processed to the target computing node.

[0008] In some embodiments, the computing resource requirement information is determined by the following method:

[0009] Parse the task to be processed to determine the task type;

[0010] If the task type is a data processing task, determine a target processing algorithm according to the data processing task, compare it with the target processing algorithm according to a preset mapping relationship to determine the computational complexity, and determine the computing resource requirement information according to the computational complexity;

[0011] If the task type is a model training task, determine the target training model, the number of training samples, and the feature dimension according to the model training task, and determine the total number of model parameters according to the target training model; and determine the computing resource requirement information according to the product of the total number of model parameters, the number of training samples, and the feature dimension;

[0012] If the task type is a real-time interaction task, determine the interaction model and the input data volume according to the real-time interaction task, and determine the computing resource requirement information according to the interaction model and the input data volume.

[0013] In some embodiments, the data storage requirement information is determined by the following method:

[0014] Analyze the task to be processed, determine the total number of bytes of the input data and the total number of bytes of the output data, and determine the task storage requirement information according to the sum of the total number of bytes of the input data and the total number of bytes of the output data;

[0015] Parse the task to be processed, determine the intermediate data generation frequency and the average number of bytes, and determine the temporary storage requirement information according to the product of the intermediate data generation frequency and the average number of bytes;

[0016] Take the sum of the task storage requirement information and the temporary storage requirement information as the data storage requirement information.

[0017] In some embodiments, the network bandwidth requirement information is determined by the following method:

[0018] Analyze the task to be processed, and determine the task type of the task to be processed;

[0019] If the task type is a data processing task, determine the data transmission volume and the transmission frequency according to the data processing task, and determine the network bandwidth requirement information according to the product of the data transmission volume and the transmission frequency;

[0020] If the task type is a model training task, determine the total number of bytes transmitted and the model parameter synchronization frequency according to the model training task, and determine the network bandwidth requirement information according to the product of the total number of bytes transmitted and the model parameter synchronization frequency;

[0021] If the task type is a real-time interaction task, determine the data input rate and the data output rate according to the real-time interaction task, and determine the network bandwidth requirement information according to the sum of the data input rate and the data output rate.

[0022] In some embodiments, matching the task resource requirement information with a preset resource registration library to determine a first matching score set specifically includes:

[0023] Determine idle resource information according to the preset resource registry; wherein, the idle resource information includes idle computing resource information, remaining storage capacity information, and idle bandwidth resource information;

[0024] Perform a difference calculation based on the task resource requirement information and the idle resource information to determine a resource difference set; perform a ratio calculation based on the task resource requirement information and the resource difference set to determine a first matching score set; wherein, the first matching score set includes a computing resource matching score, a data storage matching score, and a network bandwidth matching score.

[0025] In some embodiments, the method further includes:

[0026] Analyze the task to be processed to determine the task priority; and process the task to be processed according to a preset time prediction model to determine the predicted task execution time;

[0027] Calculate according to the task priority and the predicted task execution time to determine resource adjustment information;

[0028] Adjust the resources of the target computing node according to the resource adjustment information, and adjust the order of the task execution queue according to the task priority and the predicted task execution time.

[0029] In some embodiments, the method further includes:

[0030] Record the actual execution time, and compare the actual execution time with the predicted task execution time;

[0031] If the actual execution time is less than or equal to the predicted task execution time, keep the current task execution queue;

[0032] If the actual execution time is greater than the predicted task execution time, perform a difference calculation based on the actual execution time and the predicted task execution time to determine the time difference, and compare the time difference with a preset threshold; if the time difference is greater than or equal to the preset threshold, reduce the task priority of the corresponding task, and adjust the order of the task execution queue according to the reduced task priority; otherwise, keep the current task execution queue.

[0033] To achieve the above object, another aspect of the embodiments of the present invention provides a task scheduling system, including:

[0034] A first module, configured to analyze the task to be processed obtained to determine task resource requirement information; wherein, the task resource requirement information includes computing resource requirement information, data storage requirement information, and network bandwidth requirement information;

[0035] A second module, configured to match the task resource requirement information with a preset resource registration library to determine a first set of matching scores; perform weighted calculation according to a preset weight set and the first set of matching scores to determine a target matching score.

[0036] A third module, configured to sort the preset resource registration library according to the target matching score to determine a target computing node; allocate the to-be-processed task to the target computing node.

[0037] To achieve the above object, on the other hand, an embodiment of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the foregoing method is implemented.

[0038] To achieve the above object, on the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the foregoing method is implemented.

[0039] Implementing the embodiments of the present invention includes the following beneficial effects: The embodiments of the present invention provide a task scheduling method, system, electronic device and storage medium. The solution analyzes the to-be-processed task obtained, determines the computing resource requirement information, data storage requirement information and network bandwidth requirement information corresponding to the to-be-processed task, and uses the above requirement information as task resource requirement information; performs resource matching on the analyzed task resource requirement information and a preset resource registration library to obtain matching scores of different computing resources corresponding to the to-be-processed task, and performs weighted operation according to the preset weight information and the obtained matching scores to obtain the final matching scores of different computing resources; sorts the computing resources in the preset resource registration library according to the final matching scores of the computing resources, determines the target computing resources and constructs computing nodes, and allocates the to-be-processed tasks to the computing nodes for processing; determines the resource requirements of the task by analyzing and calculating the to-be-processed task, matches suitable computing resources in the resource library according to the resource requirements, forms a computing node to process the task, improves resource utilization rate; and improves task execution efficiency by allocating suitable resources to process the task. Description of the Drawings

[0040] Figure 1 is a schematic flowchart of the steps of a task scheduling method provided by an embodiment of the present invention;

[0041] Figure 2 is a schematic flowchart of the steps of determining computing resource requirement information in a task scheduling method provided by an embodiment of the present invention;

[0042] Figure 3It is a schematic flow chart of steps for determining data storage requirement information in a task scheduling method provided by an embodiment of the present invention;

[0043] Figure 4 It is a schematic flow chart of steps for determining network bandwidth requirement information in a task scheduling method provided by an embodiment of the present invention;

[0044] Figure 5 It is a schematic flow chart of steps for determining a first matching score set in a task scheduling method provided by an embodiment of the present invention;

[0045] Figure 6 It is a schematic flow chart of steps for performing dynamic scheduling in a task scheduling method provided by an embodiment of the present invention;

[0046] Figure 7 It is a schematic flow chart of steps for adjusting a task execution queue in a task scheduling method provided by an embodiment of the present invention;

[0047] Figure 8 It is a schematic flow chart of steps for establishing a resource registration library in a specific embodiment provided by an embodiment of the present invention;

[0048] Figure 9 It is a schematic flow chart of steps for updating a resource registration library in a specific embodiment provided by an embodiment of the present invention;

[0049] Figure 10 It is a schematic flow chart of steps for performing task resource allocation in a specific embodiment provided by an embodiment of the present invention;

[0050] Figure 11 It is a schematic flow chart of steps for performing dynamic scheduling in a specific embodiment provided by an embodiment of the present invention;

[0051] Figure 12 It is a structural block diagram of a task scheduling system provided by an embodiment of the present invention;

[0052] Figure 13 It is a schematic hardware structure diagram in an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of explanation and illustration, and no limitation is imposed on the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0055] In the following description, the terms "first", "second", and "third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first", "second", and "third" can be interchanged in a specific order or sequence when permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0056] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0057] Before further elaborating on the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0058] In the related art, machine learning tasks are assigned to a single computing device for computing and processing. As the complexity of the algorithm model adopted by machine learning tasks increases, and the scale of data processed by machine learning tasks grows, the computing resources of a single computing device are limited, and the task processing time is long. The prior art uses some distributed training frameworks to split machine learning tasks into multiple tasks and assign them to multiple computing devices for processing. However, the existing distributed training frameworks lack unified management of multi-clusters and multi-resource groups in computing devices, cannot overall integrate public and private resources, and cannot perform dynamic scheduling according to task requirements and resource performance, resulting in low resource utilization, unreasonable task scheduling, and low task execution efficiency.

[0059] In view of this, embodiments of the present invention provide a task scheduling method, system, electronic device, and storage medium. This solution analyzes the obtained task to be processed, determines the computing resource requirement information, data storage requirement information, and network bandwidth requirement information corresponding to the task to be processed, and uses the above requirement information as task resource requirement information; matches the task resource requirement information obtained through analysis with a preset resource registration library to obtain the matching scores of different computing resources corresponding to the task to be processed, and performs a weighted operation based on the preset weight information and the obtained matching scores to obtain the final matching scores of different computing resources; sorts the computing resources in the preset resource registration library according to the final matching scores of the computing resources, determines the target computing resources, constructs a computing node, and allocates the task to be processed to the computing node for processing; determines the resource requirements of the task by analyzing and calculating the task to be processed, matches suitable computing resources in the resource library according to the resource requirements, forms a computing node to process the task, and improves resource utilization; and improves the task execution efficiency by allocating appropriate resources to process the task; at the same time, uniformly registers and identifies multiple clusters and multiple resource groups including public resources and private resources, distinguishes the types and attributes of different resources, records the usage conditions of different resources, establishes a resource registration library, and realizes the unified management of resources.

[0060] Figure 1 FIG. is an alternative flowchart of a task scheduling method provided by an embodiment of the present invention. Figure 1 The method in may include, but is not limited to, steps S101 to S103.

[0061] Step S101: Analyze the obtained task to be processed and determine the task resource requirement information; wherein, the task resource requirement information includes computing resource requirement information, data storage requirement information, and network bandwidth requirement information.

[0062] Step S102: Match the task resource requirement information with a preset resource registration library to determine a first set of matching scores; perform a weighted calculation based on the preset set of weights and the first set of matching scores to determine the target matching score.

[0063] Step S103: Sort the preset resource registration library according to the target matching score to determine the target computing node; allocate the task to be processed to the target computing node.

[0064] Steps S101 to S103 illustrated in the embodiments of the present application analyze the to-be-processed tasks obtained by the system to determine relevant messages such as the task type and task attribute information of the tasks; calculate the computing resource requirements based on the determined task-related messages to determine the optimal computing resources, data storage resources, network bandwidth resources, etc. for executing the to-be-processed tasks, so as to ensure the efficiency of data reading and writing, data processing, and data transmission of the tasks; then, the system matches according to the resource requirements of the to-be-processed tasks determined by the calculation in the pre-constructed resource registration library, and calculates the matching scores between various resources in the resource registration library and the resource requirements of the tasks to determine whether the various resources in the resource registration library meet the resource requirements of the tasks, and then screen out the optimal resources from them to process the tasks; in this embodiment, considering that different types of tasks have different emphases on different resources, different resource weights are set for different types of tasks; after the system determines the matching scores between various resources in the resource registration library and the task requirements, weighted operations are performed according to the set resource weights and the obtained matching scores to improve the matching degree between resources and tasks and further improve the resource utilization rate; based on the determined optimal resources, the system builds corresponding resource groups or computing nodes and allocates the to-be-processed tasks to the computing nodes for processing.

[0065] Before step S101 of some embodiments, the system needs to uniformly manage existing multi-clusters and multi-resource groups to build a resource registration library. Among them, the multi-clusters and multi-resource groups include public and private resources, and include different types of computing nodes, storage resources, and network bandwidth resources, such as CPU nodes, GPU (Graphics Processing Unit) nodes, memory resources, storage resources, etc. The system assigns a unique digital identifier to existing clusters and resource groups, and records the types of each resource therein, such as CPU, GPU, memory, etc. It assigns corresponding cluster identifiers to the resources in each cluster and resource group, records the detailed parameters of the resources, sets the access permissions of the resources, and initializes the status flag bits of all resources to "available". The system can uniformly manage all resources according to the above settings and records, classify the existing resources according to resource types and performance based on the obtained detailed resource parameters, and assign corresponding resource identifiers. The system constructs a unified index based on the obtained identifiers to establish a resource registration library to achieve overall allocation of resources. When a new resource is added to the resource registration library, the system records the resource type, the cluster identifier it belongs to, the detailed resource parameters, and the resource access permissions of this resource, and updates the recorded information to the index of the resource registration library. The system also deploys a resource monitoring agent program in the resource registration library. This program collects the status information of resources according to the set period, including the idle rate, usage rate, and temperature of computing resources, the usage rate of memory resources, the read / write busy degree of storage resources, etc., updates the collected resource status information to the resource records of the resource registration library in real time, and modifies the status flag bits of each resource in real time according to this resource record to improve the real-time performance of subsequent resource allocation.

[0066] Please refer to Figure 2 , in some embodiments, the computing resource requirement information in step S101 can be determined through steps S201 to S204:

[0067] Step S201, parse the task to be processed to determine the task type;

[0068] Step S202, if the task type is a data processing task, determine the target processing algorithm according to the data processing task, compare it with the target processing algorithm according to the preset mapping relationship to determine the computational complexity, and determine the computing resource requirement information according to the computational complexity;

[0069] Step S203, if the task type is a model training task, determine the target training model, the number of training samples, and the feature dimension according to the model training task, and determine the total number of model parameters according to the target training model; and determine the computing resource requirement information according to the product of the total number of model parameters, the number of training samples, and the feature dimension;

[0070] Step S204, if the task type is a real-time interaction task, determine the interaction model and the amount of input data according to the real-time interaction task, and determine the computing resource requirement information according to the interaction model and the amount of input data.

[0071] In step S201 of some embodiments, after receiving the task to be processed, the system determines the task type corresponding to the task by parsing the task to be processed, and performs targeted resource allocation based on the task type; exemplarily, for a data processing task, the system allocates more computing resources to improve the efficiency of data processing; for a real-time interaction task, such as a real-time prediction task, the system allocates more network bandwidth resources to ensure the efficiency of data transmission, thereby ensuring the real-time performance of task execution.

[0072] In step S202 of some embodiments, after the system determines that the task type of the task to be processed is a data processing task by parsing, determine the corresponding data processing algorithm according to the task, compare the data processing algorithm with the preset algorithm complexity mapping table in the system, and determine the computing complexity corresponding to the data processing algorithm; determine the corresponding computing complexity level according to the determined computing complexity, such as low, medium, and high levels, and then determine the corresponding computing resource requirement information.

[0073] In step S203 of some embodiments, if the system determines that the task type of the task to be processed is a model training task by parsing, such as a neural network training task; the system determines the training model used by the task according to the task to be processed, analyzes the model structure of the training model, and calculates the total number of parameters of the training model; exemplarily, for the fully connected layer of the training model, calculate the number of parameters of the fully connected layer through the following formula:

[0074]

[0075] where L is the number of layers of the fully connected layer, and n l is the number of neurons in the l-th layer;

[0076] For the convolutional layer of the training model, calculate the number of parameters of the convolutional layer by calculating parameters such as the convolutional kernel size and the number of input and output channels of the convolutional layer; at the same time, the system also calculates the number of sample data of the training data used in the model training task and the feature dimension of the processed data, and comprehensively determines the computing resource requirement information of the model training task based on the above determined parameters; after the system determines the total number of parameters of the training model corresponding to the model training task, the total amount of training samples used, and the feature dimension information, calculate the computing resource quantization value required for the model training task according to the following formula:

[0077] C comp = P × S × F

[0078] where C compFor the computational complexity, P is the total number of model parameters, S is the total amount of training sample data, and F is the feature dimension.

[0079] In step S204 of some embodiments, the system determines that the task type of the task to be processed is a real-time interaction task by parsing. The system analyzes the real-time interaction task to determine the structure of the interaction model it adopts, as well as the amount of input data for a single interaction using this interaction model. Based on this interaction model and the amount of input data, the system calculates the computational amount for a single interaction of this real-time interaction task, and determines the computational resource requirements of the real-time interaction task based on this computational amount.

[0080] Please refer to Figure 3 , in some embodiments, the data storage requirement information in step S101 can be determined through steps S301 to S303:

[0081] Step S301, analyze the task to be processed to determine the total number of input data bytes and the total number of output data bytes, and determine the task storage requirement information based on the sum of the total number of input data bytes and the total number of output data bytes;

[0082] Step S302, parse the task to be processed to determine the intermediate data generation frequency and the average number of bytes, and determine the temporary storage requirement information based on the product of the intermediate data generation frequency and the average number of bytes;

[0083] Step S303, use the sum of the task storage requirement information and the temporary storage requirement information as the data storage requirement information.

[0084] In step S301 of some embodiments, while determining the computational resource requirements of the task to be processed, the system also needs to determine the data storage resource requirements of the task to be processed; the system analyzes the input data scale and output data scale of the task to be processed to determine the total number of input data bytes and the total number of output data bytes, and determines the data storage requirements of this task to be processed by calculating the sum of the total number of input data bytes and the total number of output data bytes; in this embodiment, according to the different types of input and output data, different calculation methods are adopted to determine the total number of input and output data bytes. Exemplarily, if the data is structured, such as a database table, etc., the system calculates the average number of bytes per row of data and the total number of rows according to the definition of the data structure, and determines the total number of data bytes by calculating the product of the two; if the data is unstructured data, such as images, text files, etc., the system accumulates and calculates according to the file size to determine the total number of data bytes.

[0085] In step S302 of some embodiments, the system also needs to analyze the temporary data storage requirements of the task to be processed during task execution for the intermediate data generated during task execution; by analyzing the task to be processed, determine the frequency of intermediate data generation and the average number of bytes of intermediate data, and calculate through the following formula to determine the temporary data storage requirements of the task to be processed:

[0086] D total -temp = f temp ×D temp

[0087] Where, D total -temp is the temporary data storage requirement, f temp is the frequency of intermediate data generation, D temp is the average number of bytes of intermediate data.

[0088] In step S303 of some embodiments, the system combines the data storage resource requirements and temporary data storage requirements obtained in step S301 and step S302 to determine the final data storage resource requirement information of the task to be processed; in this embodiment, the system also analyzes the read / write mode of the data in the task to be processed, determines the sequential read / write ratio and random read / write ratio of the data in the task to be processed, as well as the corresponding read / write speed requirements, so as to further determine the performance requirements of the data storage resources. For example, for tasks that batch process data, the sequential read / write ratio is relatively high; for tasks that frequently query and update a small amount of data, the random read / write ratio is relatively high; the system determines the read / write speed requirements of the task to be processed according to the read / write mode and data scale.

[0089] Please refer to Figure 4 , in some embodiments, the network bandwidth requirement information in step S101 can be determined through steps S401 to S404:

[0090] Step S401, analyze the task to be processed to determine the task type of the task to be processed;

[0091] Step S402, if the task type is a data processing task, determine the data transfer volume and transfer frequency according to the data processing task, and determine the network bandwidth requirement information according to the product of the data transfer volume and transfer frequency;

[0092] Step S403, if the task type is a model training task, determine the total number of bytes transferred and the model parameter synchronization frequency according to the model training task, and determine the network bandwidth requirement information according to the product of the total number of bytes transferred and the model parameter synchronization frequency;

[0093] Step S404, if the task type is a real-time interaction task, determine the data input rate and the data output rate according to the real-time interaction task, and determine the network bandwidth requirement information according to the sum of the data input rate and the data output rate.

[0094] In step S401 of some embodiments, the system also analyzes the task type of the task to be processed, determines the data transmission scenario designed for the task to be processed according to the determined task type, determines the requirement for network bandwidth resources of the task, and further ensures the data transmission requirement of the task to be processed and the real-time nature of task processing.

[0095] In step S402 of some embodiments, the system analyzes and determines that the task type of the task to be processed is a data processing task. The system further analyzes the data transmission situation of the data processing task among different computing nodes during the task processing. Exemplarily, it is transmitted from the data source node to the processing node and then from the processing node to the result storage node. The system analyzes the data transmission volume and transmission frequency during the transmission process, and calculates the network bandwidth requirement of the data processing task through the following formula:

[0096] B flow =D flow ×f flow

[0097] Wherein, B flow is the network bandwidth requirement, D flow is the data transmission volume between different nodes, and f flow is the data transmission frequency between different nodes.

[0098] In step S403 of some embodiments, the system analyzes and determines that the task type of the task to be processed is a model training task. Exemplarily, a distributed training task; the system needs to determine the scale of the training data adopted by the model training task, analyze the total number of bytes of the training data transmitted from the storage node to the computing node, and the synchronization frequency of the parameters of the target model during the execution of the model training task; calculate the network bandwidth requirement during the execution of the model training task according to the following formula:

[0099] B train =D train ×f sync

[0100] Wherein, B train is the network bandwidth requirement of the model training task, D train is the total number of bytes of the training data transmission, and f sync is the parameter synchronization frequency of the model during the training process.

[0101] In step S404 of some embodiments, the system determines through analysis that the task type of the task to be processed is a real-time interaction task. This type of task requires a high task execution efficiency to ensure the real-time nature of the task. The system allocates high-performance computing resources to this type of task to improve the data processing efficiency during task execution. At the same time, the system also needs to ensure high efficiency for the data input to and output from the high-performance computing resources to avoid transmission delay affecting the overall efficiency of the real-time interaction task. Therefore, the system determines the input rate and output rate of real-time data during the execution of the real-time interaction task, and calculates the network bandwidth requirement of the real-time interaction task according to the following formula:

[0102] B real -time = R in +R out

[0103] Wherein, B real -time is the network bandwidth requirement of the real-time interaction task, R in is the input rate of real-time data during the execution of the real-time interaction task, and R out is the output rate of real-time data during the execution of the real-time interaction task; in this embodiment, the system can also comprehensively consider the network bandwidth requirements in various data transmission scenarios to obtain the total network bandwidth requirement of the task.

[0104] Please refer to Figure 5 , in some embodiments, step S102 may include but is not limited to steps S501 to S502:

[0105] Step S501, determine the idle resource information according to the preset resource registration library; wherein, the idle resource information includes idle computing resource information, remaining storage capacity information, and idle bandwidth resource information;

[0106] Step S502, perform a difference calculation based on the task resource requirement information and the idle resource information to determine a resource difference set; perform a ratio calculation based on the task resource requirement information and the resource difference set to determine a first matching score set; wherein, the first matching score set includes a computing resource matching score, a data storage matching score, and a network bandwidth matching score.

[0107] In step S501 of some embodiments, after the system analyzes the task to be processed and determines its corresponding computing resource requirements, data storage resource requirements, and network bandwidth requirements, the system queries the resource record information in the resource registration library to determine the idle resources in the resource registration library. Exemplarily, the idle rate of the CPU, the idle situation of the memory, etc.; the system matches the idle resources with the resource requirements of the task to be processed and filters out the best resources to process the task.

[0108] In step S502 of some embodiments, the task resource requirements of the task to be processed include computing resource requirements, data storage resource requirements, and network bandwidth resource requirements. Therefore, the system matches these three types of resources according to the resource type and calculates the matching scores of these three types of resources, including computing resource matching scores, data storage matching scores, and network bandwidth matching scores. Specifically, the system determines whether the idle resources in the resource registry meet the task resource requirements of the task to be processed by calculating the difference between the task resource requirements and the corresponding idle resources in the resource registry, and then calculates the resource matching score according to the ratio of the task requirements to the difference. The larger the score, the greater the redundancy of the resource for the task to be processed and the lower the load for executing the task to be processed. In this embodiment, if the corresponding idle resources in the resource registry cannot meet the task resource requirements of the task to be processed, the matching score of the idle resources is zero. The system calculates the matching score of the idle resources in the resource registry through the following formula:

[0109]

[0110] where M comp 、M store and M bw are the computing resource matching score, data storage matching score, and network bandwidth resource matching score respectively, C comp 、D total and B total are the computing resource requirements, data storage requirements, and network bandwidth requirements respectively, C node -comp、D node -store and B node -bw are the available computing power of the computing node in the resource registry, the remaining storage capacity of the storage device where the computing node is located, and the idle bandwidth of the network link where the computing node is located respectively. Specifically, C node is all the computing resources of the computing node, comp is the used computing resources, D node is all the storage resources of the storage device where the computing node is located, store is the used storage capacity of the storage device, B node is all the bandwidth resources of the network link where the computing node is located, and bw is the used bandwidth resources of the network link.

[0111] In some embodiments, the system performs weighted calculation based on preset weights and matching scores to determine the final matching score, sorts the computing nodes in the resource registry according to the final matching score, and selects the computing node with the highest score to process the task. If the final matching scores of multiple computing nodes are the same, the system can select the target computing node for task processing by comparing the load balancing situations of each computing node. Specifically, the load balancing situation of a computing node is determined by calculating the sum of the absolute values of the differences between the average load of each computing node in the computing resource cluster and the load of the current computing node, and the computing node with the smallest sum of the absolute values of the differences is selected as the target computing node to ensure the overall load balancing of the system after task allocation. Among them, the preset weights of the system can be adjusted according to system experience and optimization goals. In this embodiment, if there are dependencies between multiple tasks to be processed, such as multiple subtasks obtained by splitting a single task, the system preferentially selects the computing nodes in the same cluster or adjacent clusters with lower network latency for task processing to reduce data transmission latency and improve task processing efficiency.

[0112] Please refer to Figure 6 , in some embodiments, a task scheduling method provided by an embodiment of the present invention may further include but is not limited to steps S601 to S603:

[0113] Step S601: Analyze the task to be processed to determine the task priority; and process the task to be processed according to a preset time estimation model to determine the estimated task execution time.

[0114] Step S602: Calculate according to the task priority and the estimated task execution time to determine the resource adjustment information.

[0115] Step S603: Adjust the resources of the target computing node according to the resource adjustment information, and adjust the order of the task execution queue according to the task priority and the estimated task execution time.

[0116] In step S601 of some embodiments, the system performs task allocation based on the task resource requirements of the task to be processed and the idle resources in the resource registration library, and determines the optimal computing node to process the received task to be processed; during the task execution, the system monitors the task, dynamically schedules the system resources, and adjusts the allocated resources and task execution order of the task in real time to ensure the task execution efficiency; in this embodiment, the system analyzes the task type and task priority of the task to be processed, and inputs the task into the constructed execution time prediction model for processing to obtain the predicted execution time of the task; for subsequent task and resource dynamic scheduling; exemplarily, the task priority of the task to be processed is classified according to the task source and the corresponding business importance, such as using 1 to 10 for priority analysis, 1 being the lowest priority, and 10 being the highest priority; if the task to be processed directly affects the critical business scope, a higher priority is set, such as 8-10; if the task to be processed is a general data statistical analysis task, a general priority is set, such as 3-5; the system constructs a corresponding execution time prediction model according to different task types, inputs the tasks of the corresponding type into the execution time prediction model for processing, and comprehensively calculates information such as the computational complexity, model parameters, and data byte volume of the task to obtain the predicted execution time of the task.

[0117] In step S602 of some embodiments, for the dynamic scheduling of computing resources, the system calculates the allocation ratio of computing resources according to the following formula:

[0118]

[0119] where CR allocation is the computing resource allocation ratio, PR is the task priority, ET is the predicted execution time, and k1 and k2 are the adjustment coefficients of the system, which can be adjusted according to the system performance and experience; for the storage resource allocation, the system schedules and allocates different performance storage resources according to the task priority, the available capacity of different performance storage resources in the resource cluster, and the expected storage requirements of the task. For example, high-priority tasks are preferentially allocated high-speed storage, and for training tasks, the system can allocate the storage resource ratio according to the priority of the training task, the access frequency of the training data, and the data volume of the training data.

[0120] In step S603 of some embodiments, the system dynamically schedules corresponding resources for the tasks being processed according to the determined resource allocation ratio; exemplarily, for computing resources, if the obtained computing resource allocation ratio is 0.8, 80% of the idle computing resources are allocated to the task; the system also adjusts the task execution order according to the task priority and the determined predicted execution time, and adjusts the tasks with higher priority and shorter predicted execution time to the front of the task execution queue for the system to execute preferentially.

[0121] Please refer to Figure 7 , in some embodiments, a task scheduling method provided by an embodiment of the present invention may further include but is not limited to steps S701 to S703:

[0122] Step S701, record the actual execution time and compare the actual execution time with the estimated task execution time;

[0123] Step S702, if the actual execution time is less than or equal to the estimated task execution time, keep the current task execution queue;

[0124] Step S703, if the actual execution time is greater than the estimated task execution time, calculate the difference based on the actual execution time and the estimated task execution time to determine the time difference, and compare the time difference with a preset threshold; if the time difference is greater than or equal to the preset threshold, reduce the task priority of the corresponding task, and adjust the order of the task execution queue according to the reduced task priority; otherwise, keep the current task execution queue.

[0125] In step S701 of some embodiments, during the task execution process, the system records the actual execution time of the task through the set task monitoring program, and compares the recorded actual execution time with the estimated execution time before the task execution to determine whether the execution efficiency of the task meets the expectation, and further determines whether to perform dynamic scheduling on the task.

[0126] In step S702 of some embodiments, if the actual execution time of the task is less than or equal to its corresponding estimated execution time, it indicates that the execution efficiency of the task meets the expectation, and the system keeps the position of the task in the task execution queue without adjustment.

[0127] In step S703 of some embodiments, if the actual execution time of a task is greater than its corresponding estimated execution time, it indicates that the execution efficiency of this task is lower than expected, and it is necessary to adjust the execution order of the tasks to ensure the execution of other tasks. The specific adjustment operation of the system for this task is determined according to the difference between the actual execution time and the estimated execution time of the task. The system compares the calculated difference between the actual execution time and the estimated execution time with the preset time threshold in the system to determine the specific task scheduling operation. If the calculated difference is greater than or equal to the set time threshold, such as exceeding the estimated time by 20%, the system reduces the priority of the currently executing task and readjusts the position of this task in the task execution queue according to the reduced priority. In this embodiment, a gradient ratio can also be set to set different levels of processing operations. For example, if the difference between the actual execution time and the estimated execution time exceeds the estimated execution time by 20%, the priority of the current task is reduced by one level; if it exceeds the estimated execution time by 50%, the system adjusts the current task to the end of the task execution queue; if it exceeds the estimated execution time by 80%, the system deletes the current task from the task execution queue. If the calculated difference is less than the set time threshold, it indicates that the computing performance may fluctuate due to factors such as high load and high hardware temperature of the computing nodes in the cluster. The system can still accept the task execution time timeout of this task and keeps the position of this task in the task execution queue without adjustment.

[0128] In some embodiments, when the system monitors that a certain task is processed and completed in advance and the corresponding computing node resources are released, the system analyzes the tasks in the task execution queue and allocates the released computing node resources to the high-priority tasks in the task execution queue for resource allocation, improving the task execution efficiency and executing some tasks that meet the resource conditions in advance.

[0129] Next, in combination with specific application examples, the solution of the embodiment of the present invention will be introduced and described in detail:

[0130] A task scheduling method provided by an embodiment of the present invention is applied in an enterprise-level machine learning environment including multiple data centers, various types of computing nodes (CPU nodes, GPU nodes), and public and private storage resources, and uses a data mining platform for model training and inference tasks. Among them, the data centers are connected by a high-speed network, the computing nodes run on a Kubernetes cluster, and the platform is responsible for managing and scheduling the allocation of tasks among different nodes and resources.

[0131] Please refer to Figure 8, the data platform first conducts overall initialization of various resources in the data center, assigns unique digital identifiers to various clusters and resource groups in the data center, constructs a centralized resource registry based on the assigned digital identifiers and various clusters and resource groups. When new resources are added, the data platform enters information such as the type of resources, cluster identifiers, parameters, access permissions, etc., and initializes the status of the resources to "available"; refer to Figure 9 , the data platform classifies various resources in the resource registry based on resource types and performance parameters. For example, computing resources are classified as high, medium, and low performance; storage resources are classified as high, medium, and low speed capacity. Then, an index is established based on the identifier information, cluster information, and current resource status information of each type of resource obtained from the classification. The data platform deploys resource monitoring agent programs in each cluster and resource group, and regularly collects resource status information in each cluster and resource group by running these programs. For example, record the idle rate, usage rate, temperature, etc. of CPU / GPU nodes; record the used amount, remaining amount, usage rate, etc. of memory resources; record the capacity usage percentage, read / write busy level, etc. of storage resources. The data platform receives the recorded resource status information, updates it to the corresponding resource records in the resource registry, and real-time updates the status flags of each resource. If a resource failure is detected, the status identifier of the resource is updated to "unavailable"; refer to Figure 10 , the data platform analyzes the tasks submitted by the system, determines the computing resource requirements, data storage requirements, and network bandwidth requirements of the tasks. Based on the computing resource requirements, data storage requirements, and network bandwidth requirements obtained from the analysis, it searches for resources in the resource registry, calculates the matching scores of each computing node, and determines the target task allocation node based on the allocation decision set by the data platform. For example, sort and filter the computing nodes according to the matching scores; allocate the tasks submitted by the system to the determined target task allocation nodes for computing; at the same time, refer to Figure 11, the data platform analyzes and quantifies the data of the tasks, determines the priorities of the tasks, differentiates and marks the types of the tasks, and calculates the estimated execution time of the tasks through a preset estimated time model; the data platform, based on the determined task priorities, task types, and calculated estimated execution times, combines a preset dynamic scheduling policy to allocate resources to the tasks and adjust the task execution order; among them, resource allocation includes computing resource allocation and storage access permission allocation; adjusting the task execution order requires obtaining the current task execution queue, inserting the new task into the appropriate position in the task execution queue according to the task priority and estimated execution time, the data platform re-evaluates the task execution queue according to the set period, determines whether the actual execution time of each task exceeds a certain proportion of the estimated execution time, the data platform downgrades the priority of the corresponding task and readjusts its position in the task execution queue; at the same time, the data platform determines in real time whether the task is completed ahead of schedule and releases a large amount of resources, allocates the large amount of resources released by the task completed ahead of schedule to the high-priority tasks that meet the resource requirements, and executes the high-priority task in advance; by establishing a resource registration library to manage and schedule resources, it solves the problem of poor resource management in traditional methods, improves the utilization rate of resources, and reduces resource idleness and waste.

[0132] Implementing the embodiments of the present invention includes the following beneficial effects: The embodiments of the present invention provide a task scheduling method, system, electronic device, and storage medium. This solution analyzes the to-be-processed tasks obtained, determines the computing resource requirement information, data storage requirement information, and network bandwidth requirement information corresponding to the to-be-processed tasks, and uses the above requirement information as task resource requirement information; performs resource matching on the task resource requirement information obtained through analysis with a preset resource registration library to obtain the matching scores of different computing resources corresponding to the to-be-processed tasks, and performs weighted operations according to the preset weight information and the obtained matching scores to obtain the final matching scores of different computing resources; sorts the computing resources in the preset resource registration library according to the final matching scores of the computing resources, determines the target computing resources and constructs a computing node, and allocates the to-be-processed tasks to the computing node for processing; by analyzing and calculating the to-be-processed tasks to determine the resource requirements of the tasks, matching appropriate computing resources in the resource library according to the resource requirements, and forming a computing node to process the tasks, the resource utilization rate is improved; and by allocating appropriate resources to process the tasks, the task execution efficiency is improved.

[0133] As Figure 12 shown, the embodiments of the present invention also provide a task scheduling system, which can implement the above-mentioned task scheduling method. The system includes:

[0134] The first module is used to analyze the obtained task to be processed and determine the task resource requirement information; wherein, the task resource requirement information includes computing resource requirement information, data storage requirement information, and network bandwidth requirement information;

[0135] The second module is used to match the task resource requirement information with a preset resource registration library to determine a first set of matching scores; and perform weighted calculation according to a preset weight set and the first set of matching scores to determine a target matching score;

[0136] The third module is used to sort the preset resource registration library according to the target matching score to determine a target computing node; and allocate the task to be processed to the target computing node.

[0137] It can be seen that the content in the above method embodiments is applicable to the system embodiment of the present application. The functions specifically implemented in the system embodiment of the present application are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0138] An embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned task scheduling method is implemented. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0139] It can be understood that the content in the above method embodiments is applicable to the device embodiment of the present application. The functions specifically implemented in the device embodiment of the present application are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0140] Please refer to Figure 13 , Figure 13 which shows the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0141] A processor 1301, which can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solution provided by the embodiment of the present application;

[0142] The memory 1302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1302 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1302 and are called by the processor 1301 to execute a task scheduling method according to an embodiment of the present application;

[0143] The input / output interface 1303 is used to implement information input and output;

[0144] The communication interface 1304 is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0145] The bus 1305 transmits information between various components of the device (such as the processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304);

[0146] Among them, the processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304 are communicatively connected to each other inside the device through the bus 1305.

[0147] Among them, the memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. The memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a remote memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] In addition, an embodiment of the present application also discloses a computer program product or a computer program. The computer program product or the computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above method. Similarly, the content in the above method embodiments is applicable to this storage medium embodiment. The functions specifically implemented by this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0149] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the above-mentioned task scheduling method.

[0150] It can be understood that the content in the above method embodiments is applicable to this storage medium embodiment. The functions specifically implemented by this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0151] It can be understood that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and their appropriate combinations. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or can be implemented as hardware, or can be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0152] A task scheduling method, a task scheduling system, an electronic device, and a storage medium provided by an embodiment of the present invention analyze the acquired task to be processed, determine the computational resource requirement information, data storage requirement information, and network bandwidth requirement information corresponding to the task to be processed, and use the above requirement information as task resource requirement information; perform resource matching on the analyzed task resource requirement information with a preset resource registry to obtain the matching scores of different computational resources corresponding to the task to be processed, and perform a weighted operation according to the preset weight information and the obtained matching scores to obtain the final matching scores of different computational resources; sort the computational resources in the preset resource registry according to the final matching scores of the computational resources, determine the target computational resource and construct a computing node, and allocate the task to be processed to the computing node for processing; by analyzing and calculating the task to be processed to determine the resource requirements of the task, matching suitable computational resources in the resource library according to the resource requirements, forming a computing node to process the task, improving resource utilization; and by allocating suitable resources to process the task, improving the task execution efficiency.

[0153] The above is a specific description of the preferred embodiment of the present invention, but the present invention is not limited to the described embodiment. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A task scheduling method, characterized in that, The method includes: Analyze the obtained task to be processed to determine the task resource requirement information; wherein, the task resource requirement information includes computing resource requirement information, data storage requirement information, and network bandwidth requirement information; Match the task resource requirement information with a preset resource registration library to determine a first set of matching scores; perform weighted calculation according to a preset weight set and the first set of matching scores to determine the target matching score; Sort the preset resource registration library according to the target matching score to determine the target computing node; allocate the task to be processed to the target computing node.

2. The method according to claim 1, wherein The computing resource requirement information is determined by the following method: Analyze the task to be processed to determine the task type; If the task type is a data processing task, determine the target processing algorithm according to the data processing task, compare it with the target processing algorithm according to a preset mapping relationship to determine the computational complexity, and determine the computing resource requirement information according to the computational complexity; If the task type is a model training task, determine the target training model, the number of training samples, and the feature dimension according to the model training task, and determine the total number of model parameters according to the target training model; and determine the computing resource requirement information according to the product of the total number of model parameters, the number of training samples, and the feature dimension; If the task type is a real-time interaction task, determine the interaction model and the input data volume according to the real-time interaction task, and determine the computing resource requirement information according to the interaction model and the input data volume.

3. The method according to claim 1, characterized in that, The data storage requirement information is determined by the following method: Analyze the task to be processed to determine the total number of bytes of input data and the total number of bytes of output data, and determine the task storage requirement information according to the sum of the total number of bytes of input data and the total number of bytes of output data; Analyze the task to be processed to determine the intermediate data generation frequency and the average number of bytes, and determine the temporary storage requirement information according to the product of the intermediate data generation frequency and the average number of bytes; Use the sum of the task storage requirement information and the temporary storage requirement information as the data storage requirement information.

4. The method according to claim 1, wherein The network bandwidth requirement information is determined by the following method: Analyze the task to be processed to determine the task type of the task to be processed; If the task type is a data processing task, determine the data transmission volume and the transmission frequency according to the data processing task, and determine the network bandwidth requirement information according to the product of the data transmission volume and the transmission frequency; If the task type is a model training task, determine the total number of bytes transmitted and the model parameter synchronization frequency according to the model training task, and determine the network bandwidth requirement information according to the product of the total number of bytes transmitted and the model parameter synchronization frequency; If the task type is a real-time interaction task, determine the data input rate and the data output rate according to the real-time interaction task, and determine the network bandwidth requirement information according to the sum of the data input rate and the data output rate.

5. The method according to claim 1, characterized in that Performing matching with a preset resource registry according to the task resource requirement information to determine a first matching score set, specifically including: Determining idle resource information according to the preset resource registry; wherein, the idle resource information includes idle computing resource information, remaining storage capacity information, and idle bandwidth resource information; Performing difference calculation based on the task resource requirement information and the idle resource information to determine a resource difference set; performing ratio calculation based on the task resource requirement information and the resource difference set to determine a first matching score set; wherein, the first matching score set includes a computing resource matching score, a data storage matching score, and a network bandwidth matching score.

6. The method according to claim 1, wherein The method further includes: Analyzing the to-be-processed task to determine the task priority; and processing the to-be-processed task according to a preset time estimation model to determine the estimated task execution time; Calculating according to the task priority and the estimated task execution time to determine resource adjustment information; Adjusting the resources of the target computing node according to the resource adjustment information, and adjusting the order of the task execution queue according to the task priority and the estimated task execution time.

7. The method according to claim 6, wherein The method further includes: Recording the actual execution time, and comparing the actual execution time with the estimated task execution time; If the actual execution time is less than or equal to the estimated task execution time, maintaining the current task execution queue; If the actual execution time is greater than the estimated task execution time, performing difference calculation based on the actual execution time and the estimated task execution time to determine a time difference, and comparing the time difference with a preset threshold; if the time difference is greater than or equal to the preset threshold, reducing the task priority of the corresponding task, and adjusting the order of the task execution queue according to the reduced task priority; otherwise, maintaining the current task execution queue.

8. A task scheduling system, characterized in that, Including: A first module, configured to analyze the to-be-processed task obtained to determine task resource requirement information; wherein, the task resource requirement information includes computing resource requirement information, data storage requirement information, and network bandwidth requirement information; A second module, configured to perform matching with a preset resource registry according to the task resource requirement information to determine a first matching score set; performing weighted calculation according to a preset weight set and the first matching score set to determine a target matching score; A third module, configured to sort the preset resource registry according to the target matching score to determine a target computing node; and allocating the to-be-processed task to the target computing node.

9. An electronic device, characterized in that, Including: At least one processor; At least one memory, configured to store at least one program; When the at least one program is executed by the at least one processor, enabling the at least one processor to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that, The program executable by the processor, when executed by the processor, is used to execute the method according to any one of claims 1-7.

Citation Information

Cited By

  • Edge computing low-power-consumption Hash acceleration and compression storage system

    CN121635655A

  • Large model application service automatic scheduling method and system under finite computing power

    CN121704990A