Task scheduling method and device

Abstract

The embodiment of the invention provides a task scheduling method, and relates to the field of industrial cloud computing, industrial cloud platforms and industrial internet platforms, the task scheduling method comprises the following steps: receiving resource data reported by each computing node, the resource data comprising cache information of a model; taking out the to-be-allocated task from the task queue, and determining a target model corresponding to the to-be-allocated task; determining a target computing node based on the target model and the resource data; and distributing the to-be-distributed task to the target computing node. According to the technical scheme of the embodiment of the invention, the problems that the container needs to be restarted and the model needs to be downloaded repeatedly during each request can be avoided, the resource utilization efficiency and the request response speed are improved, and the user experience is effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a task scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In the AI graphics ecosystem, to meet user needs for diverse styles and scenarios, it's often necessary to integrate a large number of pre-trained models into the service platform. When a user initiates a drawing request, the server typically dynamically launches a container instance. This container downloads a user-specified or default pre-trained model from a storage medium (such as cloud storage), then launches the inference engine to load the model, perform inference calculations, generate image results, and ultimately return the results to the user. Upon completion, the container is destroyed to free up GPU resources.

[0003] However, this approach lacks unified management of the model, requiring the container to be restarted for each request, resulting in a large amount of start-stop overhead and serious waste of resources.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0005] Embodiments of the present application provide a task scheduling method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a task scheduling method, the method comprising: Receive resource data reported by each computing node, wherein the resource data includes cache information of the model; Take out the task to be assigned from the task queue and determine the target model corresponding to the task to be assigned; determining a target computing node based on the target model and the resource data; Allocate the task to be assigned to the target computing node.

[0007] Optionally, the method further includes: Receive the task status submitted by the target computing node through an independent thread on a regular basis.

[0008] Optionally, determining the target computing node based on the target model and the resource data includes: Determining whether each computing node has the computing capability to process the task to be assigned based on the resource data; Selecting computing nodes that have computing capabilities to process the tasks to be assigned as candidate computing nodes; Determining whether the candidate computing node has the target model cached based on the cache information of the model; A target computing node is determined based on the candidate computing nodes that cache the target model.

[0009] Optionally, the method further includes: In a case where none of the candidate computing nodes has the target model cached, determining the remaining computing capacity of each of the candidate computing nodes; A target computing node is determined from the candidate computing nodes based on the remaining computing capacity.

[0010] Optionally, the method further includes: Get usage information of all models; dividing the model into a cold model and a hot model based on the usage information, the cold model being a model with a usage frequency lower than a first frequency threshold, and the hot model being a model with a usage frequency higher than a second frequency threshold; When the cold model exists on a first computing node and the remaining disk space of the first computing node is less than a first safety threshold, the first computing node is notified to release the cold model, where the first computing node is any computing node.

[0011] Optionally, when the cold model exists on the first computing node and the remaining disk space of the first computing node is less than a first safety threshold, notifying the first computing node to release the cold model includes: When the cold model exists on the first computing node, the cold model of the first computing node meets a preset condition, and the remaining disk space of the first computing node is less than a first safety threshold, notify the first computing node to release the cold model, wherein the preset condition includes that the cold model of the first computing node is not a new model, and the new model is a model whose joining time is less than a first time threshold.

[0012] Optionally, the method further includes: If the thermal model exists on the first computing node and the remaining disk space of the first computing node is greater than a second safety threshold, the first computing node is notified to retain the thermal model.

[0013] Optionally, the method further includes: determining real-time thermal scores of all the thermal models when the disk utilization of the first computing node is lower than a first utilization threshold; determining a target thermal model from the thermal models based on the real-time thermal score; Distribute the target thermal model to the first computing node.

[0014] Optionally, the method further includes: When the disk utilization of the first computing node is higher than a second utilization threshold, distributing the model to the first computing node is stopped.

[0015] Another aspect of an embodiment of the present application provides a task scheduling device, the device comprising: A receiving module, configured to receive resource data reported by each computing node, wherein the resource data includes cache information of the model; The first determination module is used to take out the task to be assigned from the task queue and determine the target model corresponding to the task to be assigned; A second determining module, configured to determine a target computing node based on the target model and the resource data; The allocation module is used to allocate the to-be-allocated task to the target computing node.

[0016] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0017] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0018] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0019] The above technical solution adopted in the embodiments of the present application may have the following advantages: By receiving resource data reported by each computing node, the resource data includes the cache information of the model, taking out the task to be assigned from the task queue, determining the target model corresponding to the task to be assigned, determining the target computing node based on the target model and resource data, and assigning the task to be assigned to the target computing node, the task can be preferentially assigned to the computing node that has cached the target model according to the cache information of the model, thereby avoiding the problem of restarting the container and repeatedly downloading the model for each request, facilitating effective resource allocation and task scheduling from a global perspective, improving resource utilization efficiency and request response speed, and thus effectively improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0021] Figure 1 The following schematically shows an operating environment diagram of the task scheduling method according to the first embodiment of the present application; Figure 2 The following schematically shows a flowchart of a task scheduling method according to the first embodiment of the present application; Figure 3 Schematically shows Figure 2 Flowchart of sub-steps of step S204; Figure 4 The following schematically illustrates a newly added process of the task scheduling method according to the first embodiment of the present application; Figure 5 Schematically illustrates another newly added process of the task scheduling method according to the first embodiment of the present application; Figure 6 Schematically illustrates another newly added process of the task scheduling method according to the first embodiment of the present application; Figure 7 The following schematically shows an example diagram of the principle of the task scheduling method according to the first embodiment of the present application; Figure 8 The following schematically shows a model management strategy principle diagram of the task scheduling method according to the first embodiment of the present application; Figure 9 A block diagram schematically shows a task scheduling device according to the second embodiment of the present application; and Figure 10 The following schematically shows a hardware architecture diagram of a computer device according to the third embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0024] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed. They are only used to facilitate the description of this application and to distinguish each step. Therefore, they cannot be understood as limitations on this application.

[0025] First, an explanation of the terms used in this application is provided: Pre-trained Model: A machine learning model that has been trained on a large dataset in advance and can serve as the basis for downstream tasks (such as image generation).

[0026] Containerization Technology: An operating system-level virtualization technology that packages applications and their dependencies into portable containers to achieve environmental isolation and rapid deployment.

[0027] Inference Engine: A software or hardware system responsible for loading a trained model and performing computations to generate predictions or results (in this case, images).

[0028] An independent thread is the smallest unit of a computer program that can be run, scheduled, and executed independently. It is a core concept in the multithreaded programming model. In an operating system, a process can contain multiple threads, and an independent thread is a thread that is relatively isolated from other threads, either logically or through resource allocation, and executes autonomously.

[0029] Secondly, to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the following describes the relevant technologies: To meet the needs of users for different styles and scenarios in the AI drawing ecosystem, service platforms usually integrate a large number of pre-trained models and dynamically start container instances based on user requests. The container downloads the user-specified or default pre-trained model from the storage medium, then starts the inference engine to load the model for inference calculations, and returns the generated image results to the user. After completion, the container is destroyed to release GPU resources.

[0030] However, this approach has the following disadvantages: 1. Each request requires restarting the container, resulting in a large amount of container start and stop overhead; 2. The model needs to be downloaded repeatedly, wasting network bandwidth and storage resources; 3. The inference engine takes a long time to start up, and each model is a cold start, which increases the overall response delay; 4. Lack of unified management and optimized scheduling mechanism for model usage.

[0031] These problems are particularly prominent in scenarios with high concurrency and massive models, greatly affecting user experience and system efficiency.

[0032] To this end, embodiments of the present application provide a task scheduling solution. This solution receives resource data (including model cache information) reported by each computing node, determines the target model corresponding to the task to be assigned, and then determines the target computing node for assignment based on the target model and resource data. This allows tasks to be assigned to computing nodes that already have the corresponding model cache, thus avoiding the need to restart the container and repeatedly download the model for each request. This facilitates efficient resource allocation and task scheduling from a global perspective, improving resource utilization efficiency and request response speed, thereby effectively enhancing the user experience. See below for details.

[0033] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0034] Figure 1 The environment architecture diagram of the embodiment of the present application is schematically shown as follows: Figure 1 As shown: The system can be primarily divided into a computing layer 110 and a scheduling layer 120. The computing layer 110 includes multiple computing nodes. The computing layer 110 regularly reports resource data, including model cache information, to the scheduling layer 120. The scheduling layer 120 receives tasks submitted by users and places them in a task queue. When tasks need to be assigned, the scheduling layer 120 removes them from the task queue and determines the target model corresponding to the task to be assigned. The scheduling layer then determines the target computing node based on the resource data and target model of each computing node and assigns the task to the target computing node in the computing layer 110.

[0035] The technical solutions of the present application are described below through a number of embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described herein.

[0036] Example 1 Figure 2The flowchart of the task scheduling method according to the first embodiment of the present application is schematically shown. It should be noted that the execution subject of the task scheduling method of the embodiment of the present application can be Figure 1 The scheduling layer 120 in the embodiment is described below by taking the scheduling layer as the execution body as an example.

[0037] like Figure 2 As shown, the task scheduling method may include steps S200 to S206, wherein: Step S200: receiving resource data reported by each computing node, where the resource data includes cache information of the model.

[0038] Step S202: taking out the task to be assigned from the task queue and determining the target model corresponding to the task to be assigned.

[0039] Step S204: determining a target computing node based on the target model and resource data.

[0040] Step S206: Allocate the task to be allocated to the target computing node.

[0041] The task scheduling method provided in this embodiment receives resource data reported by each computing node, the resource data includes the cache information of the model, takes out the task to be assigned from the task queue, determines the target model corresponding to the task to be assigned, determines the target computing node based on the target model and the resource data, and assigns the task to be assigned to the target computing node. It can preferentially assign the task to the computing node that has cached the target model according to the cache information of the model, thereby avoiding the problem of restarting the container and repeatedly downloading the model for each request, facilitating effective resource allocation and task scheduling from a global perspective, improving resource utilization efficiency and request response speed, and thus effectively improving user experience.

[0042] The following combination Figure 2 , each step in steps S200~S206 and other optional steps are described in detail.

[0043] Step S200 , receive resource data reported by each computing node, the resource data includes the cache information of the model.

[0044] Resource data can include CPU, GPU, local model list, disk usage, supported inference capabilities, models cached on the GPU, and other resource data. Specifically, when a compute node requests a task from the scheduling layer, it can include its resource data in the request, allowing the scheduling layer to receive resource data reported by each compute node. Alternatively, a compute node can periodically report resource data to the scheduling layer via an independent thread.

[0045] Step S202, take out the task to be assigned from the task queue and determine the target model corresponding to the task to be assigned.

[0046] The tasks in the task queue can be inference tasks that require the use of a model for inference.

[0047] Specifically, the scheduling layer receives inference tasks submitted by users and places them into a task queue. If the task queue is not empty, the scheduling layer retrieves the pending tasks from the queue according to the task retrieval rules. The scheduling layer then parses the pending tasks and determines the target model they correspond to. Optionally, the scheduling layer can pre-establish mappings between various tasks and their dependent models. After retrieving the pending tasks, the scheduling layer determines the corresponding target model based on these mappings.

[0048] Step S204 ,determine the target computing node based on the target model and resource data.

[0049] After determining the target model for the task to be assigned, the resource data of each computing node can be analyzed. If the resource data most recently reported by a computing node contains cached information for the target model, this indicates that the computing node is using the target model in the GPU, and this computing node can be prioritized as the target computing node. If multiple computing nodes have cached information for the target model, a comprehensive evaluation (such as a computing power evaluation) can be performed on these computing nodes, and the target computing node can be determined based on the results of the comprehensive evaluation. If no computing node has cached information for the target model, a comprehensive evaluation can be performed on the resource data of all computing nodes, and the target computing node can be determined based on the results of the comprehensive evaluation. For example, computing nodes with sufficient computing power can be prioritized as target computing nodes.

[0050] Step S206 , assign the tasks to be assigned to the target computing nodes.

[0051] After determining the target compute node, the scheduling layer can assign the task to the target compute node. When assigning the task to the target compute node, the task can include information such as inference parameters and model dependencies, so that the target compute node can execute the assigned task based on this information.

[0052] In an optional embodiment, the task scheduling method of the embodiment of the present application may further include: receiving a task status submitted by the target computing node through an independent thread on a regular basis.

[0053] That is, the target computing node can use the main thread to execute the assigned task. In addition to the main thread, an independent thread can be used to periodically obtain the execution status of the task (such as not executed, executing, completed, etc.), and submit the obtained task status to the scheduling layer, so that the scheduling layer receives the corresponding task status.

[0054] In this embodiment, by receiving the task status submitted by the target computing node through an independent thread at regular intervals, the scheduling layer can obtain the task status in real time. At the same time, the submission of the task status of the target computing node will not affect the execution of the main thread, thereby improving the computing efficiency of the target computing node.

[0055] In an optional embodiment, in step S204, the target computing node is determined based on the target model and resource data, such as Figure 3 As shown, this may include: Step S300: determining whether each computing node has the computing capability to process the task to be assigned based on the resource data.

[0056] Step S302: select computing nodes that have computing capabilities to process the tasks to be assigned as candidate computing nodes.

[0057] Step S304: Determine whether the candidate computing node has the target model cached based on the cache information of the model.

[0058] Step S306 : determining a target computing node based on candidate computing nodes that have cached target models.

[0059] Specifically, the scheduling layer can evaluate the resource data most recently reported by each computing node to determine whether each computing node has the computing power to process the task to be assigned. For example, it can evaluate the performance of the CPU, GPU, disk, etc. to determine whether the remaining computing power of the current computing node can support the task to be assigned. It can also determine from the model list whether the current computing node has a target model. After evaluating all computing nodes, the computing nodes that have the computing power to process the task to be assigned are selected as candidate computing nodes. It is then further determined whether the target model is cached in the candidate computing nodes. If the current computing node has the target model cached, the current computing node can be directly determined as the target computing node. In the case where there are multiple candidate computing nodes that all cache the target model, the candidate computing node with the best computing power evaluation can be used as the target computing node. Alternatively, an alternative computing node can be randomly selected from these candidate computing nodes as the target computing node.

[0060] In this embodiment, by determining whether each computing node has the computing capacity to process the task to be assigned based on resource data, the computing nodes that have the computing capacity to process the task to be assigned are used as alternative computing nodes; then, based on the cache information of the model, it is determined whether the alternative computing node has cached the target model, and the target computing node is determined based on the alternative computing node that has cached the target model. The computing node that has cached the corresponding model and has sufficient computing power can be preferentially determined as the target computing node for task assignment, thereby reducing the repeated downloading of the model and the overhead of starting and stopping the container, and improving the request response speed.

[0061] In an optional embodiment, if Figure 4 As shown, the task scheduling method of the embodiment of the present application may also include: Step S400 : When none of the candidate computing nodes has the target model cached, the remaining computing capacity of each candidate computing node is determined.

[0062] Step S402: determining a target computing node from candidate computing nodes based on the remaining computing capacity.

[0063] Among them, the remaining computing capacity of the alternative computing nodes can be comprehensively evaluated from aspects such as CPU utilization, number of idle cores, remaining memory capacity, available storage space, IO load, and bandwidth usage.

[0064] Specifically, when it is determined based on the cache information of the model that none of the candidate computing nodes have cached the target model, the remaining computing capacity of each candidate computing node is determined. Then, the remaining computing capacity of the candidate computing nodes can be sorted, and the candidate computing node with the largest remaining computing capacity can be used as the target computing node.

[0065] In this embodiment, when none of the alternative computing nodes has the target model cached, the remaining computing capacity of each alternative computing node is determined, and the target computing node is determined from the alternative computing nodes based on the remaining computing capacity. Computing nodes with sufficient remaining computing capacity can be prioritized as target computing nodes, thereby improving the efficiency of task processing.

[0066] In an optional embodiment, if Figure 5 As shown, the task scheduling method of the embodiment of the present application may also include: Step S500: Obtain usage information of all models.

[0067] Step S502 : Classify the model into a cold model and a hot model based on the usage information of the model. The cold model is a model whose usage frequency is lower than a first frequency threshold, and the hot model is a model whose usage frequency is higher than a second frequency threshold.

[0068] Step S504 : when a cold model exists on the first computing node and the remaining disk space of the first computing node is less than a first safety threshold, notify the first computing node to release the cold model, where the first computing node is any computing node.

[0069] The model usage information may include usage information within a predetermined time window, such as the number of calls, usage distribution, task dependencies, and other information in the past week.

[0070] Specifically, the scheduling layer can perform statistics based on the resource data reported historically by the computing nodes to determine the usage information of all models; then, based on the usage information, the usage frequency of the models is counted, and models with a usage frequency lower than the first frequency threshold are determined as cold models, and models with a usage frequency higher than the second frequency threshold are determined as cold models. The second frequency threshold is greater than the first frequency threshold, and in particular, the second frequency threshold can also be equal to the first frequency threshold. The scheduling layer can determine the remaining disk space of each computing node based on the resource data reported by the computing nodes. When it is determined that the remaining disk space of the first computing node is less than the first safety threshold and a cold model exists on the first computing node, the scheduling layer notifies the first computing node to release the cold model.

[0071] In this embodiment, by obtaining the usage information of all models, the models are divided into cold models and hot models based on the usage information of the models. When a cold model exists in the first computing node and the remaining disk space of the first computing node is less than the first safety threshold, the first computing node is notified to release the cold model. The models can be divided into cold and hot models for management. At the same time, the models can be recycled when the remaining disk space of the computing node is small, ensuring that the computing node's processing of tasks is not affected.

[0072] In an optional embodiment, in step S504, that is, when a cold model exists on the first computing node and the remaining disk space of the first computing node is less than the first safety threshold, notifying the first computing node to release the cold model may include: When a cold model exists on the first computing node, the cold model of the first computing node meets a preset condition, and the remaining disk space of the first computing node is less than a first safety threshold, the first computing node is notified to release the cold model, wherein the preset condition includes that the cold model of the first computing node is not a new model, and the new model is a model whose joining time is less than the first time threshold.

[0073] The first time threshold can be set according to actual conditions, for example, one week, 10 days, one month, etc. In addition to the first computing node not being a new model, the preset condition can also include other conditions, such as whether the CPU usage is greater than a certain threshold.

[0074] Specifically, in addition to determining whether the first computing node has a cold model and whether the remaining disk space on the first computing node is less than a first safety threshold, the scheduling layer also determines whether the corresponding cold model is not a new model, among other pre-set conditions. If the corresponding cold model meets these pre-set conditions, the scheduling layer notifies the first computing node to release the cold model. Accordingly, if the result of the determination is that the corresponding cold model is a new model, the release of the cold model can be skipped.

[0075] In this embodiment, when a cold model exists on the first computing node, the cold model meets preset conditions such as not being a new model, and the remaining disk space of the first computing node is less than a first safety threshold, the first computing node is notified to release the cold model. This can at least exclude the situation where the model is a new model, thereby properly protecting the new model and preventing it from being prematurely classified as a cold model and recycled.

[0076] In an optional embodiment, the task scheduling method of the embodiment of the present application may further include: If the thermal model exists on the first computing node and the remaining disk space of the first computing node is greater than a second safety threshold, the first computing node is notified to retain the thermal model.

[0077] Specifically, the scheduling layer evaluates each compute node. If a hot model exists on the first compute node, it then determines based on resource data. If the remaining disk space on the first compute node exceeds a second safety threshold, the first compute node is notified to retain the hot model and not reclaim it. The second safety threshold can be the same as or different from the first. In either case, the second safety threshold is greater than the first.

[0078] In this embodiment, by notifying the first computing node to retain the thermal model when the thermal model exists on the first computing node and the remaining disk space of the first computing node is greater than the second safety threshold, the thermal model can be retained when the computing node's capacity is relatively sufficient, thereby reducing repeated downloading of the model and improving the efficiency of task execution.

[0079] In an optional embodiment, if Figure 6 As shown, the task scheduling method of the embodiment of the present application may also include: Step S600 : When the disk utilization of the first computing node is lower than a first utilization threshold, determine the real-time heat scores of all heat models.

[0080] Step S602 : determining a target thermal model from the thermal models based on the real-time heat score.

[0081] Step S604: Distribute the target thermal model to the first computing node.

[0082] Specifically, when the disk utilization of the first computing node is lower than the first utilization threshold, the scheduling layer can calculate the model usage based on the task scheduling history, and then determine the real-time heat score of all hot models based on the model usage. It is understandable that hot models may also be new models. Since newly added models often have high popularity potential, the real-time heat score can be appropriately increased when evaluating such models. After obtaining the real-time heat score of each hot model, it can be sorted and the top N hot models in the sorting can be used as the target hot models. The target hot models are distributed to the first computing node.

[0083] In this embodiment, when the disk utilization of the first computing node is lower than the first utilization threshold, the real-time heat scores of all thermal models are determined, the target thermal model is determined from the thermal models based on the real-time heat scores, and the target thermal model is distributed to the first computing node. Thermal models with higher heat can be actively distributed to the computing nodes, thereby reducing the first inference delay of subsequent tasks of the computing nodes.

[0084] In an optional embodiment, the task scheduling method of the embodiment of the present application may further include: When the disk utilization of the first computing node is higher than a second utilization threshold, distribution of the model to the first computing node is stopped.

[0085] Specifically, during the model distribution process, the scheduling layer can make a real-time judgment based on the latest reported resource data. If the disk utilization of the first computing node exceeds a second utilization threshold, the model distribution to the first computing node will be stopped. The second utilization threshold can be the same as or different from the first utilization threshold. In different cases, the second utilization threshold is greater than the first utilization threshold.

[0086] In this embodiment, by stopping the distribution of the model to the first computing node when the disk utilization of the first computing node is higher than the second utilization threshold, the distribution of the model can be stopped when the disk utilization of the computing node is high, thereby preventing unnecessary impact on the processing performance of the computing node.

[0087] Please refer to Figure 7 , which is a principle example diagram of the task scheduling method of an embodiment of the present application. As shown in the figure, it can generally include the following contents: 1. Initialization phase: 1. The computing layer downloads core resources; 2. Collect local resource information.

[0088] 2. Resource management loop (implemented by independent thread): 1. The computing nodes of the computing layer periodically report their resource data, including node ID, resource information, and model list; 2. After receiving the reported resource data, the scheduling layer updates the global resource view, executes the model management policy, and returns resource operation instructions (adding or deleting models) to the computing layer; 3. The computing layer performs corresponding resource operations according to resource operation instructions.

[0089] 3. Task processing loop (main thread): 1. The computing layer obtains tasks from the scheduling layer, and the task request carries information such as version, resources, capabilities, and cache. 2. The scheduling layer executes the task scheduling strategy and returns the inference task to the computing layer; 3. After receiving the inference task, the computing layer starts the status reporting thread (implemented by an independent thread). The status reporting thread periodically submits the task status (including task ID, status, progress or results). After receiving the submitted task status, the scheduling layer returns the status confirmation to the computing layer. 4. The computing layer prepares the task-dependent resources; 5. Select an inference engine; 6. Perform inference calculations.

[0090] Please refer to Figure 8 , which is a schematic diagram of the model management strategy of the task scheduling method according to an embodiment of the present application. As shown in the figure, it can generally include the following contents: 1. The scheduling layer periodically collects resource information of each computing node; 2. Update the global resource view based on the collected resource information; 3. Analyze the resource status of each computing node. If resources are sufficient (for example, disk utilization is less than 70%), proceed to step 4. If resources are in a critical state (for example, disk utilization is greater than or equal to 70% and less than 85%), proceed to step 5. If resources are in a critical state (for example, disk utilization is greater than or equal to 85%), proceed to step 6. 4. Distribute hot models, preload models, and optimize distribution. Distribution optimization can be achieved through a low-probability random distribution strategy. 5. Stop the active distribution of the model, maintain the existing configuration, and prioritize tasks with higher priority; 6. Identify the cold model and conduct recycling evaluation on the model. If the evaluation passes, delete it.

[0091] Example 2 Figure 9The block diagram of the task scheduling device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 9 As shown, the apparatus 700 may include: a receiving module 710, a first determining module 720, a second determining module 730, and an allocating module 740, wherein: Receiving module 710, configured to receive resource data reported by each computing node, wherein the resource data includes cache information of the model; The first determination module 720 is used to take out the to-be-assigned task from the task queue and determine the target model corresponding to the to-be-assigned task; A second determination module 730 is configured to determine a target computing node based on the target model and the resource data; The allocation module 740 is configured to allocate the to-be-allocated task to the target computing node.

[0092] In an optional embodiment, the apparatus 700 is further configured to: Receive the task status submitted by the target computing node through an independent thread on a regular basis.

[0093] In an optional embodiment, the second determining module 730 is further configured to: Determining whether each computing node has the computing capability to process the task to be assigned based on the resource data; Selecting computing nodes that have computing capabilities to process the tasks to be assigned as candidate computing nodes; Determining whether the candidate computing node has the target model cached based on the cache information of the model; A target computing node is determined based on the candidate computing nodes that cache the target model.

[0094] In an optional embodiment, the apparatus 700 is further configured to: In a case where none of the candidate computing nodes has the target model cached, determining the remaining computing capacity of each of the candidate computing nodes; A target computing node is determined from the candidate computing nodes based on the remaining computing capacity.

[0095] In an optional embodiment, the apparatus 700 is further configured to: Get usage information of all models; dividing the model into a cold model and a hot model based on the usage information, the cold model being a model with a usage frequency lower than a first frequency threshold, and the hot model being a model with a usage frequency higher than a second frequency threshold; When the cold model exists on a first computing node and the remaining disk space of the first computing node is less than a first safety threshold, the first computing node is notified to release the cold model, where the first computing node is any computing node.

[0096] In an optional embodiment, the apparatus 700 is further configured to: When the cold model exists on the first computing node, the cold model of the first computing node meets a preset condition, and the remaining disk space of the first computing node is less than a first safety threshold, notify the first computing node to release the cold model, wherein the preset condition includes that the cold model of the first computing node is not a new model, and the new model is a model whose joining time is less than a first time threshold.

[0097] In an optional embodiment, the apparatus 700 is further configured to: If the thermal model exists on the first computing node and the remaining disk space of the first computing node is greater than a second safety threshold, the first computing node is notified to retain the thermal model.

[0098] In an optional embodiment, the apparatus 700 is further configured to: determining real-time thermal scores of all the thermal models when the disk utilization of the first computing node is lower than a first utilization threshold; determining a target thermal model from the thermal models based on the real-time thermal score; Distribute the target thermal model to the first computing node.

[0099] In an optional embodiment, the apparatus 700 is further configured to: When the disk utilization of the first computing node is higher than a second utilization threshold, distributing the model to the first computing node is stopped.

[0100] Example 3 Figure 10The following schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a task scheduling method according to the third embodiment of the present application. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers), etc. Figure 10 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other via a system bus. Memory 10010 includes at least one type of computer-readable storage medium, including flash memory, a hard disk, a multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, and the like. In some embodiments, memory 10010 may be an internal storage module of computer device 10000, such as a hard disk or memory of computer device 10000. In other embodiments, memory 10010 may also be an external storage device of computer device 10000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like equipped on computer device 10000. Of course, memory 10010 may also include both internal storage modules and external storage devices of computer device 10000. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed on the computer device 10000, such as program code of the task scheduling method, etc. In addition, the memory 10010 can also be used to temporarily store various data that has been output or is to be output.

[0101] In some embodiments, processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data exchange or communication with computer device 10000. In this embodiment, processor 10020 is used to execute program code stored in memory 10010 or process data.

[0102] Network interface 10030 may include a wireless network interface or a wired network interface. Network interface 10030 is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and a communication link between computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0103] It should be pointed out that Figure 10 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the shown components is not a requirement, and more or fewer components may alternatively be implemented.

[0104] In this embodiment, the task scheduling method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0105] Example 4 An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the task scheduling method in the embodiment when executed by a processor.

[0106] In this embodiment, computer-readable storage media include flash memory, hard disks, multimedia cards, card-type memories (e.g., SD or DX memories), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disks, optical disks, and the like. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the computer device's hard disk or memory. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like. Of course, the computer-readable storage medium may also include both the internal storage unit and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the task scheduling method described in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that has been output or is about to be output.

[0107] Example 5 An embodiment of the present application further provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0108] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using general-purpose computer devices. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Alternatively, they can be implemented using program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0109] It should be noted that the above are only preferred embodiments of the present application and do not limit the scope of patent protection of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present application.

Claims

1. A task scheduling method, characterized in that: The method comprises: Receive resource data reported by each computing node, wherein the resource data includes cache information of the model; Take out the task to be assigned from the task queue and determine the target model corresponding to the task to be assigned; determining a target computing node based on the target model and the resource data; Allocate the task to be assigned to the target computing node.

2. The method according to claim 1, characterized in that The method further comprises: Receive the task status submitted by the target computing node through an independent thread on a regular basis.

3. The method according to claim 1, characterized in that The determining of the target computing node based on the target model and the resource data includes: Determining whether each computing node has the computing capability to process the task to be assigned based on the resource data; Selecting computing nodes that have computing capabilities to process the tasks to be assigned as candidate computing nodes; Determining whether the candidate computing node has the target model cached based on the cache information of the model; A target computing node is determined based on the candidate computing nodes that cache the target model.

4. The method according to claim 3, characterized in that The method further comprises: In a case where none of the candidate computing nodes has the target model cached, determining the remaining computing capacity of each of the candidate computing nodes; A target computing node is determined from the candidate computing nodes based on the remaining computing capacity.

5. The method according to claim 1, wherein The method further comprises: Get usage information of all models; dividing the model into a cold model and a hot model based on the usage information, the cold model being a model with a usage frequency lower than a first frequency threshold, and the hot model being a model with a usage frequency higher than a second frequency threshold; When the cold model exists on a first computing node and the remaining disk space of the first computing node is less than a first safety threshold, the first computing node is notified to release the cold model, where the first computing node is any computing node.

6. The method according to claim 5, characterized in that When the cold model exists on the first computing node and the remaining disk space of the first computing node is less than a first safety threshold, notifying the first computing node to release the cold model includes: When the cold model exists on the first computing node, the cold model of the first computing node meets a preset condition, and the remaining disk space of the first computing node is less than a first safety threshold, notify the first computing node to release the cold model, wherein the preset condition includes that the cold model of the first computing node is not a new model, and the new model is a model whose joining time is less than a first time threshold.

7. The method according to claim 5, characterized in that The method further comprises: If the thermal model exists on the first computing node and the remaining disk space of the first computing node is greater than a second safety threshold, the first computing node is notified to retain the thermal model.

8. The method according to claim 7, characterized in that The method further comprises: determining real-time thermal scores of all the thermal models when the disk utilization of the first computing node is lower than a first utilization threshold; determining a target thermal model from the thermal models based on the real-time thermal score; Distribute the target thermal model to the first computing node.

9. The method according to claim 8, characterized in that The method further comprises: When the disk utilization of the first computing node is higher than a second utilization threshold, distributing the model to the first computing node is stopped.

10. A task scheduling device, characterized in that: The device comprises: A receiving module, configured to receive resource data reported by each computing node, wherein the resource data includes cache information of the model; The first determination module is used to take out the task to be assigned from the task queue and determine the target model corresponding to the task to be assigned; A second determining module, configured to determine a target computing node based on the target model and the resource data; The allocation module is used to allocate the to-be-allocated task to the target computing node.

11. A computer device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images