A container platform-based intelligent inference service clustering scheduling method and system

Abstract

The application provides a container platform-based intelligent reasoning service cluster scheduling method and system, a model warehouse is preset in the container platform, a plurality of service models are arranged in the model warehouse, the container platform is further provided with a container for providing a running environment for the service models, the method comprises the following steps: receiving a reasoning request, matching a corresponding service model in the model warehouse according to the service type corresponding to the reasoning request; placing the service model in the container, the service model uses forward reasoning, the forward reasoning runs in the running environment provided by the container; scheduling the container to a node in a node cluster based on a preset scheduling strategy of the container, a plurality of nodes are arranged in the node cluster, the nodes are used for providing hardware resources for the running of the service models in the container; running the service models in the container on the hardware resources provided by the nodes to obtain a reasoning result, and feeding back the reasoning result to the sending end of the reasoning request.

Description

Technical Field

[0001] This invention relates to the field of intelligent reasoning service technology, and in particular to a clustered scheduling method and system for intelligent reasoning services based on a container platform. Background Technology

[0002] Intelligent reasoning technology is a crucial component of artificial intelligence, with the primary goal and means of achieving this being to endow computers with human-like reasoning capabilities. With the rise of big data technology, higher demands are placed on the operational efficiency of high-concurrency applications such as deep learning and data mining. NVIDIA Triton open-source inference service technology has emerged, building a distributed collaborative and unified scheduling management system for computing resources. It achieves dynamic aggregation and pooling management of CPU and GPU computing resources, enabling optimization from within the neural network itself. To further improve high-concurrency performance, reduce response latency, and optimize server-side performance, the Triton inference server simplifies the large-scale deployment of AI models in production environments. It supports obtaining any model framework (TensorFlow, TensorRT, Caffe2, PyTorch, ONNX Runtime, or custom frameworks) through a single node and can be deployed on any GPU- or CPU-based infrastructure.

[0003] However, with the advancement of technology, the amount of resources required for intelligent inference is increasing. Existing open-source inference service technologies can achieve dynamic aggregation and scheduling of CPU and GPU resources on a single machine, but without clustered deployment and operation, service resources cannot be fully utilized. The data and architecture in inference service application scenarios are complex, and excessive load often leads to deterioration of service performance or even service interruption. Clustered dynamic scheduling of services is crucial. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and system for clustered scheduling of intelligent inference services based on a container platform, in order to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides a clustered scheduling method for intelligent inference services based on a container platform. The container platform has a pre-set model repository containing various service models, and the container platform also has pre-set containers for providing a runtime environment for the service models. The method includes the following steps: Receive inference requests and match corresponding service models in the model repository according to the service type corresponding to the inference request; The service model is placed inside the container, and the service model uses forward reasoning, which runs in the runtime environment provided by the container. Based on the container's preset scheduling strategy, the container is scheduled to a node in the node cluster. The node cluster has multiple nodes, which are used to provide hardware resources for the operation of the service model in the container. The service model in the container is run on the hardware resources provided by the node to obtain the inference result, and the inference result is fed back to the sending end of the inference request.

[0006] The present invention provides a container-based intelligent inference service cluster scheduling method. Addressing the limitation of existing technologies that can only run inference services on a single node, this method employs container technology. As a lightweight virtualization technology, container technology provides container application orchestration capabilities, which can integrate various node resources together, enabling multiple nodes to complete cluster deployment. The intelligent inference service requested by the inference request can be completed jointly by multiple nodes in the node cluster, improving processing capacity and preventing the problem of service performance degradation due to excessive load.

[0007] In some embodiments of the present invention, if the container's preset scheduling policy is a load balancing policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: The container is scheduled to all or some of the nodes in the node cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes to obtain the average amount of resources. The average amount of resources is then used to run the service model in the container.

[0008] In some embodiments of the present invention, if the container's preset scheduling strategy is a bin packing scheduling strategy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling strategy includes: Any node in the node cluster is selected to be placed in the waiting area. If the resource quantity of the node in the waiting area is less than the resource quantity required for the container to run, another node is selected from the node cluster to be placed in the waiting area until the resource quantity of the node in the waiting area is greater than or equal to the resource quantity required for the container to run. Then, the service model in the container is run on multiple nodes in the waiting area. If the resource quantity of the node in the waiting area is greater than or equal to the resource quantity required for the container to run, the service model in the container is run on that node.

[0009] In some embodiments of the present invention, if the container's preset scheduling policy is a node label policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Based on the node labels preset for nodes in the node cluster, the container is matched with the corresponding node, and the container is scheduled to run on the node with the matching node label, where the service model in the container is run.

[0010] In some embodiments of the present invention, the step of matching a container with a corresponding node based on a node label preset for a node in a node cluster includes: Determine whether the resource quantity of the node with the matching node label is greater than or equal to the resource quantity required for the container to run; If so, the container is scheduled to a node with a matching node label, where the service model in the container is run. If not, the container will not be scheduled.

[0011] In some embodiments of the present invention, if the resource quantity of a node with a matching node label is less than the resource quantity required for container operation, the container scheduling strategy is modified to a load balancing strategy, and the container is scheduled to all or some nodes in the node cluster. The resource quantity required for container operation is evenly distributed based on the number of all or some nodes to obtain an average resource quantity. The average resource quantity is then used to run the service model in the container.

[0012] In some embodiments of the present invention, the node cluster includes a first sub-cluster and a second sub-cluster, each of which includes multiple nodes. The first sub-cluster is located at a first position, and the second sub-cluster is located at a second position. If the container's preset scheduling policy is an affinity policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Based on the URL information of the inference request, the sending location of the inference request is parsed, and the distances between the sending location and the first and second locations are calculated respectively. The distances between the sending location and the first and second locations are then compared. If the distance between the sending end location and the first location is relatively close, the container will be scheduled to the first sub-cluster; If the distance between the sending end location and the second location is relatively close, the container will be scheduled to the second sub-cluster.

[0013] In some embodiments of the present invention, if the container's preset scheduling policy is an affinity policy, the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy further includes: If the container is scheduled to the first sub-cluster or the second sub-cluster, then all nodes in the first sub-cluster or the second sub-cluster are traversed, the resource quantity of each node in the first sub-cluster or the second sub-cluster is calculated, and it is determined whether there is at least one node in the first sub-cluster or the second sub-cluster whose resource quantity is greater than or equal to the resource quantity required for the container to run. If so, any node from the first or second sub-cluster cluster is selected to the confirmed running area. If the resource quantity of the node in the confirmed running area is less than the resource quantity required for the container to run, another node is selected from the first or second sub-cluster cluster to the confirmed running area until the resource quantity of the node in the confirmed running area is greater than or equal to the resource quantity required for the container to run. Then, the service model in the container is run on multiple nodes in the confirmed running area. If the resource quantity of the node in the confirmed running area is greater than or equal to the resource quantity required for the container to run, the service model in the container is run on that node. If not, the container is scheduled to all or some nodes in the first sub-cluster or the second sub-cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes in the first sub-cluster or the second sub-cluster to obtain the subset average resource amount. The subset average resource amount is then used to run the service model in the container.

[0014] In some embodiments of the present invention, if at least one node in the first sub-cluster or the second sub-cluster does not have resources greater than or equal to the resources required for container operation, the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy further includes: Determine whether there exists at least one node in the first sub-cluster or the second sub-cluster whose resource quantity is less than the average resource quantity of the subset; If so, obtain the resource quantity of each node in the first sub-cluster or the second sub-cluster, calculate the proportion of the resource quantity of each node, allocate the resource quantity required for container operation to each node according to the proportion of the resource quantity, and run the service model in the container. If not, the container is scheduled to all or some nodes in the first sub-cluster or the second sub-cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes in the first sub-cluster or the second sub-cluster to obtain the subset average resource amount. The subset average resource amount is then used to run the service model in the container.

[0015] In some embodiments of the present invention, if the container's preset scheduling policy is a taint-tolerance policy, then a taint label is preset for the node, and a taint-tolerance policy is preset for the node with the taint label. The step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Containers with the same taint tolerance policy are scheduled to nodes with taint markings to run the service model within the containers.

[0016] Compared with existing technologies, the significant advantages of this invention are: high algorithm efficiency, global uniqueness, trend-increasing, concise and refined, database query-friendly, inclusion of business semantics, and support for cross-cluster (domain) use. In particular, when making resource requests and synchronizations across clusters (domains) in a domain-based cluster deployment environment, it can quickly determine the cluster (domain) to which the resource belongs, greatly simplifying the complexity of the calling model and improving service efficiency.

[0017] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0018] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0020] Figure 1 This is a schematic diagram of one embodiment of the intelligent inference service cluster scheduling method based on a container platform according to the present invention; Figure 2 This is a schematic diagram illustrating another implementation of the intelligent inference service cluster scheduling method based on a container platform according to the present invention. Figure 3 A diagram illustrating the process of running the inference model and providing feedback to the client. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0022] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0023] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0024] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0025] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0026] like Figure 1 , 2 As shown, one aspect of the present invention provides a clustered scheduling method for intelligent inference services based on a container platform. The container platform has a pre-set model repository, which can be a Kubernetes platform. The model repository contains various service models, and the container platform also has pre-set containers for providing a runtime environment for the service models. The steps of the method include: In some embodiments of the present invention, the service model may be a pre-trained convolutional neural network model or a graph convolutional model, and the intelligent reasoning service corresponding to the convolutional neural network model or the graph convolutional model may be face recognition or fingerprint recognition, etc.

[0027] Step S100: Receive an inference request and match the corresponding service model in the model repository according to the service type corresponding to the inference request; The service type corresponding to the inference request is intelligent inference service, which can be facial recognition or fingerprint recognition, etc. In some embodiments of the present invention, if the service corresponding to the inference request is face recognition, then the service model corresponding to face recognition is matched in the model repository. The service model can be a convolutional neural network model or a graph convolutional model, etc.

[0028] like Figure 2 As shown, Figure 2 The model orchestration in this invention is the process of matching the corresponding service model in the model repository according to the service type corresponding to the inference request. In some embodiments of this invention, the inference request is accessed by the client through a C language API interface, that is, the inference service access layer conveys the request. The inference service access layer supports HTTP / REST and GRPC protocols.

[0029] The client can be a PC or a mobile phone, etc.

[0030] In addition, it can also support the IPC (Inter-Process Communication) mechanism for shared memory.

[0031] Step S200: Place the service model inside the container. The service model uses forward inference, and the forward inference runs in the runtime environment provided by the container. In some embodiments of the present invention, the forward inference is implemented by an inference engine, which may be TensorFlow, TensorRT, PyTorch, and ONNX Runtime, etc.

[0032] Step S300: Based on the container's preset scheduling strategy, schedule the container to a node in the node cluster. The node cluster has multiple nodes, which are used to provide hardware resources for the operation of the service model in the container. The above scheme has limitations in single-machine reasoning. Intelligent reasoning urgently needs to adopt a distributed architecture to complete business processing. The mining of massive amounts of data must rely on cloud computing technologies. This application completes intelligent reasoning through multiple nodes to improve processing speed.

[0033] In some embodiments of the present invention, the container is scheduled to a node in the node cluster based on the container's preset scheduling strategy. It can schedule all or some nodes in the node cluster at the same time, or it can call a single node.

[0034] By adopting the above solution, this invention utilizes container platform scheduling technology, integrates intelligent inference services, provides clustered intelligent inference services, optimizes inference service scheduling strategies, effectively solves the problem of uneven load distribution in single-machine deployment of inference services, and improves inference efficiency.

[0035] In a node cluster based on a container platform, an intelligent inference application can be provided by multiple containers. Each container has the same resources and number of images. When one container fails, other containers can still complete the inference service. It supports running multiple models on one CPU or GPU, and also supports running the same model on multiple GPUs.

[0036] In some embodiments of the present invention, the node can be a GPU or CPU resource, or it can be shared storage in the cloud. The nodes in the node cluster can be deployed in the same data center in the same location, or they can be deployed in multiple data centers in different locations.

[0037] like Figure 3 As shown, Figure 3 The Triton inference server is a type of node. In step S400, the service model in the container is run on the hardware resources provided by the node to obtain the inference result, and the inference result is fed back to the sending end of the inference request.

[0038] In some embodiments of the present invention, the service model is run, and if the service model is a face recognition model, the inference result is a face recognition result.

[0039] The present invention provides a container-based intelligent inference service cluster scheduling method. Addressing the limitation of existing technologies that can only run inference services on a single node, this method employs container technology. As a lightweight virtualization technology, container technology provides container application orchestration capabilities, which can integrate various node resources together, enabling multiple nodes to complete cluster deployment. The intelligent inference service requested by the inference request can be completed jointly by multiple nodes in the node cluster, improving processing capacity and preventing the problem of service performance degradation due to excessive load.

[0040] In some embodiments of the present invention, if the container's preset scheduling policy is a load balancing policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: The container is scheduled to all or some of the nodes in the node cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes to obtain the average amount of resources. The average amount of resources is then used to run the service model in the container.

[0041] By adopting the above scheme and using a load balancing strategy, the amount of resources required by the container can be distributed among multiple points, reducing the burden on a single node while improving the container's operating efficiency. Furthermore, when multiple containers are running simultaneously, the operating efficiency of multiple containers can be guaranteed at the same time.

[0042] In some embodiments of the present invention, if the resource required for the container to run is 100G and there are 50 nodes in the node cluster, then 100 / 50=2G, and each node runs with 2G of resources.

[0043] In some embodiments of the present invention, if the container's preset scheduling strategy is a bin packing scheduling strategy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling strategy includes: Any node in the node cluster is selected to be placed in the waiting area. If the resource quantity of the node in the waiting area is less than the resource quantity required for the container to run, another node is selected from the node cluster to be placed in the waiting area until the resource quantity of the node in the waiting area is greater than or equal to the resource quantity required for the container to run. Then, the service model in the container is run on multiple nodes in the waiting area. If the resource quantity of the node in the waiting area is greater than or equal to the resource quantity required for the container to run, the service model in the container is run on that node.

[0044] Using the above scheme, if the container's preset scheduling strategy is a bin-packing scheduling strategy, nodes are selected one by one to maximize the utilization of the resources of a single node, while ensuring the maximum number of idle nodes. When other containers need to be run, there are enough idle nodes available for use.

[0045] External access requests are determined by the load balancer, which ensures the consistency of distributed system applications. When the resources on a cluster node are insufficient, additional nodes can be added to the cluster and service scheduling can be completed. It also supports the status monitoring and visualization of GPU devices.

[0046] In some embodiments of the present invention, if the resource required for the container to run is 100G, and the resource of the first node selected from the node cluster is 200G, then the service model in the container runs on that node; if the resource of the first node selected from the node cluster is 50G, then a second node needs to be selected; if the resource of the second node is 20G, then a third node needs to be selected; if the resource of the third node is 50G, then the first and second nodes run 50G and 20G of the resource required for the container to run, respectively, and the third node runs the remaining 30G of the resource required for the container to run.

[0047] In this application, the resource quantity of the computing node refers to the remaining resource quantity of the computing node at present, that is, the node may already be running the inference model of one or more containers, and the remaining computing resources are used for computation. In some embodiments of the present invention, if the container's preset scheduling policy is a node label policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Based on the node labels preset for nodes in the node cluster, the container is matched with the corresponding node, and the container is scheduled to run on the node with the matching node label, where the service model in the container is run.

[0048] Using the above method, the corresponding node can be quickly matched to the container, thus improving the matching efficiency.

[0049] In some embodiments of the present invention, if the node labels of the nodes in the node cluster include labels A and B, and if a container needs to be scheduled to a node with label A, then the container will be scheduled to run on a node in the node cluster with label A.

[0050] In some embodiments of the present invention, the step of matching a container with a corresponding node based on a node label preset for a node in a node cluster includes: Determine whether the resource quantity of the node with the matching node label is greater than or equal to the resource quantity required for the container to run; If so, the container is scheduled to a node with a matching node label, where the service model in the container is run. If not, the container will not be scheduled.

[0051] In some embodiments of the present invention, if there is only one node A with the label A in the node cluster, and the resource quantity of node A is 50G, and the resource quantity required for container operation is 40G, then the container will be scheduled to run on node A.

[0052] In some embodiments of the present invention, if the resource quantity of a node with a matching node label is less than the resource quantity required for container operation, a load balancing strategy is executed to schedule the container to all or some nodes in the node cluster, and the resource quantity required for container operation is evenly distributed based on the number of all or some nodes to obtain an average resource quantity. The average resource quantity is then used to run the service model in the container.

[0053] If the resource quantity of a node with a matching node label cannot meet the needs of a specific application, the remaining nodes will use a load balancing strategy to ensure the normal operation of the container and thus the normal operation of the intelligent inference service.

[0054] In some embodiments of the present invention, if there is only one node with the label A in the node cluster, and another 10 nodes, if the resource amount of node A is 5G, and the resource amount required for container operation is 10G, then 10-5=5G, 5 / 10=0.5G, and each of the remaining 10 nodes operates with 0.5G of resources.

[0055] In some embodiments of the present invention, the node cluster includes a first sub-cluster and a second sub-cluster, each of which includes multiple nodes. The first sub-cluster is located at a first position, and the second sub-cluster is located at a second position. If the container's preset scheduling policy is an affinity policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Based on the URL information of the inference request, the sending location of the inference request is parsed, and the distances between the sending location and the first and second locations are calculated respectively. The distances between the sending location and the first and second locations are then compared. If the distance between the sending end location and the first location is relatively close, the container will be scheduled to the first sub-cluster; If the distance between the sending end location and the second location is relatively close, the container will be scheduled to the second sub-cluster.

[0056] By adopting the above scheme, containers are scheduled according to the location of the sending end, and the containers are scheduled to the first or second sub-cluster that is closer to the source, thereby improving the efficiency of transmission and feedback, reducing distortion during transmission, and improving the accuracy of intelligent inference services.

[0057] In some embodiments of the present invention, if the container's preset scheduling policy is an affinity policy, the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy further includes: If the container is scheduled to the first sub-cluster or the second sub-cluster, then all nodes in the first sub-cluster or the second sub-cluster are traversed, the resource quantity of each node in the first sub-cluster or the second sub-cluster is calculated, and it is determined whether there is at least one node in the first sub-cluster or the second sub-cluster whose resource quantity is greater than or equal to the resource quantity required for the container to run. If so, any node from the first or second sub-cluster cluster is selected to the confirmed running area. If the resource quantity of the node in the confirmed running area is less than the resource quantity required for the container to run, another node is selected from the first or second sub-cluster cluster to the confirmed running area until the resource quantity of the node in the confirmed running area is greater than or equal to the resource quantity required for the container to run. Then, the service model in the container is run on multiple nodes in the confirmed running area. If the resource quantity of the node in the confirmed running area is greater than or equal to the resource quantity required for the container to run, the service model in the container is run on that node. If not, the container is scheduled to all or some nodes in the first sub-cluster or the second sub-cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes in the first sub-cluster or the second sub-cluster to obtain the subset average resource amount. The subset average resource amount is then used to run the service model in the container.

[0058] Using the above scheme, if at least one node in the first or second sub-cluster has resources greater than or equal to the resources required for container operation, a bin packing scheduling strategy is adopted, requiring only one node to complete the container operation, thereby improving node utilization and ensuring more idle nodes; if at least one node in the first or second sub-cluster does not have resources greater than or equal to the resources required for container operation, a load balancing strategy is adopted to improve the container operation speed.

[0059] In some embodiments of the present invention, if the first sub-cluster or the second sub-cluster has 3 nodes with resources of 5, 4, and 8G respectively, and the resource required for container operation is 10G, then 10 / 3≈3.33G, and each of the 3 nodes uses 3.33G of resources; if the resource required for container operation is 6G, then the node with 8G of resources is called to run the service model in the container.

[0060] In some embodiments of the present invention, if at least one node in the first sub-cluster or the second sub-cluster does not have resources greater than or equal to the resources required for container operation, the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy further includes: Determine whether there exists at least one node in the first sub-cluster or the second sub-cluster whose resource quantity is less than the average resource quantity of the subset; If so, obtain the resource quantity of each node in the first sub-cluster or the second sub-cluster, calculate the proportion of the resource quantity of each node, allocate the resource quantity required for container operation to each node according to the proportion of the resource quantity, and run the service model in the container. If not, the container is scheduled to all or some nodes in the first sub-cluster or the second sub-cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes in the first sub-cluster or the second sub-cluster to obtain the subset average resource amount. The subset average resource amount is then used to run the service model in the container.

[0061] In some embodiments of the present invention, if the first sub-cluster or the second sub-cluster has 3 nodes with resources of 2, 4, and 8G respectively, and the resource required for container operation is 10G, then there is no node in the first sub-cluster or the second sub-cluster whose resource is greater than or equal to the resource required for container operation. Based on the number of nodes in the first sub-cluster or the second sub-cluster, the resource required for container operation is evenly distributed, resulting in a second average resource of 10 / 3 ≈ 3.33, where 2 < 3.33. The resource ratio of each node is calculated to be 1:2:4. Therefore, a node with 2G of resources operates with 10 × 1 / 7 ≈ 1.4G of resources, a node with 4G of resources operates with 10 × 2 / 7 ≈ 2.8G of resources, and a node with 8G of resources operates with 10 × 4 / 7 ≈ 5.7G of resources.

[0062] Using the above scheme, when running the average resource amount of the subset in each node, there may be instances where the resources of the node itself are insufficient to meet the average resource amount of the subset. This application allocates the resources required for container operation to each node according to the proportion of the resource amount, thus distributing more of the operating pressure to nodes with more resources and solving the problem that the resources of the node itself may not be sufficient to meet the average resource amount of the subset. Furthermore, in practical applications, the number of nodes in the first or second sub-cluster is much smaller than the total number of nodes. Calculating the proportion between the total number of nodes would consume too much time. However, this application only allocates the resources required for container operation to each node in a single sub-cluster according to the proportion of the resource amount, while ensuring computational efficiency.

[0063] In some embodiments of the present invention, if the container's preset scheduling policy is a taint-tolerance policy, then a taint label is preset for the node, and a taint-tolerance policy is preset for the node with the taint label. The step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Containers with the same taint tolerance policy are scheduled to nodes with taint markings to run the service model within the containers.

[0064] Using the above scheme, nodes with tainted markings usually have inherent limitations, such as nodes with small resource amounts, making them difficult to utilize. The tainted tolerance strategy can be a container that meets the requirements of such small resource amounts. If there is a container whose required resources are less than those of the node with the tainted markings, then it is a container with the same tolerance strategy, thereby improving node utilization.

[0065] In some embodiments of the present invention, the above-mentioned load balancing strategy, bin packing scheduling strategy, node labeling strategy, affinity strategy and taint tolerance strategy can be applied simultaneously in the same node cluster of the present application. Sub-clusters for each of the above strategies are set in advance, and the correspondence between inference requests and strategies is set. When an inference request is received, the container can be quickly scheduled to the corresponding sub-cluster and the corresponding scheduling strategy can be adopted.

[0066] This invention, based on a container platform, enables multi-framework, batch processing, and real-time inference capabilities. It solves problems related to high availability, elastic scaling, load balancing, and monitoring and maintenance of inference servers, effectively accelerating model deployment and meeting the high-concurrency, low-latency inference requirements of application systems. It can support large-scale data training and inference needs, supports deep learning computation scheduling capabilities for intelligent application systems, and has promising application prospects across cloud, edge, and device platforms.

[0067] This invention also provides a clustered scheduling system for intelligent inference services based on a container platform, including a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method.

[0068] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned container-based intelligent inference service cluster scheduling method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0069] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0070] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0071] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A clustered scheduling method for intelligent inference services based on a container platform, characterized in that, The container platform has a pre-defined model repository containing various service models. The platform also has pre-defined containers to provide a runtime environment for these service models. The method includes the following steps: Receive an inference request, and match the corresponding service model in the model repository according to the service type corresponding to the inference request; The service model is placed inside the container, and the service model uses forward reasoning, which runs in the runtime environment provided by the container. Based on the container's preset scheduling strategy, the container is scheduled to a node in a node cluster. The node cluster contains multiple nodes, which provide hardware resources for the operation of the service model within the container. The node cluster includes a first sub-cluster and a second sub-cluster, each containing multiple nodes. The first sub-cluster is located at a first position, and the second sub-cluster is located at a second position. If the container's preset scheduling strategy is an affinity strategy, the step of scheduling the container to a node in the node cluster based on this strategy includes: parsing the sending location of the inference request based on its URL information; calculating the distance between the sending location and the first and second locations; comparing the distances between the sending location and the first and second locations; if the sending location is closer to the first location, the container is scheduled to the first sub-cluster; if the sending location is closer to the second location, the container is scheduled to the second sub-cluster. The service model in the container is run on the hardware resources provided by the node to obtain the inference result, and the inference result is fed back to the sending end of the inference request.

2. The method according to claim 1, characterized in that, If the container's preset scheduling policy is a load balancing policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: The container is scheduled to all or some of the nodes in the node cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes to obtain the average amount of resources. The average amount of resources is then used to run the service model in the container.

3. The method according to claim 1, characterized in that, If the container's preset scheduling policy is a bin packing scheduling policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Any node in the node cluster is selected to be placed in the waiting area. If the resource quantity of the node in the waiting area is less than the resource quantity required for the container to run, another node is selected from the node cluster to be placed in the waiting area until the resource quantity of the node in the waiting area is greater than or equal to the resource quantity required for the container to run. Then, the service model in the container is run on multiple nodes in the waiting area. If the resource quantity of the node in the waiting area is greater than or equal to the resource quantity required for the container to run, the service model in the container is run on that node.

4. The method according to claim 1, characterized in that, If the container's preset scheduling policy is a node label policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy includes: Based on the node labels preset for nodes in the node cluster, the container is matched with the corresponding node, and the container is scheduled to run on the node with the matching node label, where the service model in the container runs.

5. The method according to claim 4, characterized in that, The steps for matching containers with corresponding nodes based on the pre-defined node labels assigned to nodes in the node cluster include: Determine whether the resource quantity of the node with the matching node label is greater than or equal to the resource quantity required for the container to run; If so, the container is scheduled to a node with a matching node label, where the service model in the container is run. If not, the container will not be scheduled.

6. The method according to claim 1, characterized in that, If the container's preset scheduling policy is an affinity policy, then the step of scheduling the container to a node in the node cluster based on the container's preset scheduling policy further includes: If the container is scheduled to the first sub-cluster or the second sub-cluster, then all nodes in the first sub-cluster or the second sub-cluster are traversed, the resource quantity of each node in the first sub-cluster or the second sub-cluster is calculated, and it is determined whether there is at least one node in the first sub-cluster or the second sub-cluster whose resource quantity is greater than or equal to the resource quantity required for the container to run. If so, then any node from the first sub-cluster or the second sub-cluster is selected to the confirmed running area. If the resource quantity of the node in the confirmed running area is less than the resource quantity required for the container to run, then another node is selected from the first sub-cluster or the second sub-cluster to the confirmed running area, until the resource quantity of the node in the confirmed running area is greater than or equal to the resource quantity required for the container to run. Then, the service model in the container is run on multiple nodes in the confirmed running area. If the resource quantity of the node in the confirmed running area is greater than or equal to the resource quantity required for the container to run, then the service model in the container is run on that node. If not, the container is scheduled to all or some nodes in the first sub-cluster or the second sub-cluster. The amount of resources required for the container to run is evenly distributed based on the number of all or some nodes in the first sub-cluster or the second sub-cluster to obtain the subset average resource amount. The subset average resource amount is then used to run the service model in the container.

7. The method according to claim 1, characterized in that, If the container's preset scheduling policy is a taint-tolerance policy, then the steps of scheduling the container to nodes in the node cluster based on the container's preset scheduling policy include: Pre-setting taint labels for nodes and setting taint-tolerance policies for nodes with taint labels. Containers with the same taint tolerance policy are scheduled to nodes with taint markings to run the service model within the containers.

8. A clustered scheduling system for intelligent inference services based on a container platform, comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images