Cluster resource scheduling method and device, electronic equipment and storage medium

Abstract

The invention discloses a cluster resource scheduling method and device, electronic equipment and a storage medium, which are used for solving the problems of unbalanced cluster resource scheduling and low resource utilization rate, and the method comprises the following steps: receiving a resource scheduling request sent by a client, the resource scheduling request carrying domain name information of a target model reasoning service; obtaining a target model identifier according to the domain name of the target model reasoning service, and obtaining a resource specification of the target model reasoning service according to the target model identifier; determining whether each cluster can be expanded or not according to the resource specification of the target model reasoning service and the hardware specification of the node in each cluster; obtaining the resource utilization rate of the target model reasoning service in each cluster, and determining the health degree of the target model reasoning service in the corresponding cluster according to the resource utilization rate of the target model reasoning service in each cluster; and selecting a target cluster according to whether each cluster is an expandable cluster and the health degree of the target model reasoning service in each cluster, and returning the IP address information of the target cluster to the client.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a cluster resource scheduling method, device, electronic device and storage medium. Background Art

[0002] AI (Artificial Intelligence) model refers to a mathematical model trained through machine learning or deep learning technology, which is used to perform specific reasoning tasks such as image recognition, natural language processing, speech recognition, recommendation system, translation, etc.

[0003] With the development of AI model technology, the demand for AI model reasoning continues to increase. Traditional model reasoning usually relies on a single computing cluster to complete reasoning tasks. When computing resources are scattered across multiple clusters in multiple regions in edge computing service scenarios, there is often a problem of unbalanced scheduling. In addition, edge computing service scenarios not only serve a single model, but also allow multiple models to share computing resources to perform reasoning tasks. Traditional model reasoning is mainly concentrated on single model reasoning or a few model reasoning services. Due to the single model specification, the scheduling of cluster resources is relatively simple. As the number of models increases, the cluster resource utilization rate is low.

[0004] Therefore, how to evenly schedule cluster resources and improve resource utilization is one of the technical problems that need to be solved urgently in existing technologies. Summary of the invention

[0005] In order to solve the problems of unbalanced cluster resource scheduling and low resource utilization, the embodiments of the present application provide a cluster resource scheduling method, device, electronic device and storage medium.

[0006] In a first aspect, an embodiment of the present application provides a cluster resource scheduling method, including:

[0007] Receive a resource scheduling request sent by a client, wherein the resource scheduling request carries domain name information of a target model inference service;

[0008] Acquire a target model identifier according to the domain name of the target model reasoning service, and acquire resource specifications of the target model reasoning service according to the target model identifier;

[0009] Determine whether each cluster is expandable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster, wherein the resource specifications of the target model reasoning service represent the basic hardware specifications required to run the target model reasoning service;

[0010] Obtaining resource utilization of the target model reasoning service in each cluster, and determining the health of the target model reasoning service in the corresponding cluster according to the resource utilization of the target model reasoning service in each cluster;

[0011] A target cluster is selected according to whether each cluster is an expandable cluster and the health of the target model inference service in each cluster, and the network protocol IP address information of the target cluster is returned to the client.

[0012] In one implementation, the target model identifier corresponding to the domain name of the target model inference service is determined according to a preset correspondence between the domain name and the model identifier; or

[0013] The target model identifier is extracted from a specified field of the domain name of the target model inference service.

[0014] In one implementation, the resource scheduling request also includes the IP address information of the client;

[0015] Select the target cluster based on whether each cluster is a scalable cluster and the health of the target model inference service in each cluster, including:

[0016] Obtaining geographical location information to which the IP address of the client belongs and geographical location information to which the IP addresses of each cluster belong;

[0017] A target cluster is selected based on the geographical location information of the IP address of the client, the geographical location information of the IP addresses of each cluster, whether each cluster is an expandable cluster, and the health of the target model inference service in each cluster.

[0018] In one embodiment, the method further comprises:

[0019] Detecting the availability of each cluster according to a preset time period; and

[0020] Before determining whether each cluster can be expanded according to the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster, the method further includes:

[0021] Remove unavailable clusters.

[0022] In one implementation, determining whether each cluster is expandable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster includes:

[0023] For each cluster, if it is determined that the cluster contains a node whose hardware specification is greater than or equal to the target model reasoning service resource specification, then the cluster is determined to be an expandable cluster;

[0024] If it is determined that the hardware specifications of all nodes in the cluster are smaller than the resource specifications of the target model inference service, the cluster is determined to be a non-expandable cluster.

[0025] In one embodiment, the resource utilization rate includes a combination of at least two or more of the following: CPU utilization rate, memory utilization rate, network card bandwidth utilization rate, disk utilization rate, graphics processing unit GPU utilization rate, and GPU video memory utilization rate;

[0026] Determining the health of the target model inference service in the corresponding cluster according to the resource utilization of the target model inference service in each cluster specifically includes:

[0027] For each cluster, a weighted sum is performed on resource utilization rates of the target model reasoning service in the cluster to determine the health of the target model reasoning service in the cluster.

[0028] In one implementation, selecting a target cluster according to the geographical location information to which the IP address of the client belongs, the geographical location information to which the IP addresses of each cluster belong, whether each cluster is a scalable cluster, and the health of the target model inference service in each cluster includes:

[0029] Sorting the clusters according to a regional sorting rule based on the geographical location information to which the IP address of the client belongs and the geographical location information to which the IP addresses of the clusters belong;

[0030] Determine whether each cluster is a healthy cluster of the target model reasoning service according to the health of the target model reasoning service in each cluster and a preset threshold;

[0031] If it is determined that there is an expandable cluster, a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location to which the IP address of the client belongs is selected from the expandable cluster as the target cluster;

[0032] If it is determined that all clusters are non-expandable clusters, then a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location of the client's IP address is selected from the non-expandable clusters as the target cluster.

[0033] In one implementation, if it is determined that there is an expandable cluster, a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location to which the IP address of the client belongs is selected from the expandable cluster as the target cluster, including:

[0034] If it is determined that the scalable cluster includes a healthy target model reasoning service cluster, selecting a healthy target model reasoning service cluster closest to the geographical location to which the IP address of the client belongs from the scalable cluster as the target cluster;

[0035] If it is determined that the scalable cluster does not include a healthy target model reasoning service cluster, selecting an unhealthy target model reasoning service cluster that is closest to the geographical location to which the IP address of the client belongs from the scalable set as the target cluster;

[0036] If it is determined that the target model reasoning service does not exist in any cluster in the scalable cluster, a cluster without the target model reasoning service that is closest to the geographical location of the IP address of the client is selected from the scalable cluster as the target cluster.

[0037] In one implementation, if it is determined that all clusters are non-scalable clusters, a target model reasoning service healthy cluster, a target model reasoning service unhealthy cluster, or a cluster without the target model reasoning service that is closest to the geographical location to which the IP address of the client belongs is selected from the non-scalable cluster as the target cluster, including:

[0038] If it is determined that the non-scalable cluster includes a target model reasoning service healthy cluster, then selecting a target model reasoning service healthy cluster closest to the geographical location to which the IP address of the client belongs from the non-scalable cluster as the target cluster;

[0039] If it is determined that the non-scalable cluster does not include a target model reasoning service healthy cluster, then selecting an unhealthy target model reasoning service cluster that is closest to the geographical location to which the IP address of the client belongs from the non-scalable set as the target cluster;

[0040] If it is determined that the target model reasoning service does not exist in each cluster in the non-scalable cluster, a cluster without the target model reasoning service that is closest to the geographical location of the IP address of the client is selected from the non-scalable cluster as the target cluster.

[0041] In one embodiment, determining whether each cluster is a healthy cluster of the target model reasoning service according to the health of the target model reasoning service in each cluster and a preset threshold, includes:

[0042] For each cluster, if it is determined that each resource utilization rate of the target model reasoning service is less than the corresponding set threshold, and the health of the target model reasoning service in the cluster is less than the preset threshold, then the cluster is determined to be a healthy cluster of the target model reasoning service;

[0043] If it is determined that any resource utilization of the target model reasoning service is greater than or equal to the corresponding set threshold, or the health of the target model reasoning service in the cluster is greater than or equal to the preset threshold, then the cluster is determined to be an unhealthy cluster of the target model reasoning service.

[0044] In a second aspect, an embodiment of the present application provides a cluster resource scheduling device, including:

[0045] A receiving module, used to receive a resource scheduling request sent by a client, wherein the resource scheduling request carries domain name information of a target model inference service;

[0046] An acquisition module, used to acquire a target model identifier according to a domain name of the target model reasoning service, and acquire resource specifications of the target model reasoning service according to the target model identifier;

[0047] A first determination module is used to determine whether each cluster is expandable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster, wherein the resource specifications of the target model reasoning service represent the basic hardware specifications required to run the target model reasoning service;

[0048] A second determination module is used to obtain the resource utilization of the target model reasoning service in each cluster, and determine the health of the target model reasoning service in the corresponding cluster according to the resource utilization of the target model reasoning service in each cluster;

[0049] The scheduling module is used to select a target cluster according to whether each cluster is an expandable cluster and the health of the target model reasoning service in each cluster, and return the network protocol IP address information of the target cluster to the client.

[0050] In one embodiment, the acquisition module is used to determine the target model identifier corresponding to the domain name of the target model inference service based on a preset correspondence between the domain name and the model identifier; or to extract the target model identifier from a specified field of the domain name of the target model inference service.

[0051] In one implementation, the resource scheduling request also includes the IP address information of the client;

[0052] The scheduling module is used to obtain the geographical location information of the IP address of the client and the geographical location information of the IP addresses of each cluster; select the target cluster according to the geographical location information of the IP address of the client, the geographical location information of the IP addresses of each cluster, whether each cluster is an scalable cluster, and the health of the target model inference service in each cluster.

[0053] In one embodiment, the device further comprises:

[0054] A detection module, used to detect the availability of each cluster according to a preset time period;

[0055] The elimination module is used to eliminate unavailable clusters before determining whether each cluster can be expanded based on the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster.

[0056] In one embodiment, the first determination module is used to determine, for each cluster, that the cluster is an expandable cluster if it is determined that the cluster contains nodes whose hardware specifications are greater than or equal to the resource specifications of the target model reasoning service; if it is determined that the hardware specifications of all nodes in the cluster are less than the resource specifications of the target model reasoning service, then the cluster is determined to be a non-expandable cluster.

[0057] In one embodiment, the resource utilization rate includes a combination of at least two or more of the following: CPU utilization rate, memory utilization rate, network card bandwidth utilization rate, disk utilization rate, graphics processing unit GPU utilization rate, and GPU video memory utilization rate;

[0058] The second determination module is used to perform weighted summation of resource utilization rates of the target model reasoning service in each cluster to determine the health of the target model reasoning service in the cluster.

[0059] In one embodiment, the scheduling module is used to sort the clusters according to the regional sorting rules based on the geographic location information of the IP address of the client and the geographic location information of the IP addresses of the clusters; determine whether each cluster is a healthy cluster of the target model reasoning service based on the health of the target model reasoning service in each cluster and a preset threshold; if it is determined that there is an scalable cluster, then select a healthy cluster of the target model reasoning service, an unhealthy cluster of the target model reasoning service, or a cluster without the target model reasoning service that is closest to the geographic location to the IP address of the client from the scalable cluster as the target cluster; if it is determined that each cluster is a non-scalable cluster, then select a healthy cluster of the target model reasoning service, an unhealthy cluster of the target model reasoning service, or a cluster without the target model reasoning service that is closest to the geographic location to the IP address of the client from the non-scalable cluster as the target cluster.

[0060] In one embodiment, the scheduling module is used to select a healthy target model reasoning service cluster from the scalable cluster that is closest to the geographical location to which the IP address of the client belongs as the target cluster if it is determined that the scalable cluster contains a healthy target model reasoning service cluster; if it is determined that the scalable cluster does not contain a healthy target model reasoning service cluster, then select an unhealthy target model reasoning service cluster from the scalable set that is closest to the geographical location to which the IP address of the client belongs as the target cluster; if it is determined that the target model reasoning service does not exist in any cluster in the scalable cluster, then select a cluster from the scalable cluster that is closest to the geographical location to which the IP address of the client does not exist and that does not contain the target model reasoning service as the target cluster.

[0061] In one embodiment, the scheduling module is used to select a healthy target model reasoning service cluster from the non-scalable cluster that is closest to the geographical location to which the IP address of the client belongs as the target cluster if it is determined that the non-scalable cluster contains a healthy target model reasoning service cluster; if it is determined that the non-scalable cluster does not contain a healthy target model reasoning service cluster, then select an unhealthy target model reasoning service cluster from the non-scalable set that is closest to the geographical location to which the IP address of the client belongs as the target cluster; if it is determined that the target model reasoning service does not exist in each cluster in the non-scalable cluster, then select a cluster from the non-scalable cluster that is closest to the geographical location to which the IP address of the client does not exist and that does not contain the target model reasoning service as the target cluster.

[0062] In one embodiment, the scheduling module is used to determine, for each cluster, that if it is determined that the resource utilization rates of the target model reasoning service are all less than their corresponding set thresholds, and the health of the target model reasoning service in the cluster is less than the preset threshold, then the cluster is determined to be a healthy cluster of the target model reasoning service; if it is determined that any resource utilization rate of the target model reasoning service is greater than or equal to the corresponding set threshold, or the health of the target model reasoning service in the cluster is greater than or equal to the preset threshold, then the cluster is determined to be an unhealthy cluster of the target model reasoning service.

[0063] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cluster resource scheduling method described in the present application when executing the program.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the cluster resource scheduling method described in the present application.

[0065] The beneficial effects of this application are as follows:

[0066] The cluster resource scheduling method, device, electronic device and storage medium provided in the embodiments of the present application, the scheduling center receives the resource scheduling request sent by the client, the resource scheduling request carries the domain name information of the target model reasoning service, determines the target model identifier corresponding to the domain name of the target model reasoning service according to the correspondence between the preset domain name and the model identifier, determines whether each cluster is scalable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster, wherein the resource specifications of the target model reasoning service represent the basic hardware specifications required to run the target model reasoning service, and then obtains the resource utilization of the target model reasoning service in each cluster, and determines the health of the target model reasoning service in the corresponding cluster according to the resource utilization of the target model reasoning service in each cluster, selects the target cluster according to whether each cluster is an scalable cluster and the health of the target model reasoning service in each cluster, and returns the IP (Internet Protocol) address of the target cluster to the client. In the embodiment of the present application, the dispatch center pre-sets the correspondence between the domain name and the model identifier of the model inference service. When receiving the resource scheduling request sent by the client, the target model identifier corresponding to the domain name of the target model inference service requested by the client is queried from the preset correspondence between the domain name and the model identifier. Then, the dispatch center compares the pre-set basic hardware specifications required for running the target model inference service with the hardware specifications of the nodes in each cluster managed by the dispatch center to determine whether each cluster is scalable. The dispatch center determines the health of the target model inference service in each cluster according to the resource utilization rate of the target model inference service in each cluster. Thus, according to the scalability status of each cluster and the target model inference service in each cluster, the dispatch center determines the health of the target model inference service in each cluster. The health of the target model reasoning service is selected and assigned to the client. Then, the client can directly request the target model reasoning service from the target cluster according to the IP address of the target cluster. Compared with the prior art, in the embodiment of the present application, based on the model granularity, the health of the target model reasoning service is reflected by the resource utilization of the target model reasoning service in each cluster, rather than considering the health of the cluster from the granularity of the entire cluster, thereby providing a basic basis for the scheduling of multi-model reasoning across multiple clusters. In addition, whether each cluster is scalable and the health of the target model reasoning service in each cluster are considered at the same time, which can better balance different model reasoning services among multiple clusters and improve the cluster resource utilization.

[0067] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0069] Figure 1 A schematic diagram of an application scenario of the cluster resource scheduling method provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of a cluster resource scheduling method provided in an embodiment of the present application;

[0071] Figure 3 A schematic diagram of a process for selecting a target cluster provided in an embodiment of the present application;

[0072] Figure 4 A schematic diagram of a detailed implementation process for selecting a target cluster provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram of the structure of a cluster resource scheduling device provided in an embodiment of the present application;

[0074] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to solve the problems of unbalanced cluster resource scheduling and low resource utilization, the embodiments of the present application provide a cluster resource scheduling method, device, electronic device and storage medium.

[0076] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0077] In this article, it is necessary to understand that the technical terms involved in this application are:

[0078] 1. Model reasoning: The process of using pre-trained machine learning models or deep learning models to perform specific reasoning tasks, such as image recognition, natural language processing, speech recognition, recommendation systems, translation, etc. Large Language Models (LLMs) are deep learning models with a large number of parameters and complex structures. These models can handle tasks such as natural language, text generation, image recognition, translation, etc., and have strong learning and generation capabilities.

[0079] 2. Kubernetes: K8s for short, which is an abbreviation formed by replacing the eight characters "ubernete" in the middle of the name with "8". The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. Kubernetes is an open source container orchestration engine from Google that supports automated deployment, large-scale scalability, and application container management. When deploying an application in a production environment, it is usually necessary to deploy multiple instances of the application in order to load balance application requests. In Kubernetes, multiple containers can be created, each of which runs an application instance. The management, discovery, and access to this group of application instances can be achieved through the set load balancing policy. In this application, Kubernetes technology can be used to manage the cluster, and cluster resources can be scheduled through the resource scheduling policy proposed in this application.

[0080] First reference Figure 1 , which is a schematic diagram of an application scenario of the cluster resource scheduling method provided in an embodiment of the present application, and may include a client 101, a scheduling center 102, and N clusters 103-1 to 103-N, each of which is deployed with an agent, namely: agent 1 to N. The cluster may be deployed, but is not limited to, using Kubernetes technology. The scheduling center 102 is a key component in the Kubernetes cluster, responsible for allocating unscheduled Pods to appropriate nodes. The scheduling center 102 ensures that the Pods can run on nodes that meet resource requirements and constraints through a series of evaluation and screening processes. Each cluster can be deployed to carry different AI model reasoning services. AI model reasoning can be divided into two parts: model files and model reasoning services. The model files include parameter files and configuration files of the AI ​​model. The model reasoning service is used to perform specific reasoning tasks, which is equivalent to an application that performs model reasoning tasks. The model reasoning service can be deployed on the nodes of the cluster using a container image solution. The model file and the model reasoning service are deployed independently. Running the model reasoning service can perform the corresponding model reasoning task. Since model files such as large language model files are often large, in the embodiment of the present application, the model files and model inference services are not packaged together and deployed in the cluster. Instead, the model files are stored in the cloud storage, and then sent from the cloud storage to the cluster local cache. Figure 1As shown, 1031-1 to 1031-N are cached model file collections, which can also be stored in the file system in the cluster, or in a file database, which is not limited in the embodiments of the present application. When the cluster starts an instance to run the model reasoning service, the model file is loaded locally, and the model file can use PVC (Persistent Volume Claim) and the like. The scheduling center 102 predefines the resource specifications of different model reasoning services according to the size of different model files, and stores the correspondence between the model identifier and the resource specifications of the model reasoning service. The resource specifications of the model reasoning service represent the basic hardware specifications required to run the model reasoning service, for example, the CPU (Central Processing Unit) required to run a certain model reasoning service is a CPU with several cores, how much memory is required, etc. In the cluster, the model reasoning service runs elastically. When the resources occupied by the model reasoning service (such as CPU occupancy) exceed the configured threshold and last for a period of time, the scheduling center 102 can expand the model reasoning service, such as adding an instance to run the model reasoning service. When the resources occupied by the model reasoning service are lower than the configured threshold and last for a period of time, the scheduling center 102 can scale down the model reasoning service. When the model reasoning service has not been accessed for a period of time, the scheduling center 102 can scale down the model reasoning service to zero instance.

[0081] The scheduling center 102 sets unique identifiers for different model inference services in advance, and assigns corresponding domain names to different model inference services, and stores the corresponding relationship between the domain name corresponding to each model inference service and the model identifier (that is, the identifier of the model inference service). The scheduling center 102 can also include the identifier of the model inference service in its corresponding domain name. For example, the domain name corresponding to the model inference service can be set to: {model_id}.inference.com, where {model_id} is the model identifier. This embodiment of the present application does not limit this. The scheduling center 102 informs the client 101 of the domain names corresponding to different model inference services in the form of an API (Application Programming Interface) document open to the outside world by the cluster. When the client 101 needs to request any model inference service, it first sends a resource scheduling request to the scheduling center 102, and the scheduling center 102 selects a suitable target cluster and returns the IP address information of the target cluster to the client 101. Then, the client 101 sends a model inference service request to the target cluster based on the IP address information of the target cluster. The agent in each cluster 103 periodically collects the data required for cluster resource scheduling, including: the IP address information of the cluster, the scalability of each model reasoning service cluster, and the resource utilization of each model reasoning service in the cluster, such as CPU occupancy, memory occupancy, network card bandwidth utilization, disk utilization, GPU (Graphics Processing Unit) utilization, and GPU video memory utilization, etc. Each agent uploads the collected data to the scheduling center 102 for storage. The detection point of the scheduling center 102 detects the availability of each cluster according to the preset time period and records it. The preset time period can be set by itself, such as 1 hour or 1 day, or any other duration, which is not limited in the embodiment of the present application.

[0082] Based on the above application scenarios, the following will refer to the attached Figures 2 to 4 The exemplary embodiments of the present application are described in more detail. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in any way. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0083] like Figure 2 As shown, it is a schematic diagram of the implementation process of the cluster resource scheduling method provided in the embodiment of the present application. The cluster resource scheduling method can be applied to the above-mentioned scheduling center 102, and specifically may include the following steps:

[0084] S21. The scheduling center receives a resource scheduling request sent by the client, where the resource scheduling request carries domain name information of the target model inference service.

[0085] In specific implementation, before initiating a model inference service request to the scheduling center, the client first sends a resource scheduling request to the scheduling center. The resource scheduling request carries the domain name information of the requested target model inference service and also includes the client's IP address information.

[0086] S22. The dispatch center obtains the target model identifier according to the domain name of the target model reasoning service, and obtains the resource specifications of the target model reasoning service according to the target model identifier.

[0087] In one implementation, the dispatch center determines the target model identifier corresponding to the domain name of the target model inference service according to a preset correspondence between the domain name and the model identifier.

[0088] In specific implementation, after receiving the resource scheduling request sent by the client, the scheduling center queries the target model identifier corresponding to the domain name of the target model inference service from the correspondence between the domain name of the preset model inference service and the model identifier.

[0089] In one implementation, if the domain name set by the dispatch center for the target model inference service includes the target model identifier, the target model identifier may be extracted from a specified field of the domain name of the target model inference service, wherein the specified field may be preset.

[0090] In specific implementation, if the domain name set by the dispatch center for the target model inference service contains the target model identifier, the domain name of the target model inference service can be parsed to extract the target model identifier from the specified field of the domain name of the target model inference service.

[0091] S23. The dispatch center determines whether each cluster can be expanded based on the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster.

[0092] In specific implementation, the resource specifications of the target model inference service represent the basic hardware specifications required to run the target model inference service. The scheduling center searches for the resource specifications of the target model inference service from the correspondence between the model identifier and the resource specifications of the model inference service according to the target model identifier.

[0093] For each cluster, if it is determined that the cluster contains nodes whose hardware specifications are greater than or equal to the resource specifications of the target model reasoning service, the cluster is determined to be an expandable cluster; if it is determined that the hardware specifications of all nodes in the cluster are less than the resource specifications of the target model reasoning service, the cluster is determined to be a non-expandable cluster.

[0094] Specifically, for each cluster, the scheduling center compares the pre-stored basic hardware specifications (i.e., minimum hardware specifications) required to run the target model reasoning service with the hardware specifications of each node in the cluster. If there is at least one node in the cluster whose hardware specifications are greater than the basic hardware specifications required to run the target model reasoning service, the cluster can be determined to be an expandable cluster. If the hardware specifications of all nodes in the cluster are less than or equal to the basic hardware specifications required to run the target model reasoning service, the cluster can be determined to be a non-expandable cluster. For example, if the CPU required to run the target model reasoning service is at least 2 cores, if the CPU of a node in the cluster has 3 cores (or more), the cluster can be determined to be an expandable cluster. If the CPUs of all nodes in the cluster are less than 2 cores, the cluster can be determined to be a non-expandable cluster. According to the above determination method, it can be determined whether each cluster is an expandable cluster or a non-expandable cluster.

[0095] S24. The scheduling center obtains the resource utilization of the target model inference service in each cluster, and determines the health of the target model inference service in the corresponding cluster according to the resource utilization of the target model inference service in each cluster.

[0096] In the specific implementation, the agent in each cluster obtains the resource utilization of the target model reasoning service from the corresponding cluster, wherein the resource utilization includes at least two or more of the following combinations: CPU occupancy, memory occupancy, network card bandwidth utilization, disk utilization, GPU utilization, and GPU memory utilization, etc., that is, the CPU occupancy, memory occupancy, network card bandwidth utilization, disk utilization, GPU utilization, and GPU memory utilization of the target model reasoning service in each cluster are obtained, and other indicators can be flexibly set by themselves, which is not limited in the embodiment of the present application. Each agent sends the obtained resource utilization of the target model reasoning service in each cluster to the scheduling center. For each cluster, the scheduling center performs a weighted summation of the resource utilization of the target model reasoning service in the cluster to determine the health of the target model reasoning service in the cluster. The health of the target model reasoning service in the cluster represents the health of the target model reasoning service running in the cluster. If the target model reasoning service is not currently running in a certain cluster, it can be determined that the health of the target model reasoning service is 0. The smaller the value of the health of the target model reasoning service in the cluster, the higher the health of the target model reasoning service running in the cluster.

[0097] Specifically, the weights of resource utilizations of different model inference services are different. The scheduling center sets the weights of resource utilizations for different model inference services in advance. When the resource utilization of the target model inference service includes: the CPU occupancy rate, memory occupancy rate, network card bandwidth utilization rate, disk utilization rate, GPU utilization rate, and GPU video memory utilization rate of the target model inference service, for each cluster, the resource utilization rate of the target model inference service in the cluster can be calculated by the following formula:

[0098] L=ω 1 A+ω 2 B+ω 3 C+ω 4 D+ω 5 E+ω 6 F

[0099] Where L represents the resource utilization of the target model inference service in the cluster;

[0100] A represents the CPU usage of the target model inference service in the cluster, ω 1 Indicates the weight of the CPU usage of the target model inference service in the cluster;

[0101] B represents the memory usage of the target model inference service in the cluster, ω 2 The weight representing the memory usage of the target model inference service in the cluster;

[0102] C represents the network card bandwidth usage of the target model inference service in the cluster, ω 3 The weight of the network card bandwidth usage of the target model inference service in the cluster;

[0103] D represents the disk utilization of the target model inference service in the cluster, ω 4 The weight representing the disk utilization of the target model inference service in the cluster;

[0104] E represents the GPU utilization of the target model inference service in the cluster, ω 5 The weight representing the GPU utilization of the target model inference service in the cluster;

[0105] F represents the GPU memory utilization of the target model inference service in the cluster, ω 6 The weight of the GPU memory utilization of the target model inference service in the cluster;

[0106] ω 1 +ω 2 +ω 3 +ω 4 +ω 5 +ω 6 =1.

[0107] According to the above formula, the health of the target model inference service in each cluster can be calculated.

[0108] During implementation, the scheduling center can pre-set the weights of various resource utilization indicators according to the types of different model inference services. If one or more resource utilization indicators do not need to be considered for a certain model inference service, the weights of one or more resource utilization indicators can be set to 0. For example, if the memory usage of the target model inference service in the cluster does not need to be considered for the target model inference service, its weight ω can be set to 0. 2 Set to 0.

[0109] In one embodiment, in order to improve the efficiency and accuracy of cluster resource scheduling, the scheduling center may detect the availability of each cluster according to a preset time period. Before determining whether each cluster can be expanded based on the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster, it is necessary to eliminate unavailable clusters and only determine whether available clusters can be expanded.

[0110] S25. The dispatch center selects a target cluster based on whether each cluster is an expandable cluster and the health of the target model inference service in each cluster, and returns the IP address information of the target cluster to the client.

[0111] When implementing it specifically, you can follow the following steps: Figure 3 The process shown selects the target cluster, including the following steps:

[0112] S31. The dispatch center obtains the geographical location information to which the IP address of the client belongs and the geographical location information to which the IP addresses of each cluster belong.

[0113] During specific implementation, the dispatch center may query the geographical location information to which the IP address of the client belongs and the geographical location information to which the IP addresses of each cluster belong from a public or private IP library, wherein the IP library contains the geographical location information of the IP address.

[0114] In order to improve scheduling efficiency, the scheduling center can also pre-query the geographic location information of the IP address corresponding to the domain name of each model service and the geographic location information of the IP address of each cluster, and store the geographic location information of the IP address corresponding to the domain name of each model service and the geographic location information of the IP address of each cluster in the database. When cluster resource scheduling is required, it can be directly obtained from the database.

[0115] S32. The dispatch center selects a target cluster based on the geographical location information of the client's IP address, the geographical location information of each cluster's IP address, whether each cluster is a scalable cluster, and the health of the target model inference service in each cluster.

[0116] In implementation, the following Figure 4 The process shown in Figure 1 selects the target cluster and may include the following steps:

[0117] S41 . The dispatch center sorts the clusters according to the regional sorting rule based on the geographical location information to which the IP address of the client belongs and the geographical location information to which the IP addresses of the clusters belong.

[0118] During specific implementation, after the dispatch center obtains the geographical location information of the client's IP address and the geographical location information of each cluster's IP address, it sorts each cluster according to the regional sorting rules. The regional sorting rules can be divided into domestic regional sorting rules and overseas regional sorting rules. The domestic regional sorting rules include: sorting can be carried out step by step according to the rules of same province > same region > same operator > cross-operator. Among them, the regions include: South China, Central China, North China, Northeast, East China, Southwest, and Northwest. The overseas regional sorting rules include: same country and same operator > same country > same continent. During implementation, it is not limited to the above regional sorting rules, and can be set according to the actual situation such as the number of clusters and geographical location.

[0119] S42. The dispatch center determines whether each cluster is a healthy cluster for the target model reasoning service according to the health of the target model reasoning service in each cluster and a preset threshold.

[0120] During specific implementation, for each cluster, if the scheduling center determines that the resource utilization rates of the target model reasoning service are all less than their corresponding set thresholds, and the health of the target model reasoning service in the cluster is less than the preset threshold, then the cluster is determined to be a healthy cluster of the target model reasoning service; if it is determined that any resource utilization rate of the target model reasoning service is greater than or equal to the corresponding set threshold, or the health of the target model reasoning service in the cluster is greater than or equal to the preset threshold, then the cluster is determined to be an unhealthy cluster of the target model reasoning service.

[0121] Specifically, the scheduling center pre-sets the set threshold corresponding to each single resource utilization indicator, such as setting the set threshold corresponding to the CPU occupancy rate to 70%, setting the set threshold corresponding to the memory occupancy rate to 60%, setting the set threshold corresponding to the network card bandwidth usage rate to 75%, and setting the set threshold corresponding to the disk utilization rate to 60%. During implementation, it can be set according to needs, and the embodiments of the present application are not limited to this.

[0122] When judging whether a cluster is a healthy cluster of target model reasoning service, two dimensions need to be considered comprehensively: one is whether a single resource utilization index is normal, and the other is whether the comprehensive resource utilization index is normal. For each cluster, if the utilization rates of each single resource of the target model reasoning service are less than their corresponding set thresholds, that is, the CPU occupancy rate of the target model reasoning service in the cluster is less than the corresponding set threshold (such as 70%), the memory occupancy rate of the target model reasoning service in the cluster is less than the corresponding set threshold (such as 60%), the network card bandwidth utilization rate of the target model reasoning service in the cluster is less than the corresponding set threshold (such as 75%), and the disk utilization rate of the target model reasoning service in the cluster is less than the corresponding set threshold (such as 60%), and the health of the target model reasoning service in the cluster is less than the preset threshold, then the cluster can be determined to be a healthy cluster of the target model reasoning service. When any resource utilization rate of the target model reasoning service is greater than or equal to the corresponding set threshold, such as the CPU occupancy rate of the target model reasoning service in the cluster is 80%, which is greater than the set threshold of 70%, then the cluster is determined to be an unhealthy cluster of the target model reasoning service, or, if the health of the target model reasoning service in the cluster is greater than or equal to the preset threshold, then the cluster is determined to be an unhealthy cluster of the target model reasoning service. The preset threshold value may be set according to the demand, such as 70%, which is not limited in the embodiment of the present application.

[0123] In the embodiments of the present application, based on the model granularity, the health status of the model reasoning service is reflected through single indicators and comprehensive indicators, rather than considering the health value of the cluster from the granularity of the entire cluster, thereby providing a basic basis for cluster resource scheduling across multi-cluster multi-model reasoning services.

[0124] S43. If the dispatch center determines that there is an expandable cluster, it selects a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without a target model reasoning service that is closest to the geographical location of the client's IP address from the expandable cluster as the target cluster.

[0125] During specific implementation, if the dispatch center determines that the scalable cluster contains a healthy cluster of target model reasoning services, then a healthy cluster of target model reasoning services that is closest to the geographical location of the client's IP address is selected from the scalable cluster as the target cluster; if it is determined that the scalable cluster does not contain a healthy cluster of target model reasoning services, then an unhealthy cluster of target model reasoning services that is closest to the geographical location of the client's IP address is selected from the scalable set as the target cluster; if it is determined that none of the clusters in the scalable cluster has a target model reasoning service, then a cluster without a target model reasoning service that is closest to the geographical location of the client's IP address is selected from the scalable cluster as the target cluster.

[0126] S44. If the dispatch center determines that all clusters are non-scalable clusters, it selects a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without a target model reasoning service that is closest to the geographical location of the client's IP address from the non-scalable clusters as the target cluster.

[0127] During specific implementation, if the dispatch center determines that the non-scalable cluster contains a healthy cluster of target model reasoning services, then a healthy cluster of target model reasoning services that is closest to the geographical location of the client's IP address is selected from the non-scalable cluster as the target cluster; if it is determined that the non-scalable cluster does not contain a healthy cluster of target model reasoning services, then an unhealthy cluster of target model reasoning services that is closest to the geographical location of the client's IP address is selected from the non-scalable set as the target cluster; if it is determined that none of the clusters in the non-scalable cluster has a target model reasoning service, then a cluster that does not have a target model reasoning service that is closest to the geographical location of the client's IP address is selected from the non-scalable cluster as the target cluster.

[0128] Then, the dispatch center returns the IP address information of the selected target cluster to the client, and the client can then directly send a model inference service request to the target cluster based on the IP address information of the target cluster.

[0129] During implementation, when all clusters are non-scalable clusters, if the non-scalable cluster does not contain a healthy cluster of the target model reasoning service, in order to ensure the normal operation of the cluster service, the scheduling center can also set a strategy in advance for the cluster that only contains an unhealthy cluster of the target model reasoning service or does not have a target model reasoning service, and make a selection based on the strategy, such as setting a strategy: try to serve, or avoid overload. This is because the cluster service is no longer scalable. Selecting an unhealthy cluster of the target model reasoning service or a cluster without a target model reasoning service from a non-scalable cluster will lead to deteriorated service or even no service. When the strategy is to serve as much as possible, you can select an unhealthy cluster of the target model reasoning service that is closest to the geographical location of the client's IP address as the target cluster, or select a cluster without a target model reasoning service that is closest to the geographical location of the client's IP address as the target cluster. When the strategy is to avoid overload, the target cluster is not selected from the unhealthy cluster of the target model reasoning service or the cluster without a target model reasoning service, and the scheduling failure message is directly returned to the client, and the client can be prompted to wait for a certain period of time to initiate a resource scheduling application again.

[0130] In an embodiment of the present application, based on multi-dimensional cluster resource scheduling such as the client region, cluster availability, whether the model reasoning service can be expanded, and the health value of the model reasoning service, the computing resources of different model reasoning services can be better balanced among multiple clusters.

[0131] The cluster resource scheduling method provided in the embodiment of the present application comprises the following steps: a scheduling center receives a resource scheduling request sent by a client, wherein the resource scheduling request carries domain name information of a target model reasoning service, determines the target model identifier corresponding to the domain name of the target model reasoning service according to the correspondence between the preset domain name and the model identifier, determines whether each cluster is scalable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster, wherein the resource specifications of the target model reasoning service represent the basic hardware specifications required to run the target model reasoning service, and then obtains the resource utilization of the target model reasoning service in each cluster, determines the health of the target model reasoning service in the corresponding cluster according to the resource utilization of the target model reasoning service in each cluster, selects the target cluster according to whether each cluster is an scalable cluster and the health of the target model reasoning service in each cluster, and returns the IP address of the target cluster to the client. In the embodiment of the present application, the dispatch center pre-sets the correspondence between the domain name and the model identifier of the model inference service. When receiving the resource scheduling request sent by the client, the target model identifier corresponding to the domain name of the target model inference service requested by the client is queried from the preset correspondence between the domain name and the model identifier. Then, the dispatch center compares the pre-set basic hardware specifications required for running the target model inference service with the hardware specifications of the nodes in each cluster managed by the dispatch center to determine whether each cluster is scalable. The dispatch center determines the health of the target model inference service in each cluster according to the resource utilization rate of the target model inference service in each cluster. Thus, according to the scalability status of each cluster and the target model inference service in each cluster, the dispatch center determines the health of the target model inference service in each cluster. The health of the target model reasoning service is selected and assigned to the client. Then, the client can directly request the target model reasoning service from the target cluster according to the IP address of the target cluster. Compared with the prior art, in the embodiment of the present application, based on the model granularity, the health of the target model reasoning service is reflected by the resource utilization of the target model reasoning service in each cluster, rather than considering the health of the cluster from the granularity of the entire cluster, thereby providing a basic basis for the scheduling of multi-model reasoning across multiple clusters. In addition, whether each cluster is scalable and the health of the target model reasoning service in each cluster are considered at the same time, which can better balance different model reasoning services among multiple clusters and improve the cluster resource utilization.

[0132] Based on the same inventive concept, the embodiment of the present application further provides a cluster resource scheduling device. Since the principle of solving the problem by the above cluster resource scheduling device is similar to that of the above cluster resource scheduling method, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0133] like Figure 5 As shown, it is a schematic diagram of the structure of the cluster resource scheduling device provided in an embodiment of the present application. The cluster resource scheduling device can be applied to Figure 1 The dispatch center 102 shown may include:

[0134] The receiving module 51 is used to receive a resource scheduling request sent by a client, wherein the resource scheduling request carries domain name information of a target model inference service;

[0135] An acquisition module 52, configured to acquire a target model identifier according to a domain name of the target model reasoning service, and acquire resource specifications of the target model reasoning service according to the target model identifier;

[0136] A first determination module 53 is used to determine whether each cluster is expandable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster, wherein the resource specifications of the target model reasoning service represent the basic hardware specifications required to run the target model reasoning service;

[0137] A second determination module 54 is used to obtain the resource utilization of the target model reasoning service in each cluster, and determine the health of the target model reasoning service in the corresponding cluster according to the resource utilization of the target model reasoning service in each cluster;

[0138] The scheduling module 55 is used to select a target cluster according to whether each cluster is an expandable cluster and the health of the target model reasoning service in each cluster, and return the network protocol IP address information of the target cluster to the client.

[0139] In one embodiment, the acquisition module 52 is used to determine the target model identifier corresponding to the domain name of the target model inference service based on the correspondence between the preset domain name and the model identifier; or to extract the target model identifier from a specified field of the domain name of the target model inference service.

[0140] In one implementation, the resource scheduling request also includes the IP address information of the client;

[0141] The scheduling module 55 is used to obtain the geographic location information of the IP address of the client and the geographic location information of the IP addresses of each cluster; select the target cluster according to the geographic location information of the IP address of the client, the geographic location information of the IP addresses of each cluster, whether each cluster is an expandable cluster, and the health of the target model inference service in each cluster.

[0142] In one embodiment, the device further comprises:

[0143] A detection module, used to detect the availability of each cluster according to a preset time period;

[0144] The elimination module is used to eliminate unavailable clusters before determining whether each cluster can be expanded based on the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster.

[0145] In one embodiment, the first determination module 53 is used to, for each cluster, determine that the cluster is a scalable cluster if it is determined that the cluster contains nodes whose hardware specifications are greater than or equal to the resource specifications of the target model reasoning service; if it is determined that the hardware specifications of all nodes in the cluster are less than the resource specifications of the target model reasoning service, determine that the cluster is a non-scalable cluster.

[0146] In one embodiment, the resource utilization rate includes a combination of at least two or more of the following: CPU utilization rate, memory utilization rate, network card bandwidth utilization rate, disk utilization rate, graphics processing unit GPU utilization rate, and GPU video memory utilization rate;

[0147] The second determination module 54 is used to perform weighted summation of resource utilization rates of the target model reasoning service in each cluster to determine the health of the target model reasoning service in the cluster.

[0148] In one embodiment, the scheduling module 55 is used to sort the clusters according to the regional sorting rules based on the geographic location information of the IP address of the client and the geographic location information of the IP addresses of the clusters; determine whether each cluster is a healthy cluster of the target model reasoning service based on the health of the target model reasoning service in each cluster and a preset threshold; if it is determined that there is an scalable cluster, then select a healthy cluster of the target model reasoning service, an unhealthy cluster of the target model reasoning service, or a cluster where the target model reasoning service does not exist, which is closest to the geographic location of the IP address of the client, from the scalable cluster as the target cluster; if it is determined that each cluster is a non-scalable cluster, then select a healthy cluster of the target model reasoning service, an unhealthy cluster of the target model reasoning service, or a cluster where the target model reasoning service does not exist, which is closest to the geographic location of the IP address of the client, from the non-scalable cluster as the target cluster.

[0149] In one embodiment, the scheduling module 55 is used to select a healthy target model reasoning service cluster from the scalable cluster that is closest to the geographical location to which the IP address of the client belongs as the target cluster if it is determined that the scalable cluster includes a healthy target model reasoning service cluster; if it is determined that the scalable cluster does not include a healthy target model reasoning service cluster, then select an unhealthy target model reasoning service cluster from the scalable set that is closest to the geographical location to which the IP address of the client belongs as the target cluster; if it is determined that the target model reasoning service does not exist in any cluster in the scalable cluster, then select a cluster from the scalable cluster that is closest to the geographical location to which the IP address of the client does not exist and that does not contain the target model reasoning service as the target cluster.

[0150] In one embodiment, the scheduling module 55 is used to select a healthy target model reasoning service cluster from the non-scalable cluster that is closest to the geographical location to which the IP address of the client belongs as the target cluster if it is determined that the non-scalable cluster contains a healthy target model reasoning service cluster; if it is determined that the non-scalable cluster does not contain a healthy target model reasoning service cluster, select an unhealthy target model reasoning service cluster from the non-scalable set that is closest to the geographical location to which the IP address of the client belongs as the target cluster; if it is determined that the target model reasoning service does not exist in each cluster in the non-scalable cluster, select a cluster from the non-scalable cluster that is closest to the geographical location to which the IP address of the client does not exist and that does not contain the target model reasoning service as the target cluster.

[0151] In one embodiment, the scheduling module 55 is used to, for each cluster, determine that if it is determined that the resource utilization rates of the target model reasoning service are all less than their corresponding set thresholds, and the health of the target model reasoning service in the cluster is less than the preset threshold, then determine that the cluster is a healthy cluster of the target model reasoning service; if it is determined that any resource utilization rate of the target model reasoning service is greater than or equal to the corresponding set threshold, or the health of the target model reasoning service in the cluster is greater than or equal to the preset threshold, then determine that the cluster is an unhealthy cluster of the target model reasoning service.

[0152] Based on the same technical concept, the present application embodiment also provides an electronic device 600, referring to Figure 6As shown, the electronic device 600 is used to implement the cluster resource scheduling method described in the above method embodiment. The electronic device 600 of this embodiment may include: a memory 601, a processor 602, and a computer program stored in the memory and executable on the processor, such as a cluster resource scheduling program. When the processor executes the computer program, the steps in each of the above cluster resource scheduling method embodiments are implemented.

[0153] The specific connection medium between the memory 601 and the processor 602 is not limited in the embodiment of the present application. Figure 6 In the embodiment, the memory 601 and the processor 602 are connected via a bus 603. The bus 603 is Figure 6 The connection between other components is shown by bold lines, which is only for schematic illustration and is not intended to be limiting. The bus 603 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0154] The memory 601 may be a volatile memory, such as a random-access memory (RAM); the memory 601 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 601 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 601 may be a combination of the above memories.

[0155] The processor 602 is used to implement the cluster resource scheduling method provided in the embodiment of the present application.

[0156] An embodiment of the present application also provides a computer-readable storage medium that stores computer-executable instructions required to execute the above-mentioned processor, which includes a program required to execute the above-mentioned processor.

[0157] In some possible implementations, various aspects of the cluster resource scheduling method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the cluster resource scheduling method according to various exemplary implementations of the present application described above in this specification.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0163] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cluster resource scheduling method, characterized in that: include: Receive a resource scheduling request sent by a client, wherein the resource scheduling request carries domain name information of a target model inference service; Acquire a target model identifier according to the domain name of the target model reasoning service, and acquire resource specifications of the target model reasoning service according to the target model identifier; Determine whether each cluster is expandable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster, wherein the resource specifications of the target model reasoning service represent the basic hardware specifications required to run the target model reasoning service; Obtaining resource utilization of the target model reasoning service in each cluster, and determining the health of the target model reasoning service in the corresponding cluster according to the resource utilization of the target model reasoning service in each cluster; A target cluster is selected according to whether each cluster is an expandable cluster and the health of the target model inference service in each cluster, and the network protocol IP address information of the target cluster is returned to the client.

2. The method according to claim 1, characterized in that Determine the target model identifier corresponding to the domain name of the target model inference service according to a preset correspondence between the domain name and the model identifier; or The target model identifier is extracted from a specified field of the domain name of the target model inference service.

3. The method according to claim 1, characterized in that The resource scheduling request also includes the IP address information of the client; then Select the target cluster based on whether each cluster is a scalable cluster and the health of the target model inference service in each cluster, including: Obtaining geographical location information to which the IP address of the client belongs and geographical location information to which the IP addresses of each cluster belong; A target cluster is selected based on the geographical location information of the IP address of the client, the geographical location information of the IP addresses of each cluster, whether each cluster is an expandable cluster, and the health of the target model inference service in each cluster.

4. The method according to claim 3, characterized in that Also includes: Detecting the availability of each cluster according to a preset time period; as well as Before determining whether each cluster can be expanded according to the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster, the method further includes: Remove unavailable clusters.

5. The method according to claim 3, characterized in that Determine whether each cluster can be expanded based on the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster, including: For each cluster, if it is determined that the cluster contains a node whose hardware specification is greater than or equal to the target model reasoning service resource specification, then the cluster is determined to be an expandable cluster; If it is determined that the hardware specifications of all nodes in the cluster are smaller than the resource specifications of the target model inference service, the cluster is determined to be a non-expandable cluster.

6. The method according to claim 1, characterized in that The resource utilization rate includes a combination of at least two or more of the following: CPU utilization rate, memory utilization rate, network card bandwidth utilization rate, disk utilization rate, graphics processing unit GPU utilization rate, and GPU video memory utilization rate; Determining the health of the target model inference service in the corresponding cluster according to the resource utilization of the target model inference service in each cluster specifically includes: For each cluster, a weighted sum is performed on resource utilization rates of the target model reasoning service in the cluster to determine the health of the target model reasoning service in the cluster.

7. The method according to any one of claims 3 to 6, characterized in that: Selecting a target cluster according to the geographical location information to which the IP address of the client belongs, the geographical location information to which the IP addresses of each cluster belong, whether each cluster is an expandable cluster, and the health of the target model inference service in each cluster includes: Sorting the clusters according to a regional sorting rule based on the geographical location information to which the IP address of the client belongs and the geographical location information to which the IP addresses of the clusters belong; Determine whether each cluster is a healthy cluster of the target model reasoning service according to the health of the target model reasoning service in each cluster and a preset threshold; If it is determined that there is an expandable cluster, a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location to which the IP address of the client belongs is selected from the expandable cluster as the target cluster; If it is determined that all clusters are non-expandable clusters, then a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location of the client's IP address is selected from the non-expandable clusters as the target cluster.

8. The method according to claim 7, characterized in that If it is determined that there is an expandable cluster, a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location to which the IP address of the client belongs is selected from the expandable cluster as the target cluster, including: If it is determined that the scalable cluster includes a healthy target model reasoning service cluster, selecting a healthy target model reasoning service cluster closest to the geographical location to which the IP address of the client belongs from the scalable cluster as the target cluster; If it is determined that the scalable cluster does not include a healthy target model reasoning service cluster, selecting an unhealthy target model reasoning service cluster closest to the geographical location to which the IP address of the client belongs from the scalable set as the target cluster; If it is determined that the target model reasoning service does not exist in each cluster in the scalable cluster, a cluster without the target model reasoning service that is closest to the geographical location of the IP address of the client is selected from the scalable cluster as the target cluster.

9. The method according to claim 7, characterized in that If it is determined that all clusters are non-scalable clusters, a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location to which the IP address of the client belongs is selected from the non-scalable clusters as the target cluster, including: If it is determined that the non-scalable cluster includes a target model reasoning service healthy cluster, then selecting a target model reasoning service healthy cluster closest to the geographical location to which the IP address of the client belongs from the non-scalable cluster as the target cluster; If it is determined that the non-scalable cluster does not include a target model reasoning service healthy cluster, then selecting an unhealthy target model reasoning service cluster that is closest to the geographical location to which the IP address of the client belongs from the non-scalable set as the target cluster; If it is determined that the target model reasoning service does not exist in each cluster in the non-scalable cluster, a cluster without the target model reasoning service that is closest to the geographical location of the IP address of the client is selected from the non-scalable cluster as the target cluster.

10. The method according to claim 7, characterized in that Determining whether each cluster is a healthy cluster of the target model reasoning service according to the health of the target model reasoning service in each cluster and a preset threshold, including: For each cluster, if it is determined that each resource utilization rate of the target model reasoning service is less than the corresponding set threshold, and the health of the target model reasoning service in the cluster is less than the preset threshold, then the cluster is determined to be a healthy cluster of the target model reasoning service; If it is determined that any resource utilization of the target model reasoning service is greater than or equal to the corresponding set threshold, or the health of the target model reasoning service in the cluster is greater than or equal to the preset threshold, then the cluster is determined to be an unhealthy cluster of the target model reasoning service.

11. A cluster resource scheduling device, characterized in that: include: A receiving module, used to receive a resource scheduling request sent by a client, wherein the resource scheduling request carries domain name information of a target model inference service; An acquisition module, used to acquire a target model identifier according to a domain name of the target model reasoning service, and acquire resource specifications of the target model reasoning service according to the target model identifier; A first determination module is used to determine whether each cluster is expandable according to the resource specifications of the target model reasoning service and the hardware specifications of the nodes in each cluster, wherein the resource specifications of the target model reasoning service represent the basic hardware specifications required to run the target model reasoning service; A second determination module is used to obtain the resource utilization of the target model reasoning service in each cluster, and determine the health of the target model reasoning service in the corresponding cluster according to the resource utilization of the target model reasoning service in each cluster; The scheduling module is used to select a target cluster according to whether each cluster is an expandable cluster and the health of the target model reasoning service in each cluster, and return the network protocol IP address information of the target cluster to the client.

12. The device according to claim 11, characterized in that The acquisition module is used to determine the target model identifier corresponding to the domain name of the target model inference service according to the preset correspondence between the domain name and the model identifier; or to extract the target model identifier from a specified field of the domain name of the target model inference service.

13. The device according to claim 11, characterized in that The resource scheduling request also includes the IP address information of the client; The scheduling module is used to obtain the geographical location information to which the IP address of the client belongs and the geographical location information to which the IP addresses of each cluster belong; A target cluster is selected based on the geographical location information to which the IP address of the client belongs, the geographical location information to which the IP addresses of each cluster belong, whether each cluster is an expandable cluster, and the health of the target model inference service in each cluster.

14. The device according to claim 13, characterized in that Also includes: A detection module, used to detect the availability of each cluster according to a preset time period; The elimination module is used to eliminate unavailable clusters before determining whether each cluster can be expanded based on the resource specifications of the target model inference service and the hardware specifications of the nodes in each cluster.

15. The device according to claim 13, characterized in that The first determination module is used to determine, for each cluster, that the cluster is an expandable cluster if it is determined that the cluster contains a node whose hardware specification is greater than or equal to the target model reasoning service resource specification; If it is determined that the hardware specifications of all nodes in the cluster are smaller than the resource specifications of the target model inference service, the cluster is determined to be a non-expandable cluster.

16. The device according to claim 11, characterized in that The resource utilization rate includes a combination of at least two or more of the following: CPU utilization rate, memory utilization rate, network card bandwidth utilization rate, disk utilization rate, graphics processor GPU utilization rate, and GPU video memory utilization rate; The second determination module is used to perform weighted summation of resource utilization rates of the target model reasoning service in each cluster to determine the health of the target model reasoning service in the cluster.

17. The device according to any one of claims 13 to 16, characterized in that: The scheduling module is used to sort the clusters according to the regional sorting rule according to the geographical location information to which the IP address of the client belongs and the geographical location information to which the IP addresses of the clusters belong; Determine whether each cluster is a healthy cluster of the target model reasoning service according to the health of the target model reasoning service in each cluster and a preset threshold; If it is determined that there is an expandable cluster, a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location to which the IP address of the client belongs is selected from the expandable cluster as the target cluster; If it is determined that all clusters are non-expandable clusters, then a healthy target model reasoning service cluster, an unhealthy target model reasoning service cluster, or a cluster without the target model reasoning service that is closest to the geographical location of the client's IP address is selected from the non-expandable clusters as the target cluster.

18. The device according to claim 17, characterized in that The scheduling module is used for selecting a healthy target model reasoning service cluster from the scalable cluster that is closest to the geographical location to which the IP address of the client belongs as a target cluster if it is determined that the scalable cluster includes a healthy target model reasoning service cluster; If it is determined that the scalable cluster does not include a healthy target model reasoning service cluster, selecting an unhealthy target model reasoning service cluster that is closest to the geographical location to which the IP address of the client belongs from the scalable set as the target cluster; If it is determined that the target model reasoning service does not exist in each cluster in the scalable cluster, a cluster without the target model reasoning service that is closest to the geographical location of the IP address of the client is selected from the scalable cluster as the target cluster.

19. The device according to claim 17, characterized in that The scheduling module is used for selecting a healthy target model reasoning service cluster from the non-scalable cluster, which is closest to the geographical location to which the IP address of the client belongs, as a target cluster if it is determined that the non-scalable cluster includes a healthy target model reasoning service cluster; If it is determined that the non-scalable cluster does not include a target model reasoning service healthy cluster, then selecting an unhealthy target model reasoning service cluster that is closest to the geographical location to which the IP address of the client belongs from the non-scalable set as the target cluster; If it is determined that the target model reasoning service does not exist in each cluster in the non-scalable cluster, a cluster without the target model reasoning service that is closest to the geographical location of the IP address of the client is selected from the non-scalable cluster as the target cluster.

20. The device according to claim 17, characterized in that The scheduling module is used to determine, for each cluster, that if it is determined that the resource utilization rates of the target model reasoning service are all less than their corresponding set thresholds, and the health of the target model reasoning service in the cluster is less than the preset threshold, then the cluster is determined to be a healthy cluster of the target model reasoning service; if it is determined that any resource utilization rate of the target model reasoning service is greater than or equal to the corresponding set threshold, or the health of the target model reasoning service in the cluster is greater than or equal to the preset threshold, then the cluster is determined to be an unhealthy cluster of the target model reasoning service.

21. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the cluster resource scheduling method according to any one of claims 1 to 10 is implemented.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the cluster resource scheduling method according to any one of claims 1 to 10 are implemented.

Images