Large model scheduling system and method, server, medium and product

Abstract

The invention discloses a large model scheduling system and method, a server, a medium and a product. The large model scheduling system comprises a model management module, a computing power scheduling module and a model scheduling module, the model management module is used for performing corresponding model management on a target large model in response to a management operation of the client for the target large model; the computing power scheduling module is used for determining a target computing power resource node matched with the large model according to model parameter information of the large model for each large model, and mounting the large model to a container on the target computing power resource node; and the model scheduling module is used for acquiring the container parameter information of each container, and performing model scheduling processing on the large model mounted on the abnormal container when the container parameter information is abnormal. According to the technical scheme, the computing power resource utilization rate is improved, and efficient computing power resource selection and large model scheduling are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large model scheduling system, method, server, medium and product. Background Art

[0002] Artificial Intelligence (AI) is an emerging technical science that involves theories, methods, technologies and application systems for human intelligence research. It mainly studies and develops technologies for simulating, extending and expanding human intelligence. AI technology is gradually changing modern society. Intelligent science and technology, that is, a deep understanding of the mechanism of intelligence combined with advanced AI technology, has become a key driving force for the sustainable development of artificial intelligence. Large-scale pre-trained language models, also known as "base models" or "large models", are characterized by a huge number of parameters that constitute a complex artificial neural network model. The proposal and practice of this technology is of epoch-making significance, marking that the research on artificial intelligence has entered the era of general artificial intelligence.

[0003] At present, various organizations have a large amount of heterogeneous computing resources, and there is an urgent need for a unified scheduling system to utilize these computing resources to support the operation of large-model reasoning applications. Summary of the invention

[0004] The present invention provides a large model scheduling system, method, server, medium and product to improve the utilization rate of computing power resources and realize efficient computing power resource selection and large model scheduling.

[0005] In a first aspect, an embodiment of the present disclosure provides a large model scheduling system, including: a model management module, a computing power scheduling module and a model scheduling module;

[0006] The model management module is used to respond to the management operation of the client on the target large model and perform corresponding model management on the target large model;

[0007] The computing power scheduling module is used to determine, for each large model, a target computing power resource node matching the large model according to the model parameter information of the large model, and mount the large model to a container on the target computing power resource node;

[0008] The model scheduling module is used to obtain container parameter information of each container, and when there is an abnormality in the container parameter information, perform model scheduling processing on the large model mounted on the abnormal container.

[0009] In a second aspect, the present disclosure provides a large model scheduling method, including:

[0010] Responding to the management operation of the client on the target large model through the model management module, performing corresponding model management on the target large model;

[0011] Through the computing power scheduling module, for each large model, a target computing power resource node matching the large model is determined according to the model parameter information of the large model, and the large model is mounted to a container on the target computing power resource node;

[0012] The container parameter information of each container is obtained through the model scheduling module. When there is an abnormality in the container parameter information, the model scheduling process is performed on the large model mounted on the abnormal container.

[0013] In a third aspect, an embodiment of the present disclosure provides a large model scheduling server, integrating the system described in the first aspect, wherein the large model scheduling server is connected to multiple computing servers through a switch, and is connected to a monitoring server through a switch, and the server includes:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the large model scheduling method provided by the above-mentioned second aspect embodiment.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the large model scheduling method provided in the embodiment of the second aspect above when executed.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the large model scheduling method provided by the embodiment of the second aspect above.

[0019] A large model scheduling system, method, server, medium and product according to an embodiment of the present invention, the system includes: a model management module, a computing power scheduling module and a model scheduling module; the model management module is used to respond to the client's management operation on the target large model and perform corresponding model management on the target large model; the computing power scheduling module is used to determine the target computing power resource node that matches the large model according to the model parameter information of the large model for each large model, and mount the large model to the container on the target computing power resource node; the model scheduling module is used to obtain the container parameter information of each container, and when there is an abnormality in the container parameter information, perform model scheduling processing on the large model mounted on the abnormal container. The above technical solution uses computing power resources to support the operation of large model reasoning applications, improve the utilization rate of computing power resources, and achieve efficient computing power resource selection and large model scheduling.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a structural diagram of a large model scheduling system provided by Embodiment 1 of the present invention;

[0023] Figure 2 This is an interactive schematic diagram of a large model scheduling system provided by the first embodiment of the present invention;

[0024] Figure 3 is a flow chart of a large model scheduling method provided by Embodiment 2 of the present invention;

[0025] Figure 4 It is a structural diagram of a large model scheduling server provided in Embodiment 3 of the present invention;

[0026] Figure 5 This is a deployment diagram of a large model scheduling server provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second" and "target" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] In recent years, containers and Kubernetes have become the preferred operating environment and platform for more and more AI applications. On the one hand, Kubernetes helps users standardize heterogeneous resources and runtime environments and simplify operation and maintenance processes; on the other hand, AI scenarios that rely heavily on graphics processing units (GPUs) can take advantage of Kubernetes' elasticity to save resource costs. Therefore, under the wave of AI-generated content (AIGC) / large models, this application uses Kubernetes to run large models, thereby effectively alleviating the current GPU shortage problem, and needs to improve the utilization of existing resources, and enable heterogeneous computing resources as much as possible according to different large model reasoning applications.

[0030] Embodiment 1

[0031] Figure 1 It is a structural diagram of a large model scheduling system provided in Example 1 of the present invention. This embodiment can be applied to the situation of large model monitoring and execution scheduling based on the Kubernetes platform.

[0032] like Figure 1 As shown, the large model scheduling system 1 includes: a model management module 11, a computing power scheduling module 12 and a model scheduling module 13;

[0033] The model management module 11 is used to respond to the management operation of the client on the target large model and perform corresponding model management on the target large model;

[0034] The computing power scheduling module 12 is used to determine the target computing power resource node matching the target large model according to the model parameter information of the target large model, and mount the target large model to the container on the target computing power resource node;

[0035] The model scheduling module 13 is used to obtain the container parameter information of each container, and when there is an abnormality in the container parameter information, perform model scheduling processing on the large model mounted on the abnormal container.

[0036] In this embodiment, the Kubernetes platform is a platform that provides support for the operation of the large model, and supports at least one large model scheduling server, multiple computing servers, and one monitoring server. The large model scheduling server integrates the large model scheduling system 1, and the large model uploaded by the large model scheduling system 1 is stored in each computing server. The target large model is the large model selected by the client, which can be a large model specified by the client and stored in the computing server, or a new large model specified by the client to be uploaded and imported. The model parameter information can be understood as the parameter information of the large model, including at least the model name, model type, the number of central processing units (CPU) and / or the number of graphics processors, the size of video memory, the size of memory, and the disk capacity of the large model, and can also include the large model task priority value. The computing resource node is the computing server, and one computing resource node corresponds to one computing server. The target computing resource node can be understood as the computing server whose available computing resources are most matched with the large model when the large model is imported and stored in the Kubernetes platform. Container parameter information can be understood as information about container operation, including at least container operation information of the container itself, node operation information of the computing resource node to which the container belongs, and model operation information of the large model mounted in the container.

[0037] It can be understood that the big model operated and maintained based on the Kubernetes platform and the big model scheduling system 1 in this application is a big model of the power industry, such as demand forecasting models, load scheduling models, equipment maintenance models and energy management models, etc. These models can help power organizations better optimize resource allocation, improve operational efficiency, reduce costs, improve service quality, etc.

[0038] Specifically, the model management module 11 is responsible for the management of the trained large model files throughout their life cycle and receiving the user's model submission requirements; it supports the creation of various types of large model content. System management users upload trained model files through the model management module 11 to create different types of models, and can specify the model's scale, language and other attributes.

[0039] The computing power scheduling module 12 is the core module of the large model scheduling system 1, which provides scheduling decisions and containerized deployment for the operation of large models. System management users deploy large model applications by specifying the large model to be run and resource specifications and other attributes. The computing power scheduling module 12 accepts deployment requests and first schedules computing power resource nodes based on the two indicators of large model type and large model resource specifications. Then, a Kubernetes container is created on the selected computing power resource node, and the large model storage is mounted to the Kubernetes container to complete the model deployment.

[0040] The model scheduling module 13 includes the traffic collection and status monitoring of each large model running container, and the resource usage collection and status monitoring of each computing resource node. When a large model running container or a computing resource node fails, the corresponding fault migration strategy can be issued to ensure the continuous availability of the large model application operation; the model scheduling module 13 also includes the analysis of the traffic collected from the large model running container, and performs load balancing operations on the large model application based on the two indicators of query traffic per second (Queries Per Second, QPS) and response time (Response Time, RT) of the large model running container and its mounted large model.

[0041] The present embodiment provides a large model scheduling system, including a model management module, a computing power scheduling module and a model scheduling module; the model management module is used to respond to the client's management operations on the target large model and perform corresponding model management on the target large model; the computing power scheduling module is used to determine the target computing power resource node that matches the target large model according to the model parameter information of the target large model, and mount the target large model to the container on the target computing power resource node; the model scheduling module is used to obtain the container parameter information of each of the containers, and when there is an abnormality in the container parameter information, the model scheduling processing is performed on the large model mounted on the abnormal container. The above technical scheme effectively realizes the use of computing power resources to support the operation of large model reasoning applications, improves the utilization rate of computing power resources, and realizes efficient computing power resource selection and large model scheduling based on Kubernetes.

[0042] Optionally, the model management module 11 includes:

[0043] A large model creation unit, configured to, in response to a creation operation of a target large model by a client, obtain a large model file of the target large model based on a set format, and store the large model file;

[0044] The large model processing unit is used to respond to the processing operation of the client on the target large model and perform corresponding processing on the target large model. The processing operation at least includes quantization operation, editing operation and deletion operation.

[0045] In this embodiment, the setting format can be understood as a pre-set format for importing a large model, that is, the attributes of the large model, which at least include model scale and language, and can also include model type, operating framework and resource requirements, etc. The setting format can be pre-defined or selected when the client uploads the target large model. The large model file can be understood as the file package of the target large model.

[0046] Specifically, the user uploads the big model file of the target big model through the client, and the big model creation unit obtains the big model file of the target big model in response to the creation operation of the client. The big model file can be uploaded based on the specified model scale (7B / 13B / 33B) and language (Chinese / English) and other attributes. The big model file is uploaded to the Kubernetes platform and stored by the big model creation unit in the big model scheduling system.

[0047] When it is necessary to adjust the computing power resources required for the operation of the target large model, the user can perform a quantization operation of a specified precision on the content of the created target large model through the client. The large model processing unit responds to the quantization operation and generates a new large model file with different inference precision to achieve a quantitative update of the target large model.

[0048] When there is a need to edit the properties of the target large model, the user can edit the properties of the created target large model to specify the content (such as language) through the client. The large model processing unit responds to the editing operation and makes corresponding property adjustments and transformations to achieve modification and update of the target large model.

[0049] When a deletion demand is generated for the target large model, the user can delete the created target large model attributes through the client. The large model processing unit responds to the deletion operation, deletes the corresponding large model file from the storage space, and deletes the corresponding parameter information at the same time, thereby achieving deletion and cleaning of the target large model.

[0050] Optionally, the computing power scheduling module 12 includes:

[0051] A computing power node filtering unit, used to obtain model parameter information of a target large model, and filter at least one intermediate computing power resource node matching the target large model from a plurality of computing power resource nodes according to the model parameter information;

[0052] A target node determination unit is used to determine the node weight of each intermediate computing resource node according to the node parameter information of each intermediate computing resource node, and determine the intermediate computing resource node with the highest node weight as the target computing resource node;

[0053] The running container creation unit is used to create a large model running container on the target computing resource node and hang the target large model on the large model running container.

[0054] In this embodiment, the computing power scheduling module 12 can provide intelligent scheduling decisions for the Kubernetes container running the large model. Among them, the intermediate computing power resource node can be understood as a computing power resource node that can support the mounting of the large model. The node parameter information can be understood as the parameter information of the computing power resource node, including at least the flow influencing factor, the available resource information and the weight factor of the available resources. Among them, the available resource information includes at least the available resources of the graphics processor GPU, the available resources of the video random access memory (Video Random Access Memory, VRAM), the available resources of the central processing unit CPU, the available resources of the random access memory (Random Access Memory, RAM) and the available resources of the disk DISK. The target computing power resource node can be understood as the optimal computing power resource node that can support the mounting of the large model. The large model running container can be understood as the carrier of the large model on the target computing power resource node, which is a Kubernetes container.

[0055] Specifically, the computing power node filtering unit receives the large model application scenario deployment and operation information for the target large model sent by the client, that is, the model parameter information of the target large model, and the operation information at least includes the model name, model type, resource specifications (CPU / GPU quantity, video memory size, memory size and disk capacity) and priority of the specified target large model. In response to the model parameter information of the target large model, each computing power resource node is multi-dimensionally filtered, and the computing power resource node that can meet the model parameter information of the target large model among the computing power resource nodes is determined as an intermediate computing power resource node. For example, when the CPU / GPU quantity, video memory size, memory size and disk capacity of a computing power resource node are greater than the resource specifications required by the target large model, and the large model type that the computing power resource node can run includes the model type of the target large model, and the large model priority that the computing power resource node can match includes the priority of the target large model, etc., the computing power resource node can be determined as an intermediate computing power resource node.

[0056] Specifically, for each intermediate computing resource node, the target node determination unit determines the traffic impact factor Φ of the intermediate computing resource node according to the traffic impact factor Φ of the intermediate computing resource node. n , available resource information node state and the weight factor weight of available resources factor , and the priority α in the model parameter information of the target large model n , through Priority n =α n *Φ n *f n (weight factor ,nodestate ) Calculate the node weight Priority of the intermediate computing resource node n , where f n Indicates that the weight value is calculated for each available resource information of the computing resource node. Finally, the intermediate computing resource node with the highest node weight is determined as the target computing resource node. The target computing resource node is the computing resource node with the best comprehensive performance and is most suitable for the scheduling of this large model deployment task.

[0057] After determining the target computing power resource node, the running container creation unit creates a large model running container on the target computing power resource node to hang the target large model on the large model running container to realize the running of the target large model on the target computing power resource node.

[0058] It is understandable that only one copy of the same large model storage is needed, and when starting multiple application containers of the same large model, it is only necessary to mount the large model storage to multiple containers.

[0059] Optionally, the model scheduling module 13 includes:

[0060] The abnormal migration unit is used to monitor each container. When the container parameter information of the container is abnormal, the abnormal container information of the abnormal container is obtained, and the corresponding scheduling instruction is called according to the abnormal container information. The large model mounted on the abnormal container is scheduled according to the scheduling instruction.

[0061] The load balancing unit is used to monitor each container, obtain the container parameter information of each container, and expand or reduce the container according to the model operation information in the container parameter information to perform large model scheduling.

[0062] In this embodiment, the container parameter information can be understood as the relevant parameter information of the large model running container, including the container running information of the container itself and the node running information of the computing resource node to which the container belongs, as well as the model running information of the large model mounted in the container. Among them, the container running information at least includes the flow information and the access status of the container when the container is running; the node running information at least includes the access status and service status of the computing resource node; the model running information at least includes the query flow and response time of the large model per second. Abnormal container information can be understood as the specific abnormal content of the container parameter information, including the abnormal information of the container itself and the abnormal information of the computing resource node to which the container belongs, such as the access status of the container is unreachable, the node access status is unreachable, or the node service status is abnormal. The scheduling instruction can be understood as an instruction to schedule the container, the computing resource node to which the container belongs, or the large model mounted in the container. The scheduling instruction at least includes the restart scheduling for the large model running container, the reconstruction scheduling for the large model running container, the restart scheduling for the computing resource node service, and the model mounting transfer scheduling for the computing resource node.

[0063] Specifically, the abnormal migration unit collects traffic and monitors the status of each large model running container to obtain container parameter information including container running information and node running information. When an abnormality occurs in either the container running information or the node running information, the container corresponding to the abnormal information is determined, and the container is determined as an abnormal container. The abnormal content is determined based on the abnormal container running information or the node running information, and the abnormal content is determined as abnormal container information. Each type of abnormal container information has its corresponding scheduling strategy, and the abnormal container information of the abnormal container is obtained, and the corresponding scheduling instruction is called according to the abnormal container information. The large model mounted on the abnormal container is scheduled according to the scheduling instruction to ensure the continuous availability of the large model application operation.

[0064] Specifically, the load balancing module monitors and regulates the running traffic of the large model mounted in the large model running container by monitoring the large model running container. When it is determined that there is a need for regulation based on the monitored model running information, the large model running container is expanded or reduced to achieve load balancing when the large model is running.

[0065] It can be understood that the container parameter information corresponding to the monitoring of the abnormal migration unit and the load balancing unit is monitored by the monitoring server and transmitted to the corresponding unit, so that the container parameter information can be directly applied.

[0066] Optional, abnormal migration unit, specifically used for:

[0067] a1. Monitor each container, obtain container parameter information for each container, and determine the container with abnormal container parameter information as an abnormal container. The container parameter information includes container operation information of the container and node operation information of the computing resource node to which the container belongs.

[0068] In this embodiment, each container is monitored to obtain container parameter information for each large model running container. The container parameter information includes container operation information of the large model running container and node operation information of the computing resource node to which the container belongs. If there is an abnormality in the container operation information or the node operation information, the large model running container corresponding to the container parameter information is determined to be an abnormal container.

[0069] It can be understood that no matter the exception occurs in the large model running container itself or in the computing resource node to which the large model running container belongs, the large model running container is referred to as an abnormal container.

[0070] b1. When an exception occurs in the container operation information, obtain the container exception information, call the container processing instruction according to the container exception information, and restart or rebuild the abnormal container.

[0071] In this embodiment, the container abnormal information can be understood as abnormal information about the abnormal container itself in the abnormal container information, such as the container access status is unreachable. The container processing instruction can be understood as an instruction for scheduling the abnormal container in the scheduling instruction, including restart scheduling and reconstruction scheduling for the abnormal container.

[0072] Specifically, if the container access status in the container running information is unreachable, it is determined that the container running information is abnormal, and the container abnormality is determined to be unreachable as container abnormality information. Based on the container abnormality information, the container processing instruction related to the container is called, and the abnormal container is restarted based on the container processing instruction. When the restart is invalid, the abnormal container is rebuilt.

[0073] c1. When an abnormality occurs in the node operation information, obtain the node abnormality information, call the node processing instruction according to the node abnormality information, create a backup container on the backup computing resource node, and hot migrate the large model mounted on the abnormal container to the backup container.

[0074] In this embodiment, the node abnormality information can be understood as the abnormal information of the computing power resource node to which the abnormal container belongs in the abnormal container information, such as the node access status is unreachable or the node service status is abnormal. The node processing instruction can be understood as the instruction for scheduling the computing power resource node to which the abnormal container belongs in the scheduling instruction, including the restart scheduling for the computing power resource node service, and the model mounting transfer scheduling for the computing power resource node. The spare computing power resource node can be understood as other available computing power resource nodes other than the computing power resource node to which the abnormal container belongs, such as the intermediate computing power resource node with the second highest weight. The spare container can be understood as a large model running container built on the spare computing power resource node.

[0075] Specifically, if the node service status in the node operation information is abnormal service status (service interruption, service content error, service object error, service data loss, etc.), it is determined that the node operation information is abnormal, and the service status abnormality is determined as node abnormal information. Based on the node abnormal information, the node processing instructions related to the computing power resource node are called to restart the relevant services. If the node access status in the node operation information is node access unreachable, it is determined that the node operation information is abnormal, and the node access unreachable is determined as node abnormal information. Based on the node abnormal information, the node processing instructions related to the computing power resource node are called, and a new standby large model operation container is created on the standby computing power resource node, and then the model file storage of the large model mounted in the abnormal container on the computing power resource node is hot migrated to the standby container on the standby computing power resource node.

[0076] Optional load balancing unit, specifically used for:

[0077] a2. Monitor each container and obtain container parameter information for each container. The container parameter information at least includes model operation information of the large model mounted in the container. The model operation information at least includes query traffic and response time per second.

[0078] In this embodiment, the model operation information can be understood as relevant parameter information of the large model operation, including at least the query flow per second QPS and the response time RT.

[0079] Specifically, each container is monitored to obtain container parameter information for each large model running container. The container parameter information includes model running information of the large model mounted in the large model running container. That is, by monitoring the large model running container, the query traffic per second QPS and response time RT corresponding to the large model running in the large model running container are obtained.

[0080] b2. When the query flow rate per second is greater than the first flow threshold or the response time is greater than the first time threshold, the container is expanded.

[0081] In this embodiment, the first traffic threshold can be understood as the maximum query traffic per second, and the first time threshold can be understood as the maximum response time.

[0082] Specifically, when the query traffic per second is greater than the first traffic threshold or the response time is greater than the first time threshold, it indicates that the large model running container is insufficient to support the smooth operation of the mounted large model. Therefore, it is necessary to expand the capacity of the large model running container, such as increasing the running memory capacity of the large model running container, or creating a new linked large model running container. The two containers jointly mount the large model, and the newly created large model running container and the originally deployed large model running container form a load-balanced deployment form.

[0083] c2. When the query flow rate per second is less than the second flow threshold or the response time is less than the second time threshold, the container is scaled down.

[0084] In this embodiment, the second traffic threshold can be understood as the minimum query traffic per second, and the second time threshold can be understood as the minimum response time.

[0085] Specifically, when the query flow per second is less than the second flow threshold or the response time is less than the second time threshold, it indicates that the memory capacity of the large model running container is too large. While supporting the smooth operation of the mounted large model, there is still surplus content capacity, and there is a waste of running content capacity. Therefore, it is necessary to reduce the capacity of the large model running container, for example, by reducing the running memory capacity of the large model running container. The expansion or reduction operation of the container can provide differentiated, deterministic end-to-end service quality assurance for the operation and application of large models.

[0086] Figure 2 is an interactive schematic diagram of a large model scheduling system provided by the first embodiment of the present invention, such as Figure 2 As shown, the client performs specific management operations on the large model and synchronizes the management operations to the large model scheduling system. The large model scheduling system responds to the management operations. During the storage phase of the large model, it is necessary to select the computing power resource nodes. During the operation phase of the large model, it is necessary to schedule the computing power resource nodes. Therefore, the large model scheduling system needs to connect and interact with the Kubernetes computing power resource cluster through a switch.

[0087] A large model scheduling system provided in this embodiment unifies a large number of heterogeneous computing resources by adopting a computing resource management architecture based on Kubernetes. Fully consider the differences in large model task types: different types of tasks have different requirements for computing resource configuration, so different types of tasks require different computing resources; large model task scale differences: different scales require different computing resources, and align the task requirement specifications with the computing resource configuration; large model task priority differences: different large model tasks have different priorities, and high-priority tasks need to occupy computing resources with better performance, so as to perform efficient computing resource scheduling and solve the current problem of low computing resource utilization. Fully consider the differences in large model task loads: different large model tasks have different user inference requirements, so different large model tasks have different access volumes and computing resource load conditions, so building model load balancing at the same time can well solve the performance bottleneck problem of deployed large model application containers.

[0088] Embodiment 2

[0089] Figure 3 It is a flow chart of a large model scheduling method provided in the second embodiment of the present invention, which can be applied to the situation of comprehensive management of large models. The method is executed by the large model scheduling system.

[0090] like Figure 3 As shown, the method includes:

[0091] S101. Respond to the management operation of the client on the target large model through the model management module, and perform corresponding model management on the target large model.

[0092] In this embodiment, through the large model creation unit in the model management module, in response to the client's creation operation on the target large model, the large model file of the target large model is obtained based on the set format, and the large model file is stored; through the large model processing unit in the model management module, it is used to respond to the client's processing operation on the target large model, and perform corresponding processing on the target large model, and the processing operation at least includes quantization operation, editing operation and deletion operation.

[0093] S102. Through the computing power scheduling module, for each large model, determine the target computing power resource node that matches the large model according to the model parameter information of the large model, and mount the large model to the container on the target computing power resource node.

[0094] In this embodiment, a computing power node filtering unit in a computing power scheduling module is used to obtain model parameter information of a target large model, and filter at least one intermediate computing power resource node that matches the target large model from multiple computing power resource nodes according to the model parameter information; a target node determination unit in a computing power scheduling module is used to determine the node weight of each intermediate computing power resource node according to the node parameter information of each intermediate computing power resource node, and determine the intermediate computing power resource node with the highest node weight as the target computing power resource node; a running container creation unit in a computing power scheduling module is used to create a large model running container on the target computing power resource node, and hang the target large model on the large model running container.

[0095] S103. Obtain container parameter information of each container through the model scheduling module. When there is an abnormality in the container parameter information, perform model scheduling processing on the large model mounted on the abnormal container.

[0096] In this embodiment, each container is monitored by the abnormal migration unit in the model scheduling module, and container parameter information for each container is obtained. The container with abnormal container parameter information is determined as an abnormal container, and the container parameter information includes the container operation information of the container and the node operation information of the computing power resource node to which the container belongs; when the container operation information is abnormal, the container abnormal information is obtained, and the container processing instruction is called according to the container abnormal information to restart or rebuild the abnormal container; when the node operation information is abnormal, the node abnormal information is obtained, and the node processing instruction is called according to the node abnormal information, a spare container is created on the spare computing power resource node, and the large model mounted on the abnormal container is hot migrated to the spare container.

[0097] Through the load balancing unit in the model scheduling module, each container is monitored to obtain container parameter information for each container. The container parameter information at least includes the model operation information of the large model mounted in the container. The model operation information at least includes the query flow and response time per second. When the query flow per second is greater than the first flow threshold or the response time is greater than the first time threshold, the container is expanded; when the query flow per second is less than the second flow threshold or the response time is less than the second time threshold, the container is reduced.

[0098] The present embodiment provides a large model scheduling method, which responds to the client's management operation on the target large model through the model management module, and performs corresponding model management on the target large model; through the computing power scheduling module, for each large model, determines the target computing power resource node that matches the large model according to the model parameter information of the large model, and mounts the large model to the container on the target computing power resource node; through the model scheduling module, obtains the container parameter information of each container, and when there is an abnormality in the container parameter information, performs model scheduling processing on the large model mounted on the abnormal container. The above technical solution improves the utilization rate of computing power resources and realizes efficient computing power resource selection and large model scheduling.

[0099] Embodiment 3

[0100] Figure 4 : This is a structural diagram of a large model scheduling server provided in Embodiment 3 of the present invention. The large model scheduling server is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The large model scheduling server can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0101] Figure 5 This is a deployment diagram of a large model scheduling server provided by the third embodiment of the present invention, which provides a unified access service for different heterogeneous computing resources. It is compatible with GPU and CPU resources of different manufacturers, such as Figure 5 As shown, the large model scheduling server is connected to multiple computing servers through a layer 2 switch, and is also connected to a Prometheus monitoring server through a layer 2 switch, wherein the computing server is the computing resource node, and the monitoring server performs the monitoring function, monitors and collects data from the large model running containers on each computing server, and sends the collected data to the model scheduling module in the large model scheduling system corresponding to the large model scheduling server, so that the model scheduling module can perform scheduling processing of the large model under abnormal circumstances.

[0102] Among them, the large model scheduling server 40 is as follows Figure 4As shown, it includes at least one processor 41, and a memory connected to at least one processor 41 in communication, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In RAM 43, various programs and data required for the operation of the large model scheduling server 40 can also be stored. The processor 41, ROM 42 and RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0103] Multiple components in the large model scheduling server 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the large model scheduling server 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 41 executes the various methods and processes described above, such as a large model scheduling method.

[0105] In some embodiments, the large model scheduling method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the large model scheduling server 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the large model scheduling method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to execute the large model scheduling method in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input, and at least one output device, and transmit data and instructions to the storage system, the at least one input, and the at least one output device.

[0107] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0108] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein may be implemented on a large model scheduling server having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball), through which the user can provide input to the large model scheduling server. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0111] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A large model scheduling system, characterized in that: The system includes: a model management module, a computing power scheduling module and a model scheduling module; The model management module is used to respond to the management operation of the client on the target large model and perform corresponding model management on the target large model; The computing power scheduling module is used to determine a target computing power resource node matching the target large model according to the model parameter information of the target large model, and mount the target large model to a container on the target computing power resource node; The model scheduling module is used to obtain container parameter information of each container, and when there is an abnormality in the container parameter information, perform model scheduling processing on the large model mounted on the abnormal container.

2. The system according to claim 1, characterized in that The model management module includes: A large model creation unit, configured to, in response to a creation operation of a target large model by a client, obtain a large model file of the target large model based on a set format, and store the large model file; The large model processing unit is used to perform corresponding processing on the target large model in response to the processing operation of the client on the target large model, and the processing operation at least includes quantization operation, editing operation and deletion operation.

3. The system according to claim 1, characterized in that The computing power scheduling module includes: A computing power node filtering unit, used to obtain model parameter information of the target large model, and filter at least one intermediate computing power resource node matching the target large model from multiple computing power resource nodes according to the model parameter information; A target node determination unit, configured to determine the node weight of each of the intermediate computing power resource nodes according to the node parameter information of each of the intermediate computing power resource nodes, and determine the intermediate computing power resource node with the highest node weight as the target computing power resource node; The running container creation unit is used to create a large model running container on the target computing resource node and hang the target large model on the large model running container.

4. The system according to claim 1, characterized in that The model scheduling module includes: an abnormal migration unit, configured to monitor each of the containers, and when an abnormality occurs in the container parameter information of the container, obtain abnormal container information of the abnormal container, call a corresponding scheduling instruction according to the abnormal container information, and perform model scheduling processing on the large model mounted on the abnormal container according to the scheduling instruction; The load balancing unit is used to monitor each of the containers, obtain container parameter information of each container, and expand or reduce the capacity of the container according to the model operation information in the container parameter information to perform large model scheduling.

5. The system according to claim 4, characterized in that The abnormal migration unit is specifically used for: Monitor each of the containers, obtain container parameter information for each of the containers, and determine a container with abnormal container parameter information as an abnormal container, wherein the container parameter information includes container operation information of the container and node operation information of a computing resource node to which the container belongs; When the container operation information is abnormal, the container abnormal information is obtained, and the container processing instruction is called according to the container abnormal information to restart or rebuild the abnormal container; When an abnormality occurs in the node operation information, the node abnormality information is obtained, and the node processing instruction is called according to the node abnormality information, a backup container is created on the backup computing resource node, and the large model mounted on the abnormal container is hot migrated to the backup container.

6. The system according to claim 4, characterized in that The load balancing unit is specifically used for: Monitor each of the containers to obtain container parameter information for each of the containers, wherein the container parameter information includes at least model operation information of the large model mounted in the container, and the model operation information includes at least query flow and response time per second; When the query flow rate per second is greater than a first flow rate threshold or the response time is greater than a first time threshold, the container is expanded; When the query flow rate per second is less than a second flow rate threshold or the response time is less than a second time threshold, the container is scaled down.

7. A large model scheduling method, characterized in that: include: Responding to the management operation of the client on the target large model through the model management module, performing corresponding model management on the target large model; Through the computing power scheduling module, for each large model, a target computing power resource node matching the large model is determined according to the model parameter information of the large model, and the large model is mounted to a container on the target computing power resource node; The container parameter information of each container is obtained through the model scheduling module. When there is an abnormality in the container parameter information, the model scheduling process is performed on the large model mounted on the abnormal container.

8. A large model scheduling server, characterized in that: The system according to any one of claims 1 to 6 is integrated, wherein the large model scheduling server is connected to a plurality of computing servers via a switch, and is connected to a monitoring server via a switch; The large model scheduling server includes at least one processor; a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the large model scheduling method described in claim 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a large model scheduling method as described in claim 7 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which implements a large model scheduling method according to claim 7 when executed by a processor.

Images