Model reasoning service deployment method and device, equipment and storage medium

Abstract

The invention discloses a model reasoning service deployment method and device, equipment and a storage medium. The method comprises the following steps: in a model reasoning process, obtaining reasoning index parameters of all model reasoning services in a computing resource cluster; and for each model reasoning service, determining a threshold according to the reasoning index parameter corresponding to the model reasoning service and each pre-constructed mode, and determining a target scheduling mode corresponding to the model reasoning service, the target scheduling mode comprising a service quality priority mode and a resource utilization rate priority mode. And determining a service deployment mode corresponding to the model reasoning service based on a service deployment strategy in the target scheduling mode, and performing service deployment on the model reasoning service based on the service deployment mode, so as to dynamically adjust the scheduling strategy according to needs, ensure that the model reasoning service can meet predetermined performance requirements, and improve the performance of the model reasoning service. And meanwhile, the utilization rate of the stock GPU resources is improved.

Description

Technical Field

[0001] The present invention relates to the field of computing resource allocation technology, and in particular to a model reasoning service deployment method, device, equipment and storage medium. Background Art

[0002] With the development of machine learning and artificial intelligence technologies, various businesses in the field of automotive finance have gradually used machine learning and artificial intelligence technologies to develop and train artificial intelligence models in various business verticals in accordance with their own business development needs, and applied them to normal business areas by deploying model inference services. Due to the strong isolation between various vertical businesses in the field of automotive finance, the models used by each vertical business are quite different, and there is basically no scenario of model usage between businesses. This has also led to an increasing demand for GPU resources for model inference as more and more artificial intelligence models are added to various vertical business fields, and the pressure on GPU resources has gradually increased. Limited by the scarcity of GPU resources, the cost of expanding GPU resources is high, and there is an urgent need to provide a method to improve the utilization rate of existing GPU resources. Summary of the invention

[0003] The present invention provides a model reasoning service deployment method, device, equipment and storage medium to dynamically adjust the scheduling strategy as needed to ensure that the model reasoning service can meet the predetermined performance requirements while improving the utilization rate of existing GPU resources.

[0004] According to one aspect of the present invention, a model reasoning service deployment method is provided. The method comprises:

[0005] During the model inference process, obtain the inference indicator parameters of all model inference services in the computing resource cluster;

[0006] For each of the model reasoning services, a threshold is determined according to the reasoning indicator parameters corresponding to the model reasoning service and each pre-built mode, and a target scheduling mode corresponding to the model reasoning service is determined, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode;

[0007] Based on the service deployment strategy in the target scheduling mode, the service deployment mode corresponding to the model reasoning service is determined, and based on the service deployment mode, the model reasoning service is deployed.

[0008] According to another aspect of the present invention, a model reasoning service deployment device is provided. The device comprises:

[0009] The inference indicator parameter acquisition module is used to obtain the inference indicator parameters of all model inference services in the computing resource cluster during the model inference process;

[0010] A target scheduling mode determination module is used to determine, for each of the model reasoning services, a target scheduling mode corresponding to the model reasoning service according to the reasoning indicator parameters corresponding to the model reasoning service and the threshold values ​​of each pre-built mode, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode;

[0011] The service deployment mode determination module is used to determine the service deployment mode corresponding to the model reasoning service based on the service deployment strategy in the target scheduling mode, and to deploy the model reasoning service based on the service deployment mode.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model reasoning service deployment method described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model reasoning service deployment method described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention obtains the inference index parameters of all model inference services in the computing resource cluster during the model inference process. For each of the model inference services, a threshold is determined based on the inference index parameters corresponding to the model inference service and each pre-built mode, and the target scheduling mode corresponding to the model inference service is determined, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode. Based on the service deployment strategy in the target scheduling mode, the service deployment mode corresponding to the model inference service is determined, and based on the service deployment mode, the model inference service is deployed, which solves the problem of insufficient or unbalanced service resource utilization, and realizes dynamic adjustment of the scheduling strategy as needed, ensuring that the model inference service can meet the predetermined performance requirements, optimize resource allocation, improve overall resource utilization, and reduce operating costs.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a model reasoning service deployment method provided according to Embodiment 1 of the present invention;

[0021] Figure 2 is a flowchart of a model reasoning service deployment method provided in Embodiment 2 of the present invention;

[0022] Figure 3 is a structural diagram of a model reasoning service deployment device provided according to Embodiment 3 of the present invention;

[0023] Figure 4 It is a structural diagram of an electronic device for implementing the model reasoning service deployment method of an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] Figure 1 This is a flowchart of a model reasoning service deployment method provided in the first embodiment of the present invention. This embodiment is applicable to the case of optimizing the allocation of computing resources required for model reasoning services. The method can be executed by a model reasoning service deployment device. The model reasoning service deployment device can be implemented in the form of hardware and / or software. The model reasoning service deployment device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0028] S101. During the model inference process, obtain inference index parameters of all model inference services in the computing resource cluster.

[0029] It should be noted that in the field of machine learning and artificial intelligence, model reasoning services refer to the process of using trained artificial intelligence models to predict or analyze new input data. This service is usually deployed on computing resources, such as GPUs, to provide efficient reasoning capabilities. In areas such as automotive finance, model reasoning services are widely used in various vertical businesses to support intelligent decision-making and operations of the business.

[0030] A computing resource cluster can refer to a group of interconnected computing resources (such as servers, GPUs, etc.) that work together to provide model inference services. These computing resources are organized together so that they can efficiently process a large number of model inference requests.

[0031] The inference indicator parameters may refer to the inference parameters of the model inference service during the model inference process, such as response time, throughput, resource utilization (such as GPU memory occupancy), and error rate.

[0032] Specifically, the corresponding monitoring indicators and time periods are selected through the inference service scheduling strategy controller, the monitoring items related to the model inference service are screened out, and the required inference indicator parameters are exported or viewed.

[0033] S102: For each of the model reasoning services, determine a threshold value according to the reasoning indicator parameters corresponding to the model reasoning service and each pre-built mode, and determine a target scheduling mode corresponding to the model reasoning service.

[0034] The target scheduling mode includes a service quality priority mode and a resource utilization priority mode. The mode determination threshold can be set according to actual conditions, and the present invention does not limit the determination of the specific value of the mode determination threshold.

[0035] It should be noted that the attributes of the mode determination threshold include at least: the average number of requests with a time period as a window, the average response time with a percentile as a window, the maximum response time with a time period as a window, whether the default scheduling method is always used within a specified time, and whether the default scheduling method is always not used within a specified time. Exemplarily, the average number of requests with a time period as a window may include the number of requests per second, the number of requests per minute, and the number of requests per hour. The average response time with a percentile as a window may include the P99 response time, the P95 response time, and the P90 response time. The maximum response time with a time period as a window includes the maximum response time per minute and the maximum response time per hour.

[0036] Specifically, for each model reasoning service, the size relationship between the thresholds can be determined based on the reasoning indicator parameters and the pre-built modes, and then it can be determined whether the target scheduling mode corresponding to the model reasoning service is a service quality priority mode or a resource utilization priority mode.

[0037] Exemplarily, determining the target scheduling mode corresponding to the model reasoning service according to the inference indicator parameter corresponding to the model reasoning service and the threshold values ​​of each pre-built mode includes:

[0038] The inference indicator parameter and the mode determination threshold are compared and processed, and the target scheduling mode corresponding to the model inference service is determined according to the matching result between the inference indicator parameter and the mode determination threshold.

[0039] That is to say, the inference indicator parameters are compared and matched with the mode determination threshold, that is, the inference indicator parameters are determined from the dimension of the average number of requests with the time period as the window, from the dimension of the average response time with the percentile as the window, from the dimension of the longest response time with the time period as the window, from the dimension of always using the default scheduling method within the specified time, and from the dimension of whether the default scheduling method is always not used within the specified time, whether the target scheduling mode corresponding to the model reasoning service is the service quality priority mode or the resource utilization priority mode.

[0040] It should be noted again that the specific value of the threshold determined for each mode is not specifically limited in the present invention and can be set according to actual usage or historical experience.

[0041] S103. Determine the service deployment mode corresponding to the model reasoning service based on the service deployment strategy in the target scheduling mode, and deploy the model reasoning service based on the service deployment mode.

[0042] In the technical solution of the present invention, when the target scheduling mode is a service quality priority mode, the service deployment method includes an independent deployment method and a joint deployment method. When the target scheduling mode is a resource utilization priority mode, the service deployment method includes a joint deployment method and a card sharing deployment method.

[0043] It should be noted that the independent deployment mode is that a single model inference service occupies a computing resource in the computing resource cluster alone. The joint deployment mode is that multiple model inference services of the same inference type using the same inference service framework occupy the same computing resource in the computing resource cluster in a shared context manner. The card sharing deployment mode is that multiple model inference services occupy the same computing resource in the computing resource cluster in a time slice rotation manner.

[0044] It should be further explained that the independent deployment mode is that the model inference service monopolizes the entire GPU card resources, and the model inference service has exclusive use of the GPU. The model inference service occupies the GPU until the life cycle of the model inference service ends (the model inference service is officially offline). The joint deployment mode is that multiple model inference services use the same inference service framework (such as Nvidia Triton framework) to jointly use the same GPU of the same computing node in a shared context mode (Nvidia MPS). This deployment method avoids the delay caused by context switching when multiple working processes use the GPU. The shared card deployment mode is that multiple model inference services are directly configured to use the same GPU of the same computing node. This deployment method will perform explicit GPU context switching when different processes access the GPU; multiple processes obtain the right to use the GPU through time slice rotation.

[0045] In the present invention, different target scheduling modes have different service deployment strategies. For each target scheduling mode, a corresponding service deployment strategy is adopted to determine the service deployment mode corresponding to the model reasoning service. After determining the service deployment mode, the model deployment service is notified to redeploy the model reasoning service according to the new service deployment mode.

[0046] The technical solution of the embodiment of the present invention obtains the inference index parameters of all model inference services in the computing resource cluster during the model inference process. For each of the model inference services, a threshold is determined based on the inference index parameters corresponding to the model inference service and each pre-built mode, and the target scheduling mode corresponding to the model inference service is determined, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode. Based on the service deployment strategy in the target scheduling mode, the service deployment mode corresponding to the model inference service is determined, and based on the service deployment mode, the model inference service is deployed, which solves the problem of insufficient or unbalanced service resource utilization, and realizes dynamic adjustment of the scheduling strategy as needed, ensuring that the model inference service can meet the predetermined performance requirements, optimize resource allocation, improve overall resource utilization, and reduce operating costs.

[0047] Embodiment 2

[0048] Figure 2 This is a flow chart of a model reasoning service deployment method provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment further refines the service deployment method corresponding to the model reasoning service based on the service deployment strategy in the target scheduling mode. Figure 2 As shown, the method includes:

[0049] S201. During the model inference process, obtain the inference index parameters of all model inference services in the computing resource cluster.

[0050] S202. For each of the model inference services, determine a threshold value based on the inference indicator parameters corresponding to the model inference service and each pre-built mode, and determine a target scheduling mode corresponding to the model inference service, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode.

[0051] S203. When the target scheduling mode is a quality of service priority mode, determine the service deployment mode corresponding to the model reasoning service based on the deployment mode threshold of the quality of service priority mode and the reasoning indicator parameter.

[0052] Among them, the deployment mode threshold may refer to a judgment threshold for determining the deployment mode, which is used to judge whether the service deployment mode is an independent deployment mode or a joint deployment mode when the target scheduling mode is a service quality priority mode. Exemplarily, the deployment mode threshold may be set to a single model reasoning using more than 75% of the video memory of a single card. It is worth noting that in the service quality priority mode, the card sharing deployment method is not used.

[0053] Exemplarily, the deployment mode threshold based on the service quality priority mode and the inference indicator parameter determine the service deployment mode corresponding to the model reasoning service, including: when the inference indicator parameter is greater than the deployment mode threshold, determining that the service deployment mode corresponding to the model reasoning service is an independent deployment mode; when the inference indicator parameter is less than or equal to the deployment mode threshold, determining that the service deployment mode corresponding to the model reasoning service is a joint deployment mode.

[0054] That is to say, in the service quality priority mode, the service quality scheduling strategy of the model inference service needs to be met as much as possible. This strategy will schedule the model inference service with high resource occupancy (that is, the inference index parameter is greater than the deployment mode threshold) as a separate deployment mode as much as possible. The model inference service with medium and low resource occupancy (that is, the inference index parameter is less than or equal to the deployment mode threshold) will be scheduled as a joint deployment mode.

[0055] S204: When the target scheduling mode is a resource utilization priority mode, determine a service deployment mode corresponding to the model reasoning service based on a scheduling strategy that maximizes resource utilization.

[0056] Specifically, the resource utilization priority mode is a mode that maximizes resource utilization. It is necessary to use a scheduling strategy that maximizes resource utilization to determine the service deployment method corresponding to the model inference service.

[0057] Exemplarily, based on a greedy algorithm, with reference to a scheduling strategy that maximizes resource utilization, the deployment proportions of the joint deployment method and the card sharing deployment method in the service deployment method are determined respectively, or the first resource utilization corresponding to when the joint deployment method is adopted and the second resource utilization corresponding to when the card sharing deployment method is adopted are calculated respectively, and the joint deployment method is adopted when the first resource utilization is greater than the second resource utilization, and the card sharing deployment method is adopted when the first resource utilization is less than or equal to the second resource utilization.

[0058] That is to say, when the target scheduling mode is the resource utilization priority mode, a variety of methods can be used to determine the service deployment method, including at least determining the deployment proportion of the joint deployment method and the card sharing deployment method in the service deployment method based on a greedy algorithm; or determining the service deployment method corresponding to the model reasoning service based on the resource utilization of the joint deployment method and the card sharing deployment method.

[0059] Specifically, when the target scheduling mode is the resource utilization priority mode, it is necessary to meet the scheduling strategy of maximizing resource utilization as much as possible. This strategy defaults to deploying model reasoning services that occupy various resources on the same computing resource as much as possible through a greedy algorithm in a joint deployment and card sharing deployment manner. On the other hand, when the first resource utilization is greater than the second resource utilization, the joint deployment method is adopted, and when the first resource utilization is less than or equal to the second resource utilization, the card sharing deployment method is adopted.

[0060] Exemplarily, when the target scheduling mode is a resource utilization priority mode, the detailed scheduling process based on the greedy algorithm and referring to the scheduling strategy for maximizing resource utilization includes:

[0061] Submit a query request to the resource management service to obtain a list of all resource nodes in the current cluster that are marked as being used for the resource utilization priority policy;

[0062] Traverse the resource node list to obtain the resource node with the largest remaining available video memory and the corresponding GPU card number in the resource node list;

[0063] Query the deployment affinity of the inference service being deployed, and obtain whether the deployment mode of the inference service uses joint deployment or shared card deployment. If the deployment mode is not specified for the inference service, the deployment mode with the most remaining resources is used by default;

[0064] Bind the inference service and the corresponding resource node and GPU card number, generate a scheduling plan, and notify the model deployment service to deploy the model inference service.

[0065] Furthermore, if the resource management service does not return a list of nodes that are being used for the resource utilization priority policy, or all nodes in the returned resource node list do not meet the GPU resources required by the inference service:

[0066] Submit a query request to the resource management service to obtain a list of idle resource nodes in the current cluster;

[0067] Traverse the resource node list to obtain the resource node with the largest total video memory and the corresponding GPU card number in the resource node list;

[0068] Submit a resource allocation request to the resource management service and mark the specified node as a node being used for resource utilization priority scheduling policy;

[0069] Query the deployment affinity of the inference service being deployed, and obtain whether the deployment mode of the inference service uses joint deployment or shared card deployment. If the deployment mode of the inference service is not specified, the joint deployment mode is used by default.

[0070] Bind the inference service and the corresponding resource node and GPU card number, generate a scheduling plan, and notify the model deployment service to deploy the model inference service.

[0071] S205. Based on the service deployment method, the model reasoning service is deployed.

[0072] The technical solution of the embodiment of the present invention determines the service deployment mode corresponding to the model reasoning service based on the deployment mode threshold of the service quality priority mode and the reasoning indicator parameter when the target scheduling mode is the service quality priority mode. When the target scheduling mode is the resource utilization priority mode, the service deployment mode corresponding to the model reasoning service is determined based on the scheduling strategy that maximizes resource utilization. According to different target scheduling modes, different service deployment methods are adopted, which can reasonably plan the allocation of tasks on hardware resources, avoid the situation where multiple tasks compete for the same resource at the same time, resulting in performance degradation or system crash, and promote the efficient, stable and reliable operation of the model reasoning service.

[0073] Embodiment 3

[0074] Figure 3 This is a schematic diagram of the structure of a model reasoning service deployment device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0075] The inference index parameter acquisition module 301 is used to obtain the inference index parameters of all model inference services in the computing resource cluster during the model inference process;

[0076] The target scheduling mode determination module 302 is used to determine the target scheduling mode corresponding to each model reasoning service according to the reasoning indicator parameters corresponding to the model reasoning service and the threshold values ​​of each pre-built mode, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode;

[0077] The service deployment mode determination module 303 is used to determine the service deployment mode corresponding to the model reasoning service based on the service deployment strategy in the target scheduling mode, and to deploy the model reasoning service based on the service deployment mode.

[0078] The technical solution of the embodiment of the present invention obtains the inference index parameters of all model inference services in the computing resource cluster during the model inference process. For each of the model inference services, a threshold is determined based on the inference index parameters corresponding to the model inference service and each pre-built mode, and the target scheduling mode corresponding to the model inference service is determined, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode. Based on the service deployment strategy in the target scheduling mode, the service deployment mode corresponding to the model inference service is determined, and based on the service deployment mode, the model inference service is deployed, which solves the problem of insufficient or unbalanced service resource utilization, and realizes dynamic adjustment of the scheduling strategy as needed, ensuring that the model inference service can meet the predetermined performance requirements, optimize resource allocation, improve overall resource utilization, and reduce operating costs.

[0079] Optionally, the attributes of the mode determination threshold include at least: the average number of requests in a time period window, the average response time in a percentile window, the longest response time in a time period window, always using the default scheduling method within a specified time, and never using the default scheduling method within a specified time.

[0080] Optionally, the target scheduling mode determination module 302 is specifically configured to:

[0081] The inference indicator parameter and the mode determination threshold are compared and processed, and the target scheduling mode corresponding to the model inference service is determined according to the matching result between the inference indicator parameter and the mode determination threshold.

[0082] Optionally, when the target scheduling mode is a service quality priority mode, the service deployment mode includes an independent deployment mode and a joint deployment mode, wherein the independent deployment mode is that a single model inference service occupies a computing resource in the computing resource cluster alone, and the joint deployment mode is that multiple model inference services of the same inference type using the same inference service framework occupy the same computing resource in the computing resource cluster in a shared context manner;

[0083] When the target scheduling mode is the resource utilization priority mode, the service deployment method includes a joint deployment method and a card sharing deployment method, wherein the card sharing deployment method is that multiple model inference services occupy the same computing resource in the computing resource cluster in a time slice rotation manner.

[0084] Optionally, the service deployment mode determination module 303 includes:

[0085] A first deployment mode determination unit is used to determine, when the target scheduling mode is a service quality priority mode, based on the deployment mode threshold of the service quality priority mode and the reasoning indicator parameter to determine the service deployment mode corresponding to the model reasoning service;

[0086] The second deployment mode determination unit is used to determine the service deployment mode corresponding to the model reasoning service based on the scheduling strategy of maximizing resource utilization when the target scheduling mode is the resource utilization priority mode.

[0087] Optionally, the first deployment mode determining unit is specifically configured to:

[0088] When the inference index parameter is greater than the deployment mode threshold, determining that the service deployment mode corresponding to the model inference service is an independent deployment mode;

[0089] When the reasoning index parameter is less than or equal to the deployment mode threshold, it is determined that the service deployment mode corresponding to the model reasoning service is a joint deployment mode.

[0090] Optionally, the second deployment mode determining unit is specifically configured to:

[0091] Based on a greedy algorithm, referring to the scheduling strategy that maximizes resource utilization, the deployment proportions of the joint deployment mode and the card sharing deployment mode in the service deployment mode are determined respectively.

[0092] or,

[0093] The first resource utilization rate corresponding to the joint deployment method and the second resource utilization rate corresponding to the card sharing deployment method are calculated respectively. When the first resource utilization rate is greater than the second resource utilization rate, the joint deployment method is adopted; when the first resource utilization rate is less than or equal to the second resource utilization rate, the card sharing deployment method is adopted.

[0094] The model reasoning service deployment device provided in the embodiment of the present invention can execute the model reasoning service deployment method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] Embodiment 4

[0096] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0097] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0098] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0099] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the model reasoning service deployment method.

[0100] In some embodiments, the model reasoning service deployment method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model reasoning service deployment method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the model reasoning service deployment method in any other appropriate manner (e.g., by means of firmware).

[0101] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0103] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0106] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0107] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0108] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model reasoning service deployment method, characterized in that: include: During the model inference process, obtain the inference indicator parameters of all model inference services in the computing resource cluster; For each of the model reasoning services, a threshold is determined according to the reasoning indicator parameters corresponding to the model reasoning service and each pre-built mode, and a target scheduling mode corresponding to the model reasoning service is determined, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode; Based on the service deployment strategy in the target scheduling mode, the service deployment mode corresponding to the model reasoning service is determined, and based on the service deployment mode, the model reasoning service is deployed.

2. The method according to claim 1, characterized in that: The attributes of the mode determination threshold include at least: the average number of requests with a time period as a window, the average response time with a percentile as a window, the longest response time with a time period as a window, always using the default scheduling method within a specified time, and always not using the default scheduling method within a specified time.

3. The method according to claim 1, characterized in that The step of determining a threshold value according to the inference indicator parameter corresponding to the model inference service and each pre-built mode, and determining a target scheduling mode corresponding to the model inference service, includes: The inference indicator parameter and the mode determination threshold are compared and processed, and the target scheduling mode corresponding to the model inference service is determined according to the matching result between the inference indicator parameter and the mode determination threshold.

4. The method according to claim 1, characterized in that: The method comprises: When the target scheduling mode is the service quality priority mode, the service deployment mode includes an independent deployment mode and a joint deployment mode, wherein the independent deployment mode is that a single model inference service occupies a computing resource in the computing resource cluster alone, and the joint deployment mode is that multiple model inference services of the same inference type using the same inference service framework occupy the same computing resource in the computing resource cluster in a shared context manner; When the target scheduling mode is the resource utilization priority mode, the service deployment method includes a joint deployment method and a card sharing deployment method, wherein the card sharing deployment method is that multiple model inference services occupy the same computing resource in the computing resource cluster in a time slice rotation manner.

5. The method according to claim 4, characterized in that The determining, based on the service deployment strategy in the target scheduling mode, a service deployment mode corresponding to the model reasoning service includes: In the case where the target scheduling mode is a service quality priority mode, determining a service deployment mode corresponding to the model reasoning service based on a deployment mode threshold of the service quality priority mode and the reasoning indicator parameter; When the target scheduling mode is a resource utilization priority mode, the service deployment mode corresponding to the model reasoning service is determined based on a scheduling strategy that maximizes resource utilization.

6. The method according to claim 5, characterized in that The determining of the service deployment mode corresponding to the model reasoning service based on the deployment mode threshold of the service quality priority mode and the reasoning indicator parameter includes: When the inference index parameter is greater than the deployment mode threshold, determining that the service deployment mode corresponding to the model inference service is an independent deployment mode; When the reasoning index parameter is less than or equal to the deployment mode threshold, it is determined that the service deployment mode corresponding to the model reasoning service is a joint deployment mode.

7. The method according to claim 5, characterized in that The scheduling strategy based on maximizing resource utilization determines the service deployment mode corresponding to the model reasoning service, including: Based on a greedy algorithm, referring to the scheduling strategy that maximizes resource utilization, the deployment proportions of the joint deployment mode and the card sharing deployment mode in the service deployment mode are determined respectively. or, The first resource utilization rate corresponding to the joint deployment method and the second resource utilization rate corresponding to the card sharing deployment method are calculated respectively. When the first resource utilization rate is greater than the second resource utilization rate, the joint deployment method is adopted; when the first resource utilization rate is less than or equal to the second resource utilization rate, the card sharing deployment method is adopted.

8. A model reasoning service deployment device, characterized in that: include: The inference indicator parameter acquisition module is used to obtain the inference indicator parameters of all model inference services in the computing resource cluster during the model inference process; A target scheduling mode determination module is used to determine, for each of the model reasoning services, a target scheduling mode corresponding to the model reasoning service according to the reasoning indicator parameters corresponding to the model reasoning service and the threshold values ​​of each pre-built mode, wherein the target scheduling mode includes a service quality priority mode and a resource utilization priority mode; The service deployment mode determination module is used to determine the service deployment mode corresponding to the model reasoning service based on the service deployment strategy in the target scheduling mode, and to deploy the model reasoning service based on the service deployment mode.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model reasoning service deployment method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model reasoning service deployment method described in any one of claims 1 to 7 when executed.

Images