Service resource management method and device, electronic equipment, medium and product

Abstract

The invention provides a service resource management method and device, electronic equipment, a medium and a product, and belongs to the technical field of computers, and the method comprises the following steps: determining load data of a large language model service and a copy number return value of the large language model service at the current moment; determining a target copy number of the large language model service according to the load data and the copy number return value; and performing service resource scaling management on the large language model service according to the target copy number. According to the method, the load data with high service relevance is adopted as the elastic scaling measurement index, the service concurrency capability can be accurately measured, the target copy number is determined in combination with the time-based copy number return value, the accuracy of the elastic scaling opportunity is effectively improved, and the user experience is improved. Therefore, service resource scaling management can be timely and accurately carried out on the large language model service according to the target copy number, and GPU resource waste is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a service resource management method, device, electronic device, medium and product. Background Art

[0002] Currently, Kubernetes has become the core orchestration platform in the field of cloud computing. As an emerging technology, the deployment, management and maintenance of large language models are mostly based on the Kubernetes orchestration platform. Among them, Kubernetes is an open source container orchestration platform for automated deployment, expansion and management of containerized applications.

[0003] Elastic scaling is the core capability of the Kubernetes orchestration platform. For large language model services, GPU-related performance indicators or service queries per second (QPS) data are usually used to scale services. However, in large language model scenarios, GPU-related performance indicator data cannot be used to measure the concurrency of services, and service QPS data is greatly distorted in large language model long-link scenarios. Therefore, using GPU performance data and QPS data as the basis for elastic scaling has problems such as low business relevance, inaccurate elastic scaling timing, and easy waste of GPU resources. Summary of the invention

[0004] The present application provides a service resource management method, device, electronic device, medium and product, which are used to solve the problems of low business relevance, inaccurate elastic scaling timing and easy waste of GPU resources in the current use of GPU performance data and QPS data as the basis for elastic scaling measurement, so as to improve the low business relevance, improve the accuracy of elastic scaling timing and reduce GPU resource waste.

[0005] The present application provides a service resource management method, the method comprising: determining load data of a large language model service and a return value of the number of copies of the large language model service at a current moment; determining a target number of copies of the large language model service according to the load data and the return value of the number of copies; and performing scaling management of service resources of the large language model service according to the target number of copies.

[0006] According to a service resource management method provided by the present application, the load data includes the number of real-time links and the first byte response delay; the target number of copies of the large language model service is determined based on the load data and the copy number return value, including: based on the real-time link number in the load data, combined with the expected number of links and the current number of copies, determining a first expected number of copies; based on the first byte response delay in the load data, combined with the expected response delay and the current number of copies, determining a second expected number of copies; based on the first expected number of copies, the second expected number of copies and the copy number return value, determining the target number of copies of the large language model service.

[0007] According to a service resource management method provided by the present application, the first expected number of copies is determined based on the number of real-time links in the load data, combined with the expected number of links and the current number of copies, including: performing a ratio operation on the number of real-time links in the load data and the expected number of links to obtain a first result; multiplying the first result by the current number of copies to obtain a second result; and rounding up the second result to obtain the first expected number of copies.

[0008] According to a service resource management method provided by the present application, the target number of copies of the large language model service is determined based on the first expected number of copies, the second expected number of copies and the copy number return value, including: determining the maximum value, minimum value or average value among the first expected number of copies, the second expected number of copies and the copy number return value as the target number of copies of the large language model service.

[0009] According to a service resource management method provided by the present application, the scaling management of service resources for the large language model service according to the target number of copies includes: determining a copy number difference between the target number of copies and the current number of copies; comparing the copy number difference with a preset numerical range to obtain a comparison result; and performing scaling management of service resources for the large language model service according to the comparison result.

[0010] According to a service resource management method provided by the present application, the return value of the number of copies of the large language model service at the current moment is the current number of copies when the preset trigger condition is not met at the current moment, and is the third expected number of copies when the preset trigger condition is met at the current moment.

[0011] The present application also provides a service resource management device, including a load balancer, a monitoring component, a monitoring adapter component, an elastic scaling controller component and a computer program stored on the elastic scaling controller component. When the computer program is executed by a processor, it implements any of the service resource management methods described above.

[0012] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the service resource management method described in any one of the above is implemented.

[0013] The present application also provides a medium, which is a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the service resource management method described in any one of the above is implemented.

[0014] The present application also provides a product, which is a computer program product, including a computer program, and when the computer program is executed by a processor, it implements any of the service resource management methods described above.

[0015] The service resource management method, device, electronic device, medium and product provided by the present application, by determining the load data of the large language model service and the return value of the number of copies of the large language model service at the current moment; it is possible to timely and accurately determine the target number of copies of the large language model service according to the load data and the return value of the number of copies; and then the service resource scaling management of the large language model service can be performed according to the target number of copies. Since the load data with strong business relevance is used as the elastic scaling measurement indicator, the business concurrency capability can be accurately measured, and the target number of copies is determined in combination with the time-based number of copies return value, which effectively improves the accuracy of the elastic scaling timing, thereby timely and accurately managing the service resource scaling of the large language model service according to the target number of copies, reducing the waste of GPU resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flowchart of the service resource management method provided by this application.

[0018] Figure 2 It is a schematic diagram of the application scenario of the service resource management method provided in this application.

[0019] Figure 3 It is a structural diagram of the service resource management device provided in this application.

[0020] Figure 4 This is a schematic diagram of a real-time link number collection scenario of the service resource management method provided in this application.

[0021] Figure 5 This is a schematic diagram of the first byte response delay monitoring scenario of the service resource management method provided in this application.

[0022] Figure 6 This is a schematic diagram of the first byte response delay collection process of the service resource management method provided in this application.

[0023] Figure 7 It is a logical schematic diagram of the elastic scaling controller of the service resource management method provided in this application.

[0024] Figure 8 It is a schematic diagram of the structure of the electronic device provided by this application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0027] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] Combine the following Figure 1-Figure 8 Describe the service resource management method, device, electronic device, medium and product of the present application.

[0029] Figure 1 It is a flowchart of the service resource management method provided by this application, such as Figure 1 As shown, the method includes the following steps 100 to 300.

[0030] Step 100: Determine the load data of the large language model service and the return value of the number of replicas of the large language model service at the current moment.

[0031] Step 200: Determine the target number of replicas for the large language model service based on the load data and the return value of the number of replicas.

[0032] Step 300: Perform scaling management of service resources for the large language model service according to the target number of replicas.

[0033] Among them, the return value of the number of copies of the large language model service at the current moment is the current number of copies when the preset trigger condition is not met at the current moment, and is the third expected number of copies when the preset trigger condition is met at the current moment.

[0034] Furthermore, the load data includes the number of real-time links and the first byte response delay; according to the load data and the return value of the number of replicas, the target number of replicas of the large language model service is determined, including: Determine a first expected number of replicas based on the number of real-time links in the load data and the expected number of links and the current number of replicas; Based on the first byte response delay in the load data, combined with the expected response delay and the current number of replicas, determine the second expected number of replicas; Based on the first expected number of replicas, the second expected number of replicas, and the returned value of the number of replicas, a target number of replicas for the large language model service is determined.

[0035] Further, based on the real-time link number in the load data, combined with the expected link number and the current number of replicas, determining a first expected number of replicas includes: Performing a ratio operation on the real-time link number in the load data and the expected link number to obtain a first result; Multiply the first result by the current number of replicas to obtain the second result; The second result is rounded up to obtain the first expected number of replicas.

[0036] Further, based on the first expected number of replicas, the second expected number of replicas, and the returned value of the number of replicas, determining the target number of replicas for the large language model service includes: The maximum value, minimum value or average value among the first expected number of replicas, the second expected number of replicas and the returned value of the number of replicas is used to determine the target number of replicas for the large language model service.

[0037] Furthermore, the large language model service is scaled according to the target number of replicas, including: Determine the difference between the target number of copies and the current number of copies; Compare the copy number difference with the preset value range to obtain the comparison result; The service resources of the large language model service are managed on a scalable basis based on the comparison results.

[0038] Figure 2 This is a schematic diagram of an application scenario of the service resource management method provided by this application. Figure 2 As shown, the service resource management method provided in the embodiment of the present application can be applied to a service resource management device including a load balancer, a monitoring component, a monitoring adapter component, and an elastic scaling controller component, wherein the service resource management device can be set on a Kubernetes cluster.

[0039] Specifically, the application can deploy a large language model (hereinafter referred to as a large model) service on a computing node (also referred to as a Work node) in a Kubernetes cluster, deploy a load balancer on a load balancing node in the Kubernetes cluster, and proxy the traffic of the large model service to the Pod instance of the large model service through the load balancer. Among them, Pod is the basic deployment unit in Kubernetes.

[0040] Deploy monitoring components on the management node (also called Master node) in the Kubernetes cluster to collect business indicators of large model services such as the number of real-time links exposed by the load balancer and the first byte response delay (Time To First Token, TTFT). Deploy monitoring adapter components (hereinafter referred to as monitoring adapters) and elastic scaling controller components on the management node in the Kubernetes cluster to collaboratively calculate and adjust the number of replicas of large model services. Among them, the elastic scaling controller component of the present application may include a load elastic scaling controller and a time elastic scaling controller (hereinafter referred to as a time scaling controller).

[0041] In addition, an artificial intelligence (AI) computing power platform can be deployed. Users can use the AI ​​computing power platform to send load elastic scaling policies or time elastic scaling policies to the corresponding elastic scaling controller.

[0042] This application can realize non-invasive large-model load indicator monitoring and collection capabilities and multi-strategy elastic scaling based on Kubernetes' Ingress-Nginx and Operator technologies. Ingress-Nginx is a commonly used ingress controller in Kubernetes, which is used to manage the traffic of external access to services in the cluster. Operator is a method of extending Kubernetes by encoding the management logic of the application as a controller to automate complex operation and maintenance tasks.

[0043] This application uses Lua to expand the capabilities of the Nginx load balancer, customize the large model forwarding algorithm of the load balancer, control the forwarding strategy of the large model streaming request, and realize the statistics and monitoring capabilities of the number of links and TTFT indicators of the large model service. Among them, Lua is a lightweight and efficient scripting language. In addition, Nginx is a high-performance server and reverse proxy server. Lua scripts can be embedded in Nginx to achieve more complex logic and custom processing.

[0044] This application uses Kubernetes' Operator technology to expand the capabilities of elastic scaling components, provide elastic scaling strategy selection based on load and time dimensions, enhance the ability to control expansion and contraction behaviors, improve the accuracy of elastic scaling in large model scenarios, and ensure the accuracy and efficiency of elastic scaling of large model services.

[0045] Figure 3 is a schematic diagram of the structure of the service resource management device provided by this application, such as Figure 3As shown, the load balancer in this application mainly implements the large model load forwarding algorithm, real-time link statistics and TTFT indicator collection, and exposes monitoring indicators through the monitoring module for collection by the monitoring component in the cluster (also called Prometheus).

[0046] The monitoring adapter is a bridge between the load elastic scaling controller and the monitoring component. It connects the monitoring query statements to the Kubernetes standard metrics API to achieve the query capabilities of basic indicators (CPU, memory), custom indicators (Kubernetes resource objects related) and external indicators (other types of monitoring indicators). Among them, metricsAPI is an interface provided by Kubernetes for obtaining and monitoring resource usage and performance indicators in the cluster.

[0047] The load elastic scaling controller is a large-model elastic scaling control executor based on load data, which implements capabilities such as indicator query, replica calculation, and expansion and contraction behavior control.

[0048] The time elastic scaling controller is an elastic scaling control executor based on the time strategy, which realizes the ability to register scheduled tasks and control the expansion and contraction behavior.

[0049] The collection of real-time connection numbers in this application is mainly achieved by customizing the load balancer logic in Kubernetes.

[0050] Figure 4 This is a schematic diagram of a real-time link number collection scenario of the service resource management method provided by this application, such as Figure 4 As shown, specifically, the load balancer will monitor the changes of the resource object (APIServer) on Kubernetes through the ingress-nginx-controller component, and inject the available backend list into the backend list data of the Lua module (process 1 and 2). Among them, APIServer (API server) is a service component for processing requests and responses, and usually plays a key role in microservice architecture or cloud native applications. It is responsible for receiving client requests, performing corresponding business logic processing, and returning the results to the client.

[0051] Furthermore, each time a request enters nginx, the custom Lua load balancing algorithm module is called through the balancer_by_lua_block instruction in lua-nginx-module, and the nested polling or minimum link number algorithm is used. The nested algorithm selects the request forwarding backend address from the backend list data, and at the same time calls the Lua link monitoring module to complete the backend address link establishment information record (process 3-7).

[0052] Nginx selects an address based on the extended load algorithm, establishes a link, and forwards the request to the Pod instance of the backend large model service. The large model request is returned to the user in a streaming response (process 8-10).

[0053] After the request response is completed, the connection is disconnected, and the Nginx process will pass through the log_by_lua_block extension point, where the monitoring module is called to reduce the number of current backend connections (processes 11 and 12).

[0054] The link number monitoring module periodically (for example, every 30 seconds) pushes the current real-time link information to the ingress-nginx-controller component through the socket link. The ingress-nginx-controller component converts the indicators into standard metrics interfaces, and then Prometheus can actively complete the interface data capture (process 13, 14). Therefore, the real-time link number of the large language model service can be determined as one of the load data.

[0055] It should be noted that the first byte response delay is a key quality evaluation indicator for large model services. The collection of this indicator currently requires intrusive modification of the inference framework of large model services, which is costly and has a great impact on the business.

[0056] Specifically, the request of the big model service is responded to using the block transmission protocol. When the user request is forwarded to the big model server, the big model server inference engine responds directly.

[0057] When the client receives the response, it will maintain a long connection and wait for the server to continue to output data until it receives a connection closing signal from the server.

[0058] The inference engine passes the conversation content to the GPU, enters the large model Prefill phase, generates the first byte, and then the inference engine pushes the first byte to the client. The Prefill phase is a key step in the large model inference process, which involves preloading input data (such as conversation content) into the GPU to enable more efficient subsequent calculations and inferences.

[0059] The GPU continues to output content, and the inference engine pushes the subsequent byte responses of the conversation to the client.

[0060] After the GPU responds, the inference engine ends the long connection and the session ends.

[0061] TTFT is the time from when the request is sent to when the first byte of the response is completed in the Prefill phase. There are three optional points for this time statistics: Client-side statistics record the time of request and first byte response on the client side and calculate TTFT data. However, there are problems such as client modification and data dispersion that are difficult to collect.

[0062] Statistics on the inference engine side record request and response time on the inference engine. However, there are many different inference frameworks in the industry. Metrics collection requires customized development of multiple different inference frameworks. In addition, containers cause the inference framework to be out of touch with the community and become an isolated version for internal use. This is not conducive to the introduction of the latest technology and community costs, and affects the rapid iteration of large model technology.

[0063] Statistics on the service gateway side record request and response time at the gateway layer, which can shield various backend differences. The service gateway is located inside the cluster, making data collection convenient.

[0064] Based on this, this application can achieve key indicator monitoring of large model services without intruding on the business or changing the business process.

[0065] Specifically, this application is based on Nginx+Lua extended logic support, does not invade Nginx native operation logic, and is mainly completed using two extension points: access_by_lua_block and body_filter_by_lua_block.

[0066] Figure 5 This is a schematic diagram of the first byte response delay monitoring scenario of the service resource management method provided by this application, such as Figure 5 As shown, the request is executed inside the Nginx load balancer as follows: The request for the large model service is sent from the client. The Nginx process processes the request and enters the ngx_access stage. The Lua handler extended by the access_by_lua_block extension point is executed, which mainly completes three operations: 1. Generates a request id (identification code) and sets the change amount to the nginx variable of the current request. 2. Records the start time of request processing based on the id.

[0067] Nginx then forwards the request to the Pod instance of the large model service to process the request, and the response result is returned to the client through Nginx. In the Content phase, the Nginx service calls the TTFT extension program through the body_filter_by_lua_block extension point (each time the block transfer protocol pushes data to the client, it will pass through this extension point) to complete the following four operations: 1. Parse the request response content, 2. Get the request id, 3. Confirm whether the request has completed the first byte response, and 4. For requests that have not completed the first byte response, record the first byte response time. Among them, in the request processing flow of Nginx, the Content phase is the key link for generating and sending response content.

[0068] Figure 6 This is a schematic diagram of the first byte response delay collection process of the service resource management method provided by this application, such as Figure 6 As shown, the TTFT Monitor (monitoring) module in the monitoring component periodically (for example, every 30 seconds) collects TTFT data of all completed requests of the current service from nginx according to the above structure data, and then passes the statistical records to the nginx-ingress-controller component through the socket, and the nginx-ingress-controller component is exposed as a standard metrics API indicator so that Prometheus can capture monitoring data. Among them, the socket is a programming interface for network communication, which allows data exchange between different hosts or processes.

[0069] It should be further explained that native Kubernetes resource objects only support elastic scaling of load types. When monitoring indicators drop, a scaling-down operation will occur. However, the business volume is still large at this time, and large model services start slowly due to large models and images. Frequent scaling-up and scaling-down operations will cause service quality jitters.

[0070] This application aims to address the problems existing in existing capabilities and designs elastic scaling capabilities that meet the needs of large model services based on the Operator mechanism of Kubernetes.

[0071] Figure 7 This is a logical schematic diagram of the elastic scaling controller of the service resource management method provided by this application, such as Figure 7 As shown, the elastic scaling controller of this application supports two sources of replica calculation data: load and time. In addition to the number of real-time links and the first byte response delay, load data can also support central processing unit, memory or other cluster monitoring indicators. In the elastic scaling of large model services, three main types are supported: cmos_llm_ttft (first byte response delay) and cmos_peer_conn (number of real-time links) two load sources and cornjob time source. Among them, cronJob is a scheduled expansion and contraction task based on time control.

[0072] The load expansion and contraction tasks in this application. Among them, two customized indicators cmos_peer_conn and cmos_llm_ttft are supported for large model services, and the direction of the indicator is declared through direction. The default is to support expansion and contraction, up only supports expansion, and down only supports contraction.

[0073] Specifically, the expected number of replicas is calculated based on the load data using the following formula in this application: Expected number of replicas = ceil[current number of replicas * (current index / expected index)]; Among them: ceil is rounded up.

[0074] When the direction of the indicator is declared as up in the configuration, if the expected number of replicas is less than the current number of replicas, the current number of replicas is used as the final calculation result of the indicator; if it is greater than the current number of replicas, the calculation result is used. The opposite is true for down.

[0075] The cronjob data returns the number of copies based on the trigger time and duration. If the cronjob has not arrived, the current number of copies is used.

[0076] The replica calculation decider determines the target number of replicas calculated and selected at the current time based on the specified algorithm (maximum value max, minimum value min, or average value avg, etc.), and inserts the output into the record table. The time record table records the replica calculation data within ten minutes by default.

[0077] The execution decision calculates the final executed copy data in the record table according to the specified algorithm (maximum value, minimum value or average value, usually the maximum value), and then hands it over to the executor to complete the expansion and contraction of the service, thereby completing the scaling management of service resources for the large language model service.

[0078] It should also be noted that each Pod instance (also referred to as a backend) in the large language model service of this application can be considered as a replica, and the large language model service can include one or more replicas.

[0079] Based on this, the present application can collect the load data of the large language model service and the return value of the number of copies of the large language model service at the current moment based on the monitoring component. Among them, the load data includes the number of real-time links and the first byte response delay. In addition, the return value of the number of copies of the large language model service at the current moment is the current number of copies (that is, the current number of copies is returned before the trigger time) when the preset trigger condition (such as trigger time and duration) is not reached at the current moment, and is the third expected number of copies (that is, the preset expected number of copies is returned within the duration after the trigger time) when the preset trigger condition is reached at the current moment.

[0080] Furthermore, a ratio operation may be performed between the number of real-time links in the load data and the expected number of links to obtain a first result; the first result may be multiplied by the current number of replicas to obtain a second result; and the second result may be rounded up to obtain a first expected number of replicas.

[0081] And, perform a ratio operation on the first byte response delay in the load data and the expected response delay to obtain a third result; multiply the third result by the current number of copies to obtain a fourth result; round up the fourth result to obtain a second expected number of copies.

[0082] Further, according to a pre-specified algorithm, the maximum value, minimum value or average value among the first expected number of copies, the second expected number of copies and the return value of the number of copies is determined as the target number of copies for the large language model service. It should be noted that for the first expected number of copies, the second expected number of copies and the return value of the number of copies as the third expected number of copies, when the direction of the indicator is declared as up in the configuration, if the corresponding expected number of copies is less than the current number of copies, the current number of copies is used as the final calculation result of the indicator, and if the corresponding expected number of copies is greater than the current number of copies, the expected number of copies is used as the final calculation result of the indicator. When the direction of the indicator is declared as down in the configuration, if the corresponding expected number of copies is less than the current number of copies, the expected number of copies is used as the final calculation result of the indicator, and if the corresponding expected number of copies is greater than the current number of copies, the current number of copies is used as the final calculation result of the indicator.

[0083] Furthermore, the target number of replicas can be handed over to the executor for service expansion and contraction.

[0084] When performing scaling management of service resources, this application can set a jitter interval as a preset value interval to avoid constantly scaling the service up and down when the expected number of replicas fluctuates. If the expected number is 2 and the jitter interval is 1, the number of replicas will not be adjusted within the current indicator range of 1 to 3.

[0085] Therefore, when scaling services, the difference between the target number of replicas and the current number of replicas can be calculated, and the difference in the number of replicas can be compared with a preset numerical range to obtain a comparison result; if the difference in the number of replicas is within the preset numerical range, no adjustment is made to the service resources of the large language model service; if the difference in the number of replicas is greater than the maximum value of the preset numerical range, the service resources of the large language model service are expanded according to the difference in the number of replicas; if the difference in the number of replicas is less than the minimum value of the preset numerical range, the service resources of the large language model service are scaled down according to the difference in the number of replicas.

[0086] The service resource management method provided in the embodiment of the present application determines the load data of the large language model service and the return value of the number of copies of the large language model service at the current moment; it makes it possible to timely and accurately determine the target number of copies of the large language model service based on the load data and the return value of the number of copies; and then the service resource scaling management of the large language model service can be performed according to the target number of copies. Since the load data with strong business relevance is used as an elastic scaling measurement indicator, the business concurrency capability can be accurately measured, and the target number of copies is determined in combination with the time-based number of copies return value, which effectively improves the accuracy of the elastic scaling timing, thereby timely and accurately managing the service resource scaling of the large language model service according to the target number of copies, reducing the waste of GPU resources.

[0087] This application combines Kubernetes' load balancer and elastic scaling technology to monitor the concurrent connections of large model services and TTFT indicators, collect indicators that can accurately measure business concurrency capabilities, and expand elastic scaling capabilities. At the same time, it supports large model service expansion and contraction strategies based on load and time. Each strategy can independently control the service expansion and contraction behavior, and realize the combination of service expansion capability based on the real-time number of links as load and service contraction capability based on time when the business is low, forming a large model elastic scaling capability with accurate timing control, high execution efficiency, and strong expansion capability, avoiding the jitter of large model service quality caused by frequent expansion and contraction of the business, and ensuring that the large model service quality and concurrency capability meet business needs.

[0088] Furthermore, compared with traditional expansion and contraction, this application has the following advantages: 1. High accuracy: Using indicators with strong business relevance to large models for elastic scaling has a higher elastic scaling accuracy and can make more reasonable and full use of GPU resources; 2. Non-intrusive: The large-scale model business indicator collection provided by this application does not require intrusive modification of business processes, which is more friendly to business development; 3. Good compatibility: The service gateway is customized to complete the collection of concurrent indicators of large model business, shielding the differences in business reasoning framework and heterogeneous intelligent computing resources, and is compatible with mainstream large model reasoning framework and intelligent computing resources; 4. Easy to expand: The overall elastic scaling solution can be flexibly migrated to other monitoring indicators that can be used to measure business load or make comprehensive decisions based on multi-dimensional indicators.

[0089] The service resource management device provided in the present application is described below. The service resource management device described below and the service resource management method described above can be referenced to each other.

[0090] Furthermore, the present application also provides a service resource management device.

[0091] The service resource management device includes a load balancer, a monitoring component, a monitoring adapter component, an elastic scaling controller component, and a computer program stored on the elastic scaling controller component. When the computer program is executed by the load balancer, the computer can execute the service resource management method provided by the above methods, and the method includes: determining the load data of the large language model service and the return value of the number of copies of the large language model service at the current moment; Determining a target number of replicas for the large language model service according to the load data and the returned value of the number of replicas; The service resources of the large language model service are managed for scalability according to the target number of copies.

[0092] Figure 8An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the service resource management method, which includes: determining the load data of the large language model service and the number of copies of the large language model service at the current moment; Determining a target number of replicas for the large language model service according to the load data and the returned value of the number of replicas; The large language model service is managed by scaling service resources according to the target number of replicas.

[0093] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0094] On the other hand, the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the service resource management method provided by the above methods, the method includes: determining the load data of the large language model service and the return value of the number of copies of the large language model service at the current moment; Determining a target number of replicas for the large language model service according to the load data and the returned value of the number of replicas; The large language model service is managed by scaling service resources according to the target number of replicas.

[0095] In another aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the service resource management method provided by the above methods, the method comprising: determining the load data of the large language model service and the return value of the number of replicas of the large language model service at the current moment; Determining a target number of replicas for the large language model service according to the load data and the returned value of the number of replicas; The large language model service is managed by scaling service resources according to the target number of replicas.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A service resource management method, characterized in that: The method comprises: Determine the load data of the large language model service and the return value of the number of replicas of the large language model service at the current moment; Determining a target number of replicas for the large language model service according to the load data and the returned value of the number of replicas; The service resources of the large language model service are managed for scalability according to the target number of copies.

2. The service resource management method according to claim 1, characterized in that: The load data includes the number of real-time links and the first byte response delay; and determining the target number of copies of the large language model service according to the load data and the copy number return value includes: Determine a first expected number of replicas based on the number of real-time links in the load data and in combination with the expected number of links and the current number of replicas; Determine a second expected number of replicas based on a first byte response delay in the payload data, combined with an expected response delay and a current number of replicas; Based on the first expected number of replicas, the second expected number of replicas, and the replica number return value, a target number of replicas for the large language model service is determined.

3. The service resource management method according to claim 2, characterized in that: The determining of a first expected number of replicas based on the number of real-time links in the load data and in combination with the expected number of links and the current number of replicas includes: Performing a ratio operation on the real-time link number in the load data and the expected link number to obtain a first result; Multiplying the first result by the current number of replicas to obtain a second result; The second result is rounded up to obtain a first expected number of copies.

4. The service resource management method according to claim 2, characterized in that: The determining the target number of copies of the large language model service based on the first expected number of copies, the second expected number of copies, and the copy number return value includes: The maximum value, minimum value or average value among the first expected number of copies, the second expected number of copies and the returned value of the number of copies is determined as the target number of copies for the large language model service.

5. The service resource management method according to claim 1, characterized in that: The scaling management of service resources of the large language model service according to the target number of replicas includes: Determine a copy number difference between the target copy number and the current copy number; Compare the copy number difference with a preset value interval to obtain a comparison result; The service resources of the large language model service are managed scalably according to the comparison result.

6. The service resource management method according to claim 1, characterized in that: The return value of the number of copies of the large language model service at the current moment is the current number of copies when the preset trigger condition is not met at the current moment, and is the third expected number of copies when the preset trigger condition is met at the current moment.

7. A service resource management device, characterized in that: It includes a load balancer, a monitoring component, a monitoring adapter component, an elastic scaling controller component and a computer program stored on the elastic scaling controller component, and is characterized in that when the elastic scaling controller component executes the computer program, it implements the service resource management method as described in any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the service resource management method according to any one of claims 1 to 6 is implemented.

9. A medium, the medium being a non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, the service resource management method according to any one of claims 1 to 6 is implemented.

10. A product, the product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the service resource management method according to any one of claims 1 to 6 is implemented.

Images