Large language model load balancing method and device, electronic equipment, medium and product

Abstract

The invention provides a large language model load balancing method and device, electronic equipment, a medium and a product, and belongs to the technical field of data processing.The method comprises the steps that a service request of an external system is received, and the connection number of each available service instance in a large language model is determined; determining the service instance with the minimum connection number in the available service instances as a target service instance; and forwarding the service request to the instance address of the target service instance. According to the method and the device, the request management is performed for the connection number of each service instance, so that the load balance of each service instance is ensured on the basis of controlling the connection number of each service instance, and the instance-level load balance is realized, and therefore, the streaming transmission interaction quality of the large language model can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a large language model load balancing method, device, electronic device, medium and product. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, large language models have become an important means to achieve new quality productivity, and container cloud platforms have also become the standard of cloud computing platform as a service (PAAS) platforms. The implementation of large language models is basically based on container cloud platforms. Through container cloud platform technology, containerized orchestration and deployment of large language model reasoning services can be realized. However, the carrying capacity of large language model services is relatively low compared with ordinary services due to the large amount of model calculations and lower concurrency. For example, a model with 13B parameters (i.e., 13 billion parameters) usually has only 5 concurrency per card to meet the service quality requirements, and if it exceeds 5 concurrency, the service quality drops sharply. Therefore, a service quality assurance method with the carrying capacity of large language model services as the core is needed. The existing service quality assurance method is mainly the load balancer strategy, but the load balancing strategy provided by the default cloud native load balancer is relatively simple, which cannot meet the load balancing requirements in large language model reasoning scenarios, and cannot provide current limiting protection for backend services in high concurrency scenarios.

[0003] The current load balancing strategy of the load balancer is mainly completed through the polling strategy provided by the cloud native load balancer, and also supports the service-level traffic limiting strategy. However, in the scenario of large language model inference service, due to the particularity of large language model service, the current load balancing strategy will cause problems such as unbalanced load of large language model inference service and low granularity of service quality assurance. Summary of the invention

[0004] The present application provides a large language model load balancing method, device, electronic device, medium and product, which are used to solve the problem that the current load balancing strategy will cause load imbalance of large language model inference service and the service quality assurance granularity is low, so as to improve the interaction quality of large language model streaming transmission.

[0005] The present application provides a large language model load balancing method, the method comprising: receiving a service request from an external system, determining the number of connections of each available service instance in the large language model; determining the service instance with the smallest number of connections among the available service instances as a target service instance; and forwarding the service request to the instance address of the target service instance.

[0006] According to a large language model load balancing method provided by the present application, after receiving a service request from an external system, it also includes: if the number of connections of each service instance in the large language model is greater than or equal to the upper limit threshold of the number of connections of the corresponding service instance, it is determined that there is no available service instance in the large language model, and the service request is rejected.

[0007] According to a large language model load balancing method provided by the present application, after determining that there is no available service instance in the large language model, it also includes: determining that there is a service instance in the large language model whose connection number is less than a target connection number threshold, and determining the service instance with the smallest connection number among the service instances whose connection number is less than the target connection number threshold as the target service instance; the target connection number threshold is determined based on a connection number upper limit threshold and a preset connection number threshold; and forwarding the service request to the instance address of the target service instance.

[0008] According to a large language model load balancing method provided by the present application, before and after receiving a service request from an external system, the following operations are performed based on a preset time interval: each service instance in the large language model is probed for activity; and service instances whose number of probe failures is greater than or equal to a fuse threshold are isolated.

[0009] According to a large language model load balancing method provided in the present application, before and after receiving a service request from an external system, the following operations are performed based on a preset time interval: monitoring resource change events, and performing configuration data changes based on the resource change events.

[0010] According to a large language model load balancing method provided by the present application, when the configuration data is changed based on the resource change event, it includes: if the resource change event is a service instance deletion event, and the service instance indicated by the service instance deletion event has a preset mark, stop forwarding the service request to the service instance indicated by the service instance deletion event; determine that the number of connections of the service instance indicated by the service instance deletion event is zero, clear the preset mark of the service instance indicated by the service instance deletion event, and delete the service instance indicated by the service instance deletion event.

[0011] The present application also provides a large language model load balancing device, including a load balancer and a computer program stored on the load balancer, and when the computer program is executed by a processor, it implements any of the large language model load balancing methods described above.

[0012] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the large language model load balancing method as described above is implemented.

[0013] The present application also provides a medium, which is a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the large language model load balancing method as described above is implemented.

[0014] The present application also provides a product, which is a computer program product, including a computer program, and when the computer program is executed by a processor, it implements any of the large language model load balancing methods described above.

[0015] The large language model load balancing method, device, electronic device, medium and product provided by the present application, after receiving a service request from an external system, determines the number of connections of each available service instance in the large language model; further, the service instance with the smallest number of connections among the available service instances can be determined as the target service instance, and then the service request can be forwarded to the instance address of the target service instance, thereby realizing service scheduling of the large language model. Since request management is performed for the number of connections of each service instance, the load balancing of each service instance is guaranteed on the basis of controlling the number of connections of each service instance, and instance-level load balancing is realized, the quality of streaming interaction of the large language model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flow chart of the large language model load balancing method provided in this application.

[0018] Figure 2 It is a schematic diagram of the application scenario of the large language model load balancing method provided in this application.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0022] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0023] Combine the following Figure 1-Figure 3 The present invention describes the large language model load balancing method, device, electronic device, medium and product.

[0024] Figure 1 It is a flowchart of the large language model load balancing method provided by this application, such as Figure 1As shown, the method includes the following steps 100 to 300.

[0025] Step 100: Receive a service request from an external system and determine the number of connections for each available service instance in the large language model.

[0026] Step 200: Determine the service instance with the smallest number of connections among the available service instances as the target service instance.

[0027] Step 300: Forward the service request to the instance address of the target service instance.

[0028] Furthermore, after receiving the service request from the external system, the method further includes: If the number of connections of each service instance in the large language model is greater than or equal to the upper threshold of the number of connections of the corresponding service instance, it is determined that there is no available service instance in the large language model, and the service request is rejected.

[0029] Further, after determining that there is no available service instance in the large language model, the method further includes: Determine that there are service instances in the large language model whose number of connections is less than a target connection number threshold, and determine the service instance with the smallest number of connections among the service instances whose number of connections is less than the target connection number threshold as the target service instance; Forwards the service request to the instance address of the target service instance.

[0030] Furthermore, before receiving the service request from the external system and after receiving the service request from the external system, the following operations are performed based on a preset time interval: Activate each service instance in the large language model; Service instances whose liveness detection failure times are greater than or equal to the circuit breaker threshold are isolated.

[0031] Figure 2 This is a schematic diagram of an application scenario of the large language model load balancing method provided by this application. Figure 2 As shown, the large language model load balancing method provided in the embodiment of the present application can be applied to a platform including an external system layer, a load balancing layer and an internal application layer.

[0032] Among them, the external system layer includes the external system that calls the inference service of the large language model. The external system accesses the load balancing layer and then forwards it to the service instance of the large language model inside the container cluster.

[0033] The reasoning services of the large language model in this application may include but are not limited to text generation, question-answering system, translation service, code generation, etc.

[0034] The load balancing layer can be provided with a large language model load balancing device, which can include one or more cloud-native load balancers. The cloud-native load balancer includes three parts: a cloud-native Controller, a load balancing Nginx, and a Lua module.

[0035] Among them, the Controller is responsible for interacting with the APIServer of the kubernetes cluster, monitoring service resource changes, and updating the Nginx configuration; the Lua module is mainly responsible for maintaining the Nginx configuration in memory, and is embedded in different stages of Nginx request processing to provide expansion capabilities; the Nginx module accesses the call traffic of the external system, and interacts with the Lua module to obtain the backend address of traffic processing, and forwards the traffic to the service instance corresponding to the backend address. Kubernetes is an open source container orchestration platform for automated deployment, expansion, and management of containerized applications. The API Server of the Kubernetes cluster is the core component in the cluster, responsible for processing API requests.

[0036] Specifically, the cloud-native load balancer can be deployed on multiple Nginx nodes in the container cluster.

[0037] The internal application layer has a containerized deployment and operation of the inference service of the large language model. Each service instance of the inference service corresponds to a Pod instance. Among them, in Kubernetes, Pod is the basic deployment unit.

[0038] The load balancer of the present application may include functions such as load strategy configuration, traffic monitoring, and service assurance strategy for large language models.

[0039] Among them, load policy configuration supports enabling the service quality assurance policy of the large language model for the specified service, and supports configuring the traffic threshold for a single Pod instance. The cloud-native load balancer will dynamically perceive the configuration of the policy.

[0040] Traffic monitoring supports generating traffic metrics for proxy services, including: the number of queries per second (QPS) processed by the service at the service instance level and the number of connections processed by the instance at the service instance level.

[0041] This application can realize non-intrusive, lossless real-time load calculation and statistics for long-connection streaming requests of large language models.

[0042] The quality assurance strategies for large language models include dynamic maintenance of the number of request connections, load algorithms based on the number of connections, and current limiting strategies for the number of request connections.

[0043] Dynamic maintenance of the number of request connections: Request access changes in real time. To meet the needs of real-time computing, the cloud-native load balancer maintains the real-time request connection processing status of each backend service instance in memory.

[0044] Load algorithm based on the number of connections: To improve the service quality of the backend service instances, the cloud-native load balancer uses a load algorithm based on the number of connections to prioritize the service instance with the smallest number of connections to process the request based on the number of request connections of the backend service instances dynamically maintained in the memory.

[0045] Request connection limit strategy: Each backend service instance has an upper limit on request processing. When the upper limit is reached, the cloud native load balancer will filter the corresponding service instances according to the upper limit of the instance's connection number. The service instances that have reached the upper limit of the connection number will no longer accept new service requests for large language models.

[0046] This application can first configure the load policy in the load balancer. Specifically, the relevant configuration of the load policy for service quality assurance is set on the resource object Ingress used for service exposure in the form of annotations. The configuration involves: policy configuration, threshold configuration, and active detection and fuse configuration.

[0047] For the implementation of load strategies, in the container scenario, in order to avoid the problem of NginxUpstream configuration being constantly reloaded and refreshed due to frequent changes in container Pod instances, the Lua module is responsible for Nginx Upstream configuration maintenance by default, dynamically refreshes in memory, and avoids Nginx actively reloading and updating. At the same time, the Lua module implements the polling algorithm by default to provide a load balancing strategy for Nginx. In Nginx, upstream refers to a group of backend servers used to process requests from clients. In Nginx, reload refers to reloading the configuration file without completely stopping the service.

[0048] On this basis, in order to ensure the service quality of large models and improve the quality of streaming interaction, a set of load balancing strategies are specially designed to support the use of application service quality assurance in large model scenarios.

[0049] Specifically, the Lua module is equipped with a Balancer module. A new service quality policy part is added to the Balancer module. A memory space is opened through the lua_shared_dict method to store real-time data on the number of service request connections, including: service name (service_name), instance address (peer_name), number of instance connections (conn_count), instance status (peer_status, the status is 0 when the instance is isolated), instance connection limit (conn_limit), and instance additional connections (conn_extra, optional).

[0050] After the storage space for the number of service request connections is opened, each time a service request enters Nginx, the Balancer module will return the instance address of a backend service instance and increase the number of connections for the service instance by 1, and Nginx will forward the traffic to the address. After Nginx completes this connection proxy, it enters the log writing phase and triggers the Balancer module again, which will decrease the number of connections for the service instance by 1.

[0051] When the Balancer module returns a backend instance address to Nginx to forward and process requests, it will give priority to forwarding large model long connection requests with the smallest number of connections. At the same time, according to the service guarantee policy settings, when filtering instances, service instances whose number of connections has reached the upper limit will be filtered out.

[0052] In order to avoid sudden traffic peaks, the system supports configuring additional acceptable concurrent connection number control based on the current limit (optional capability configuration, users need to tolerate the possibility of some service quality degradation) to handle more traffic and avoid the situation where all requests fail under traffic peaks.

[0053] For the service detection capability, the cloud-native load balancer supports active detection of backend service instances, and implements fuses for abnormal Pod instances based on the fuse threshold to ensure the overall service quality of the application. The detection and fuse actions of the cloud-native load balancer are performed by the Balancer module. The specific configuration includes: Checkpath: The Balancer module initiates a checkpath request based on peer information (IP:Port) and the checkpath. Check period: The Balancer module periodically initiates a check request based on the preset check period time interval. Timeout (checktimeout): After the Balancer module initiates a liveness check request, the request must be completed within the timeout period, otherwise the liveness check request is considered to have failed; Fuse threshold (checkfailure): It is not required by default. When the fuse threshold is filled in, the Balancer module will compare the number of failed liveness requests with the fuse threshold. When it reaches the threshold, the corresponding Pod instance will be fused. The Balancer module will not add the corresponding instance peer to the request processing list during the isolation period, that is, after the service instance triggers the fuse, it will no longer process business requests during the isolation period; peer refers to the backend service instance, which is usually composed of its Internet Protocol (IP) address and port number; Isolation time (checkisolationtime): used in conjunction with the circuit breaker threshold. When the backend service instance detection reaches the upper limit of the circuit breaker threshold and isolation is triggered, the isolation time will determine how long the isolated instance will be isolated.

[0054] Therefore, when the present application receives a service request from an external system through Nginx, it can determine whether there is an available service instance in the large language model through the Lua module. Specifically, the number of connections of each service instance of the large language model can be compared with the upper limit threshold of the number of connections of the corresponding service instance, and the service instance whose number of connections is less than the upper limit of the number of connections of the corresponding service instance is determined as an available service instance. The number of connections of each available service instance is further obtained. For service instances whose number of connections is greater than or equal to the upper limit of the number of connections of the corresponding service instance, it can be determined as an unavailable service instance.

[0055] In one embodiment, if the number of connections of each service instance in the large language model is greater than or equal to the upper limit threshold of the number of connections of the corresponding service instance, that is, each service instance in the large language model is an unavailable service instance, it is determined that there is no available service instance in the large language model, and the service request can be rejected.

[0056] If there are available service instances in the large language model, the number of connections of each available service instance can be compared, and the service instance with the smallest number of connections can be determined as the target service instance, so as to obtain the instance address of the target service instance returned to Nginx through the Lua module, and Nginx forwards the service request to the target service instance according to the instance address, so that the target service instance can process the service request accordingly.

[0057] It should be noted that in order to avoid sudden traffic peaks, if the user configures additional acceptable concurrent connection control, even if each service instance in the large language model is an unavailable service instance, the number of connections of each service instance can be compared with the target connection number threshold of the corresponding service instance, where the target connection number threshold of each service instance is obtained by adding the upper limit threshold of the connection number of the service instance and the connection number threshold set in advance for the service instance.

[0058] If it is determined that there are service instances in the large language model whose number of connections is less than the target connection number threshold of the corresponding service instance, the service instance with the smallest number of connections is selected as the target service instance from the service instances whose number of connections is less than the target connection number threshold of the corresponding service instance. If there are multiple service instances with the smallest number of connections among the service instances whose number of connections is less than the target connection number threshold of the corresponding service instance, one can be randomly selected as the target service instance, so as to obtain the instance address of the target service instance returned to Nginx through the Lua module, and Nginx forwards the service request to the target service instance according to the instance address, so that the target service instance can process the service request accordingly. In this way, more traffic can be processed to avoid the situation where all requests fail under traffic peaks.

[0059] The large language model load balancing method provided in the embodiment of the present application, after receiving a service request from an external system, determines the number of connections of each available service instance in the large language model; further, the service instance with the smallest number of connections among the available service instances can be determined as the target service instance, and then the service request can be forwarded to the instance address of the target service instance, thereby realizing service scheduling of the large language model. Since request management is performed for the number of connections of each service instance, the load balancing of each service instance is guaranteed on the basis of controlling the number of connections of each service instance, and instance-level load balancing is realized, thus improving the quality of streaming interaction of the large language model.

[0060] This application is based on Kubernetes' Controller technology and combines the Lua language to expand the load strategy for ensuring the service quality of large language models. It builds a cloud-native load balancer with the load balancing capability based on the number of backend connections, and implements the ability to manage the number of requested connections for each container service backend instance. On the basis of controlling the number of requested connections for each backend instance, it ensures the load balancing of each service instance as much as possible to ensure the service quality of large models.

[0061] In one embodiment, before receiving the service request from the external system and after receiving the service request from the external system, the following operations are further performed based on a preset time interval: Monitor resource change events and make configuration data changes based on the resource change events.

[0062] Furthermore, when the configuration data is changed based on the resource change event, it includes: If the resource change event is a service instance deletion event, and the service instance indicated by the service instance deletion event has a preset tag, stop forwarding the service request to the service instance indicated by the service instance deletion event; It is determined that the number of connections of the service instance indicated by the service instance deletion event is zero, and a preset mark of the service instance indicated by the service instance deletion event is cleared to delete the service instance indicated by the service instance deletion event.

[0063] It should be noted that the Controller in the cloud-native load balancer monitors changes in Ingress resources and builds a monitoring informer for Ingress resources. In Kubernetes, the monitoring informer is a mechanism for monitoring resource status changes in the cluster.

[0064] Specifically, when an Ingress resource is created or updated, the quality assurance configurations contained in the annotations are collected and submitted to the Lua module. Similarly, when an Ingress resource sends a deletion event, the Controller cleans up the corresponding configuration data in the Lua module. Throughout the entire process, quality assurance configurations are performed in the memory maintained by the Lua module, refreshed in real time, and dynamically effective.

[0065] Furthermore, the present application can also achieve zero-loss release of services.

[0066] In the daily application release and maintenance process, the application Pod instance deletion is involved, such as when the application is released for rolling upgrade, and when the specified application instance is deleted during the application operation and maintenance process. In order to avoid the problem of interruption of the traffic being processed or about to be processed on the application Pod instance when deleting the application Pod instance, the cloud native load balancer provides zero-loss traffic switching capability.

[0067] When a Pod instance of an application is deleted, it will first stop accepting new traffic and wait until the traffic being processed is completed before it is recycled, ensuring zero-loss switching of application traffic.

[0068] Specifically, there are: 1. When deploying an application, add a preset zero-loss traffic switching tag (finalizers tag) to the application. When a Pod instance with the tag is deleted, it will not be cleaned up immediately, but will wait for the Controller corresponding to the tag to perform pre-deletion processing; 2. The user performs a rolling upgrade of the application, which triggers a request to delete the application instance; 3. If the application instance has a finalizers tag, the Pod recycling controller will not clean up the corresponding Pod instance immediately, but will enter the waiting for deletion phase, waiting for the corresponding controller to perform pre-deletion actions; 4. The Controller corresponding to the tag will listen to the Pod deletion event, call the Balancer module periodically to query the number of requested connections of the Pod instance, and notify the Balancer module to mark the Pod instance as unavailable. When the number of Pod instance connections is 0, the finalizers tag will be cleared; 5. After receiving the notification from the Controller, the Balancer module will immediately mark the peer_status of the Pod instance as 0, no longer forward new traffic for the Pod instance, and return the number of connections of the Pod instance; 6. When the finalizers mark of the Pod instance is cleared, the Pod instance is immediately recycled and cleaned up.

[0069] This application can improve the service quality of large language models in release and upgrade scenarios by providing zero-loss publishing capabilities for long connection services.

[0070] This application has high adaptability to large language model scenarios and is close to the usage scenarios of large language model services. It provides instance-level service quality improvement strategies according to large language model scenarios to ensure the service quality of large language models.

[0071] At the same time, it provides active detection and fuse breaking capabilities, providing the gateway with richer large language model business active detection capabilities, quickly and accurately perceive the health status of the large language model backend instances, isolate faulty instances, and ensure the service quality of the large language model.

[0072] It also provides the capability to release services with zero loss and offers zero loss traffic switching at the gateway, covering the entire life cycle of large language model services and avoiding the impact on service quality during service release and rollback.

[0073] This application combines the characteristics of large language models with cloud-native technology capabilities to build a new load strategy that can ensure the quality of large model services and improve the quality of interactive streaming interactive transmission services. On this basis, it realizes zero-loss traffic switching to ensure that the long connections that have been established for large language model services in scenarios such as release and upgrade are not interrupted, and service responses are completed in priority, further improving the quality of large model services.

[0074] The large language model load balancing device provided in the present application is described below. The large language model load balancing device described below and the large language model load balancing method described above can refer to each other.

[0075] Furthermore, the present application also provides a large language model load balancing device.

[0076] The large language model load balancing device includes a load balancer and a computer program stored on the load balancer. When the computer program is executed by the load balancer, the computer can execute the large language model load balancing method provided by the above methods, the method comprising: receiving a service request from an external system, determining the number of connections of each available service instance in the large language model; Determine the service instance with the smallest number of connections among all available service instances as the target service instance; The service request is forwarded to the instance address of the target service instance.

[0077] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the large language model load balancing method, the method comprising: receiving a service request from an external system, determining the number of connections of each available service instance in the large language model; Determine the service instance with the smallest number of connections among all available service instances as the target service instance; The service request is forwarded to the instance address of the target service instance.

[0078] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0079] On the other hand, the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the large language model load balancing method provided by the above methods, the method includes: receiving a service request from an external system, determining the number of connections of each available service instance in the large language model; Determine the service instance with the smallest number of connections among all available service instances as the target service instance; The service request is forwarded to the instance address of the target service instance.

[0080] In another aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the large language model load balancing method provided by the above methods, the method comprising: receiving a service request from an external system, determining the number of connections of each available service instance in the large language model; Determine the service instance with the smallest number of connections among all available service instances as the target service instance; The service request is forwarded to the instance address of the target service instance.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A large language model load balancing method, characterized in that: The method comprises: Receive service requests from external systems and determine the number of connections for each available service instance in the large language model; Determine the service instance with the smallest number of connections among all available service instances as the target service instance; The service request is forwarded to the instance address of the target service instance.

2. The large language model load balancing method according to claim 1, characterized in that: After receiving the service request from the external system, it also includes: If the number of connections of each service instance in the large language model is greater than or equal to the upper threshold of the number of connections of the corresponding service instance, it is determined that there is no available service instance in the large language model, and the service request is rejected.

3. The large language model load balancing method according to claim 2, characterized in that: After determining that there is no available service instance in the large language model, the method further includes: Determine that there are service instances in the large language model whose number of connections is less than a target connection number threshold, and determine the service instance with the smallest number of connections among the service instances whose number of connections is less than the target connection number threshold as the target service instance; the target connection number threshold is determined based on an upper limit threshold of the number of connections and a preset connection number threshold; The service request is forwarded to the instance address of the target service instance.

4. The large language model load balancing method according to claim 1, characterized in that: Before receiving the service request from the external system and after receiving the service request from the external system, the following operations are also performed based on a preset time interval: Activate each service instance in the large language model; Service instances whose liveness detection failure times are greater than or equal to the circuit breaker threshold are isolated.

5. The large language model load balancing method according to claim 1, characterized in that: Before receiving the service request from the external system and after receiving the service request from the external system, the following operations are also performed based on a preset time interval: A resource change event is monitored, and configuration data is changed based on the resource change event.

6. The large language model load balancing method according to claim 5, characterized in that: When the configuration data is changed based on the resource change event, it includes: If the resource change event is a service instance deletion event, and the service instance indicated by the service instance deletion event has a preset mark, stop forwarding the service request to the service instance indicated by the service instance deletion event; Determine that the number of connections of the service instance indicated by the service instance deletion event is zero, clear a preset mark of the service instance indicated by the service instance deletion event, and delete the service instance indicated by the service instance deletion event.

7. A large language model load balancing device, characterized in that: It includes a load balancer and a computer program stored on the load balancer, characterized in that when the load balancer executes the computer program, the large language model load balancing method as described in any one of claims 1 to 6 is implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the large language model load balancing method as described in any one of claims 1 to 6 is implemented.

9. A medium, the medium being a non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, the large language model load balancing method as claimed in any one of claims 1 to 6 is implemented.

10. A product, the product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the large language model load balancing method as claimed in any one of claims 1 to 6 is implemented.

Images