Load balancing method and device for reasoning, electronic equipment and storage medium

Abstract

The application discloses a reasoning load balancing method and device, electronic equipment and storage medium, and relates to the technical field of data processing. The method comprises the following steps: a service gateway receives a reasoning request, forwards the reasoning request to a distributed key-value pair database, and determines a target reasoning server from a plurality of reasoning servers included in a reasoning service cluster, wherein the distributed key-value pair database serves as a load balancing server middleware between the service gateway and the reasoning service cluster, the reasoning request is processed by the target reasoning server, and a reasoning result is fed back. By introducing the distributed key-value pair database as the load balancing server middleware between the service gateway and the reasoning service cluster, the application can uniformly distribute the reasoning request to each reasoning server, so that the hardware resources can be fully utilized to the greatest extent.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a load balancing method, apparatus, electronic device, and storage medium for inference. Background Technology

[0002] The online inference process is as follows: the requester sends an inference request to the service gateway, which selects an inference server based on a load balancing strategy and forwards the inference request to the selected server. The selected inference server then processes the inference request and obtains the inference result. However, the service gateway may experience uneven resource allocation, resulting in insufficient utilization of the inference server's hardware resources. Summary of the Invention

[0003] In view of the above problems, this application proposes a load balancing method, apparatus, electronic device, and storage medium for reasoning to solve the above problems.

[0004] In a first aspect, embodiments of this application provide a load balancing method for inference, the method comprising: a service gateway receiving an inference request; forwarding the inference request to a distributed key-value pair database, and determining a target inference server from a plurality of inference servers included in the inference service cluster, wherein the distributed key-value pair database serves as a load balancing server middleware between the service gateway and the inference service cluster; performing inference processing on the inference request through the target inference server, and feeding back the inference result.

[0005] Secondly, embodiments of this application provide a load balancing device for inference, the device comprising: an inference request receiving module, configured to receive an inference request from a service gateway; an inference server selection module, configured to forward the inference request to a distributed key-value pair database and determine a target inference server from a plurality of inference servers included in the inference service cluster, wherein the distributed key-value pair database serves as a load balancing server middleware between the service gateway and the inference service cluster; and an inference processing module, configured to perform inference processing on the inference request through the target inference server and feed back the inference result.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor performs the above-described method.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the above-described method.

[0008] The inference load balancing method, apparatus, electronic device, and storage medium provided in this application embodiment allow a service gateway to receive an inference request, forward the inference request to a distributed key-value pair database, and determine a target inference server from among multiple inference servers included in the inference service cluster. The distributed key-value pair database acts as a load balancing server middleware between the service gateway and the inference service cluster. The target inference server performs inference processing on the inference request and feeds back the inference result. By introducing a distributed key-value pair database as a load balancing server middleware between the service gateway and the inference service cluster, inference requests can be evenly distributed to each inference server to maximize the utilization of hardware resources. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic diagram of the current online reasoning framework is shown;

[0011] Figure 2 A schematic diagram of the current online inference process is shown;

[0012] Figure 3 A schematic diagram of the polling algorithm is shown;

[0013] Figure 4 A schematic diagram of task scheduling is shown;

[0014] Figure 5 A schematic diagram of the framework of a load balancing method that can be used for inference provided in the embodiments of this application is shown;

[0015] Figure 6 A schematic flowchart of a load balancing method for inference provided in an embodiment of this application is shown;

[0016] Figure 7 A schematic flowchart of a load balancing method for inference provided in an embodiment of this application is shown;

[0017] Figure 8 A schematic flowchart of a load balancing method for inference provided in an embodiment of this application is shown;

[0018] Figure 9 This illustration shows a framework diagram of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uniform;

[0019] Figure 10 This paper illustrates a flowchart of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uniform.

[0020] Figure 11 A schematic flowchart of a load balancing method for inference provided in an embodiment of this application is shown;

[0021] Figure 12 This illustration shows a framework diagram of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uneven;

[0022] Figure 13 This application shows Figure 11 The flowchart of step S430 of the load balancing method shown is illustrated.

[0023] Figure 14 This paper illustrates a flowchart of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uneven.

[0024] Figure 15 A schematic diagram showing the comparison of latency fluctuations before and after using the super-resolution inference service is presented;

[0025] Figure 16 This paper shows a block diagram of a load balancing device for inference provided in one embodiment of the present application;

[0026] Figure 17 A block diagram of an electronic device for performing a load balancing method according to an embodiment of the present application is shown;

[0027] Figure 18 A storage unit for storing or carrying program code implementing a load balancing method according to an embodiment of the present application is shown. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0029] Please see Figure 1 , Figure 1 A schematic diagram of the current online inference framework is shown. Figure 1In this context, Request represents an inference request; Gateway Cluster represents a service gateway cluster; Gateway Instance represents a service gateway; routing rategy represents a routing strategy; traffic strategy represents a traffic strategy; listener represents a listener; update represents an update; serverlist represents a server list; discover represents discovery; Registry represents a registry center; register represents registration; Inference Cluster represents an inference service cluster; inference server represents an inference service; and load balance represents load balancing.

[0030] The Inference Cluster utilizes a microservice architecture, integrating the algorithm inference process with environment images and Graphics Processing Unit (GPU) servers as independent components, which are then deployed in a cluster as atomic services. These inference services are built around algorithmic functions, with each service performing a model inference function.

[0031] The service gateway, acting as a layer of gateway service, is needed before processing inference requests to enhance management and control between the application and the inference service. A service gateway typically includes the following main functions: routing and forwarding, traffic control, protocol conversion (covering network and data protocols), security management, cache management, and log monitoring. The gateway is primarily responsible for request routing and forwarding, protocol conversion, security verification, and load balancing.

[0032] The registry is responsible for managing and maintaining the registration and discovery of inference services. It provides a centralized service registry to store and manage information about all available service instances in the system. Each inference service instance registers its information with the registry, including its service name, network address, port number, and health status. The registry stores this information and provides a query interface for the gateway service to use, assisting the gateway in distributing requests to multiple available inference service instances to achieve load balancing and high availability.

[0033] Please see Figure 2 , Figure 2 A schematic diagram of the current online inference process is shown. (For example...) Figure 2 As shown in the diagram, the process diagram includes the following four processes:

[0034] 1. Inference server launch and registration

[0035] Online: After the Inference Service starts, it enters the "online" state.

[0036] Registration: The inference server initiates a "service register" operation with the registry, registering its own service information (such as service address, port, service capabilities, etc.) with the registry so that other components can discover and use the service.

[0037] 2. Service Gateway Discovery

[0038] Service discovery request: The service gateway initiates a "service discover" request to the registry center in order to obtain information about currently registered inference servers.

[0039] Obtaining the instance IP: The registry center responds to the service gateway's request and returns relevant information such as the "instance IP" of the inference service, allowing the gateway to know the specific network location of the inference service.

[0040] 3. Inference Request Processing

[0041] User request: The user initiates a "request" (inference request) to the service gateway.

[0042] Request forwarding: After receiving an inference request, the service gateway performs a "request forward" operation based on the inference server information obtained from the registry center, forwarding the inference request to the corresponding inference server.

[0043] 4. Response Return

[0044] Processing and Return: After receiving the forwarded request, the inference server processes it. Once the processing is complete, it sends a "return" message to the service gateway, returning the inference result to the service gateway.

[0045] Responding to the user: The service gateway then returns the inference result received from the inference server to the user who initiated the request, completing the entire request-response process.

[0046] The inventors discovered through research that, in practical applications, the above solutions have the following drawbacks:

[0047] First, it is not suitable for low-concurrency scenarios:

[0048] Generative large-scale models suffer from low inference concurrency (<10) on a single inference server, high request processing latency, and limited scale due to high computing power costs. Current service gateways store load balancing strategies, routing strategies, and inference server weight lists locally, catering to service scenarios with high request volume, low latency, and high single-instance throughput. While current solutions can ensure a macroscopically even distribution of requests across different backend nodes, when processing AI inference requests, the characteristics of AI inference service instances (high latency, low concurrency) mean that multiple inference requests forwarded to the same inference server at a given time can cause congestion. This is because each service gateway independently calculates the backend load, and request forwarding between different service gateways within the service gateway cluster can conflict, as exemplified by the most common round-robin algorithm:

[0049] like Figure 3 As shown, there is a three-instance API service gateway cluster. Each instance polls the inference service backend instances sequentially. After the system has been running for a period of time, the polling ranges of 10.210.31.4 and 10.210.31.5 will overlap. If each inference server has only 1 concurrent connection, another inference request will be waiting in the inference server's buffer. If there is a high probability of this, all service gateways may even forward inference requests to the same inference server simultaneously, resulting in high overall service latency.

[0050] Second, task scheduling conflicts:

[0051] In AI inference service scenarios, uneven inference latency can further exacerbate conflicts between inference requests. For example, a super-resolution algorithm inference service might select different pipelines based on factors such as the resolution, focal length, and number of faces in the input image, resulting in fluctuations in inference latency: most images might take 3 seconds to process, while a very small number might take 20 seconds. If a round-robin load balancing strategy is chosen, requests might accumulate on the service instance that has been processing for 20 seconds.

[0052] like Figure 4 As shown, Figure 4 The image shows three input queues: Input queue 1, Input queue 2, and Input queue 3. Each queue contains several tasks. Figure 4 In the upper part, each of the three input queues contains a different number of tasks that are in a pending state.

[0053] exist Figure 4The lower half shows the queue state after some processing. We can see that some tasks in Input queue 1 and Input queue 3 have been processed, reducing the number of tasks in the queues. In Input queue 2, one task is marked as "blocked tasks," indicating that the task encountered obstacles during processing and cannot proceed normally.

[0054] To address the aforementioned problems, the inventors, through extensive research, discovered and proposed the inference load balancing method, apparatus, electronic device, and storage medium provided in the embodiments of this application. By introducing a distributed key-value database as a load balancing server middleware between the service gateway and the inference service cluster, inference requests can be evenly distributed to each inference server to maximize the utilization of hardware resources. The specific inference load balancing method will be described in detail in subsequent embodiments.

[0055] The following will describe the application environment for the load balancing method that can be used for inference provided in the embodiments of this application. Please refer to... Figure 5 , Figure 5 A schematic diagram of the framework of a load balancing method that can be used for inference provided in embodiments of this application is shown. Figure 5 In this context, Request represents an inference request; Gateway Cluster represents a service gateway cluster; Gateway Instance represents a service gateway; route strategy represents a routing strategy; traffic strategy represents a traffic strategy; listener represents a listener; discover represents discovery; Registry represents a registry center; register represents registration; InferenceCluster represents an inference service cluster; inference server represents an inference service; load balance represents load balancing; depend represents dependency; KV DB Cluster represents a distributed key-value database; serverlist represents a server list; input represents an input queue; and output represents an output queue.

[0056] The client, represented by a mobile phone icon, is the initiator of the reasoning request (Request), sending business requests to the system.

[0057] Among them, Gateway Instance (service gateway): receives client requests and contains route strategy to determine the request forwarding path and traffic strategy to manage traffic.

[0058] Among them, the listener listens for changes in the system, such as changes in the registry server information, and updates the internal server list in a timely manner.

[0059] The Registry is where the inference server performs a register operation, and the service gateway obtains the inference server's registration information through a discover operation, thus realizing the registration and discovery management of services.

[0060] The Inference Cluster consists of multiple inference servers responsible for executing inference tasks. A load balancer distributes inference requests forwarded by the service gateway across the inference servers to prevent uneven load distribution.

[0061] KV DB Cluster (Distributed Key-Value Database): Both the service gateway cluster and the inference service cluster depend on it. It stores a list of servers (serverlist), and also includes an input queue (input) and an output queue (output). The input queue temporarily stores tasks to be processed, while the output queue stores inference results, aiding in asynchronous task processing and load balancing.

[0062] Therefore, this application proposes a technical solution for global load balancing of an AI inference service cluster based on a distributed key-value database and an asynchronous automatic processing strategy. Unlike conventional local load balancing strategies that rely on service gateways, this solution uses a distributed key-value database as a load balancing server middleware between the service gateway and the inference service cluster. This ensures that request load is evenly distributed across each inference server in both uniform and non-uniform inference latency scenarios, maximizing the utilization of expensive GPU hardware resources. Optionally, the distributed key-value database may include Redis, etcd, etc., and is not limited thereto.

[0063] Using a distributed key-value database as a load balancing server middleware between the service gateway and the inference service cluster has the following advantages:

[0064] Weight Management: In an inference service cluster, different inference servers may have varying performance. A distributed key-value database records the weight information of each inference server. When forwarding requests, the service gateway uses the weights stored in the distributed key-value database and a specific algorithm (such as weighted round-robin) to distribute inference requests reasonably to different inference servers, allowing high-performance inference servers to handle more tasks and achieving load balancing.

[0065] Task queue management: When inference task processing latency fluctuates significantly, a distributed key-value database can be used to construct input and output queues. The service gateway places received tasks into the input queue. The inference service cluster pulls tasks from the input queue according to certain rules (such as first-in, first-out) for processing. The processed results are placed into the output queue, and the service gateway then retrieves the results from the output queue and returns them to the client. This approach avoids requests being concentrated on a few inference servers and enables asynchronous processing, improving the overall system processing capacity and stability.

[0066] Status information storage: The distributed key-value database can also store real-time status information of the inference server (such as busy or idle status), task processing progress, etc. The service gateway can dynamically adjust the request forwarding strategy based on this information to further optimize load balancing.

[0067] Please see Figure 6 , Figure 6 This illustration shows a flowchart of a load balancing method for inference provided in an embodiment of this application. This method uses a distributed key-value database as a load balancing server middleware between the service gateway and the inference service cluster, enabling inference requests to be evenly distributed across each inference server to maximize the utilization of hardware resources. In a specific embodiment, this load balancing method for inference is applied to, for example... Figure 16 The load balancing device 200 for inference and the electronic device 100 configured with the load balancing device 200 for inference are shown. Figure 17 The following will use an electronic device as an example to illustrate the specific process of this embodiment. It is understood that the electronic device used in this embodiment may include an inference service system, which may include a service gateway, an inference service cluster, and a distributed key-value database. The following will focus on... Figure 6 The process shown will be described in detail. The load balancing method described in the inference method may specifically include the following steps:

[0068] Step S110: The service gateway receives an inference request.

[0069] In this embodiment, the service gateway can receive inference requests. Specifically, when a user has an AI inference need, they can send an inference request through a client, and the service gateway can receive this request.

[0070] In some implementations, upon receiving an inference request, the service gateway can check whether the format of the inference request is correct. For example, it can check whether the parameters of the inference request are complete and whether the data types match. If an error is found in the format of the inference request, an error message can be directly returned to the client to inform the client that the inference request is invalid; if the format of the inference request is correct, the next step can be taken.

[0071] In some implementations, upon receiving an inference request, the service gateway can check whether the protocol of the inference request is the same as the protocol used by the inference server. If the protocol of the inference request is different from that used by the inference server, the service gateway can perform protocol conversion on the inference request, transforming it into a protocol form that the inference server can understand and process; if the protocol of the inference request is the same as that used by the inference server, the next step can be performed.

[0072] In some implementations, before the service gateway receives the inference request, the process may further include: when the inference server starts up and enters the online state, the inference server may initiate a service registration process with the registry center, registering its relevant information with the registry center so that other components can discover and use the inference server. The service gateway initiates a service discovery request to the registry center to obtain relevant information about currently registered inference servers. In response to the service gateway's discovery request, the registry center returns relevant information about the inference server, allowing the service gateway to know the specific network location of the inference server and preparing for subsequent request forwarding.

[0073] Of course, during this process, the service gateway can also monitor changes in the registry center in real time through listeners. When a new inference server is added or an old inference server is removed, the service gateway can update the server list in the distributed key-value pair database.

[0074] Step S120: Forward the inference request to the distributed key-value database, and determine the target inference server from the multiple inference servers included in the inference service cluster, wherein the distributed key-value database serves as a load balancing server middleware between the service gateway and the inference service cluster.

[0075] In this embodiment, when the service gateway receives an inference request, it can forward the inference request to the distributed key-value database and determine the inference server that will perform inference processing on the inference request from among the multiple inference servers included in the inference service cluster as the target inference server.

[0076] In some implementations, upon receiving an inference request, the service gateway can interact with a distributed key-value database, forward the inference request to the distributed key-value database, and determine the target inference server from among the multiple inference servers included in the inference service cluster based on the information of the server list stored in the distributed key-value database.

[0077] Step S130: The inference request is processed by the target inference server, and the inference result is fed back.

[0078] In this embodiment, when a target inference server is determined, the inference request can be processed by the target inference server, and the inference result can be fed back.

[0079] In some implementations, when a target inference server is determined, the inference request can be processed by the target inference server to obtain the inference result. Then, the target inference server can feed back the inference result to the service gateway, and the service gateway can feed back the inference result to the client that initiated the inference request.

[0080] Optionally, the target inference server may forward the inference result to the service gateway, or broadcast the output result to the service gateway, or notify the requesting worker thread to return the result through inter-thread communication, or have the target inference server actively call back to the gateway layer, etc., without limitation.

[0081] An embodiment of this application provides a load balancing method for inference. Upon receiving an inference request, the service gateway forwards the request to a distributed key-value database and determines a target inference server from among multiple inference servers in the inference service cluster. The distributed key-value database acts as a load balancing middleware between the service gateway and the inference service cluster. The target inference server processes the inference request and returns the inference result. By introducing the distributed key-value database as a load balancing middleware between the service gateway and the inference service cluster, inference requests can be evenly distributed across each inference server, maximizing the utilization of hardware resources.

[0082] Please see Figure 7 , Figure 7 A schematic flowchart of a load balancing method for inference provided in an embodiment of this application is shown. The following will focus on... Figure 7 The process shown will be described in detail. The load balancing method described in the inference method may specifically include the following steps:

[0083] Step S210: The service gateway receives an inference request.

[0084] For a detailed description of step S210, please refer to step S110, which will not be repeated here.

[0085] Step S220: Determine the fluctuation of the inference latency corresponding to the inference request.

[0086] In this embodiment, upon receiving an inference request, the service gateway can determine the fluctuation of the inference latency corresponding to the inference request, and select different processing methods based on different fluctuation conditions. Optionally, the fluctuation of the inference latency corresponding to the inference request may include: the inference latency corresponding to the inference request is uniform or the inference latency corresponding to the inference request is non-uniform.

[0087] Uniform inference latency for inference requests means that the processing time for different inference requests on the inference server fluctuates little. In other words, for the same or similar inference tasks, the processing time is basically the same each time. For example, in image classification tasks, the inference time is within a similar range regardless of when similar images are processed. In this case, the system load is relatively stable, and load balancing strategies are relatively easy to formulate. Weighted load balancing algorithms, such as weighted round-robin, can be used because the processing capacity of each inference server is relatively stable and predictable.

[0088] The uneven inference latency for inference requests indicates significant differences in processing time between different inference requests. This could be due to varying complexity of the inference tasks themselves (e.g., super-resolution of complex images takes longer than that of simple images), or significant variations in the load and hardware resource utilization of the inference server at different times. In such scenarios, the load on the inference service fluctuates greatly, making traditional simple load balancing strategies inadequate. More complex mechanisms, such as combining asynchronous processing and queue management, are needed to avoid request congestion and improve overall service performance.

[0089] In some implementations, the inference latency fluctuations for each inference task can be pre-set and stored. For example, the inference latency fluctuations for each inference task can be evaluated and deployed during the development phase. Then, when the service gateway receives an inference request, it can determine the inference task corresponding to that request and, based on the pre-set information, determine the inference latency fluctuations for that inference task.

[0090] Step S230: Based on the fluctuation situation, forward the inference request to the distributed key-value pair database, and determine the target inference server from the plurality of inference servers, wherein the distributed key-value pair database serves as a load balancing server middleware between the service gateway and the inference service cluster.

[0091] In this embodiment, if the fluctuation of the inference latency corresponding to the inference request is determined, the inference request can be forwarded to the distributed key-value database based on the fluctuation, and the target inference server can be determined from multiple inference servers.

[0092] Optionally, the fluctuation of inference latency corresponding to the inference request can include: uniform inference latency or non-uniform inference latency. A first processing method can be pre-defined when the inference latency is uniform, and a second processing method can be pre-defined when the inference latency is non-uniform. Based on this, if the fluctuation determines that the inference latency is uniform, the first processing method can be used to forward the inference request to the distributed key-value database and determine the target inference server from multiple inference servers; if the fluctuation determines that the inference latency is non-uniform, the second processing method can be used to forward the inference request to the distributed key-value database and determine the target inference server from multiple inference servers.

[0093] Step S240: The inference request is processed by the target inference server, and the inference result is fed back.

[0094] For a detailed description of step S240, please refer to step S130, which will not be repeated here.

[0095] One embodiment of this application provides a load balancing method for inference. The service gateway receives an inference request, determines the fluctuation of the inference latency corresponding to the request, and based on the fluctuation, forwards the inference request to a distributed key-value database. It then determines a target inference server from multiple inference servers. The distributed key-value database acts as a load balancing middleware between the service gateway and the inference service cluster. The target inference server processes the inference request and returns the inference result. Compared to... Figure 6 The load balancing method for inference shown in this embodiment also ensures that the request load is evenly distributed to each inference server in both algorithm inference scenarios with uniform and non-uniform inference latency, so as to make full use of hardware resources to the greatest extent.

[0096] Please see Figure 8 , Figure 8 A schematic flowchart of a load balancing method for inference provided in an embodiment of this application is shown. The following will focus on... Figure 8 The process shown will be described in detail. The load balancing method described in the inference method may specifically include the following steps:

[0097] Step S310: The service gateway receives an inference request.

[0098] For a detailed description of step S310, please refer to step S110, which will not be repeated here.

[0099] Step S320: Determine the fluctuation of the inference latency corresponding to the inference request.

[0100] For a detailed description of step S320, please refer to step S220, which will not be repeated here.

[0101] Step S330: If it is determined based on the fluctuation situation that the inference delay corresponding to the inference request is uniform, then the inference request is forwarded to the distributed key-value pair database.

[0102] Optionally, the fluctuation of inference latency corresponding to an inference request can include uniform inference latency. For scenarios with small inference latency fluctuations (less affected by inference input), this embodiment can use a distributed key-value database to implement the weight counter of the inference service cluster, replacing the original local weight counter of the service gateway. The inference server list is globally unique and updated in real time.

[0103] Please see Figure 9 , Figure 9 This diagram illustrates a framework of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uniform. For example... Figure 9 As shown, it includes the following components:

[0104] User: This refers to the client that issues the inference request and is the starting point of the entire process.

[0105] Service Gateway: Receives inference requests from clients. It also connects to the registry to obtain information about inference servers for load balancing decisions.

[0106] Registry: Responsible for managing the registration and discovery of inference servers, and providing inference service-related information to the service gateway.

[0107] Distributed Key-Value Database (KV DB): As a core component, it stores the weights (weight1, weight2, weight3, etc.) corresponding to each inference server (server1, server2, server3, etc.) in the inference service cluster. The service gateway retrieves this weight information from the distributed key-value database through a "fetch" operation.

[0108] Inference Service Cluster: Composed of multiple inference servers, the service gateway forwards inference requests to appropriate inference servers for inference processing through a "load balancing" operation based on weight information obtained from the distributed key-value database.

[0109] Specifically, in this embodiment, if it is determined that the inference latency corresponding to the inference request is uniform based on the fluctuation, the inference request can be forwarded to the distributed key-value pair database. This allows the distributed key-value pair database to check the stored list of inference servers based on the received inference request, and select a suitable inference server from the multiple inference servers included in the inference service cluster to perform inference processing on the inference request according to the load balancing information recorded in the inference server list.

[0110] In some implementations, a distributed key-value database stores the weight information of each inference server in the inference service cluster. For example, it stores the inference server identifier as the key and the weight value as the value. When the inference latency corresponding to an inference request is uniform, it means that the processing capacity of each inference server is relatively stable and predictable. Based on this, after the service gateway forwards the inference request to the distributed key-value database, the distributed key-value database can perform load balancing calculations based on the weights of each inference server according to preset calculation rules. For example, a weighted round-robin algorithm can be used, where servers with higher weights are more likely to be selected. Each time a request arrives, the weight information is retrieved from the distributed key-value database, and the target inference server is determined according to the rules.

[0111] Step S340: Based on the load balancing information stored in the distributed key-value pair database, determine the target inference server from the plurality of inference servers, wherein the load balancing information includes the weights of each of the plurality of inference servers, and the distributed key-value pair database serves as a load balancing server middleware between the service gateway and the inference service cluster.

[0112] In this embodiment, the distributed key-value database stores the weights of each inference server in the inference service cluster. Based on preset calculation rules, the distributed key-value database performs load balancing calculations according to the weights of each inference server to obtain load balancing information. Based on this, the distributed key-value database can determine the target inference server from multiple inference servers according to the load balancing information.

[0113] In some implementations, multiple inference servers stored in the distributed key-value database can be traversed. Based on the effective weights of each inference server, their current weights are obtained. The inference server with the highest current weight is then selected as the target inference server. Subsequently, the current weights of the target inference server stored in the distributed key-value database can be updated to ensure the inference server list is updated in real time.

[0114] As an feasible approach, four parameters are defined:

[0115] weight: The weight of each inference server, which remains constant;

[0116] total: The sum of the weights of all inference servers, which remains constant;

[0117] effective_weight: The effective weight of each inference server, with an initial value of weight;

[0118] current_weight: The current weight of each inference server, with an initial value of 0;

[0119] The specific iteration process is roughly as follows:

[0120] Iterate through all inference servers in the inference service cluster and update the weight of each inference server:

[0121] current_weight+=effective_weight

[0122] Select the inference server with the largest current_weight as the target inference server to process this request;

[0123] Update the selected target inference server weights:

[0124] current_weight -= total

[0125] Initially, each inference server in the inference service cluster has a current weight (current_weight) of 0 and an effective weight (effective_weight) of the static weight, meaning all inference servers have the same initial selection probability. Each time an inference server is selected, the operation `current_weight += effective_weight` is performed on each server. This dynamically adjusts the current weight based on the effective weight, which may change after each selection to ensure different current weights for different servers. Gradually increasing the current weight increases the chances of higher-weighted inference servers being selected in each round, achieving weight-based load balancing.

[0126] The process of selecting the inference server with the highest current weight from among multiple inference servers ensures fairness and efficiency in allocating requests to inference servers according to their weights. The server with the highest current weight is selected to process the current inference request, and its current weight is updated. The selected inference server will then be used to process the request, and its current weight will be updated (current_weight -= total). This updated weight will influence the next inference server selection process, ensuring the proper functioning of the load balancing algorithm.

[0127] These two key steps enable the dynamic adjustment of the probability of each inference server being selected based on its weight, achieving weight-based load balancing. This design ensures efficient utilization of inference server resources and improves system performance and stability.

[0128] Taking the open-source distributed key-value database Redis as an example, a zset is used to store the list of servers in the inference service cluster and their corresponding current_weights. The zset member is the inference server identifier, and the score is the current_weight. Leveraging the automatic sorting feature of zset, the member with the highest score is selected as the inference server for the current request, and then the score of the corresponding member for that inference server is updated.

[0129] Data structure design:

[0130] Naming Rules Key infer_{service}_{region}

[0131] Value illustrate member {Instance IP} Service instance IP score {Current Weight} Current weight value of this instance

[0132] Step S350: Send the inference request to the target inference server through the service gateway.

[0133] In this embodiment, when the distributed key-value database determines the target inference server, it can feed back the IP address of the target inference server to the service gateway. Accordingly, the service gateway sends the inference request to the target inference server based on the IP address of the target inference server.

[0134] In some implementations, when the distributed key-value database determines the target inference server, it can feed back the IP address of the target inference server to the service gateway. Accordingly, the service gateway sends the inference request to the target inference server through a forward operation based on the IP address of the target inference server, so as to ensure that the inference request can reach the appropriate processing node.

[0135] Step S360: The inference request is processed by the target inference server, and the inference result is fed back.

[0136] In this embodiment, upon receiving an inference request, the target inference server can invoke its own inference function to process the input data corresponding to the inference request using a pre-trained model, thus completing the inference task. After completing the inference task, the target inference server returns the inference result to the service gateway, which then returns the final response inference result to the user, thereby completing a full inference request processing flow. The entire process, through the collaboration between various components, achieves a reasonable distribution of inference requests among multiple inference service instances, achieving global load balancing and ensuring the efficient and stable operation of the system.

[0137] One embodiment of this application provides a load balancing method for obtaining inference. A service gateway receives an inference request, determines the fluctuation of the inference latency corresponding to the request, and if the fluctuation determines that the inference latency is uniform, forwards the request to a distributed key-value database. Based on the load balancing information stored in the distributed key-value database, a target inference server is determined from multiple inference servers. The load balancing information includes the weights of each inference server. The distributed key-value database acts as a load balancing middleware between the service gateway and the inference service cluster. The service gateway sends the inference request to the target inference server, which then processes the request and returns the inference result. Compared to... Figure 6 The load balancing method for inference shown in this embodiment also addresses scenarios with uniform latency by using a distributed key-value database to uniformly store and update the weight list of the inference server, thereby implementing a global load balancing strategy for the inference service. Compared with conventional load balancing implementations, the inference service experiences less latency fluctuation and higher resource utilization.

[0138] Please see Figure 10 , Figure 10 This diagram illustrates a flowchart of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uniform. For example... Figure 10 As shown in the diagram, the process diagram includes the following steps:

[0139] 1. Inference Service Registration: After the Inference Service starts, it actively initiates a registration operation with the Registry, submitting its own relevant information, such as the service's network address, port number, resource configuration, and service version, so that it can be identified and managed in the Inference Service Cluster.

[0140] 2. Gateway Discovery Service: The API Gateway interacts with the registry center to periodically or in real-time obtain information on registered inference servers, and knows the available inference servers and their status in the current inference service cluster, in order to prepare for subsequent request forwarding.

[0141] 3. User request submission: When a user generates an inference request, they send an inference request to the service gateway.

[0142] 4. Load Balancing Decision: After receiving an inference request, the service gateway interacts with the distributed key-value database (KV-Cache). Based on the weight information of each inference server maintained internally, the KV-Cache selects suitable candidate inference servers (pop candidate), updates the weights of each inference server according to the load balancing algorithm (update weights), and returns the IP address of the selected target inference server (instance IP) to the service gateway. This process is the core of load balancing, ensuring that requests are distributed appropriately to different inference services to prevent overload of any single inference server.

[0143] 5. Request forwarding: Based on the instance IP address obtained from KV-Cache, the service gateway forwards the user's inference request to the corresponding target inference server to ensure that the request reaches the appropriate processing node.

[0144] 6. Inference Service Processing: After receiving the forwarded request, the target inference server calls its own inference function (infer) to process the request data using the pre-trained model and complete the inference task.

[0145] 7. Result Return: After completing the inference operation, the target inference server returns the processing result to the service gateway, which then returns the final response to the user. This completes a full inference request processing flow. The entire process, through the collaboration between various components, achieves the reasonable distribution of inference requests among multiple inference servers, realizing global load balancing and ensuring the efficient and stable operation of the system.

[0146] Please see Figure 11 , Figure 11 A schematic flowchart of a load balancing method for inference provided in an embodiment of this application is shown. The following will focus on... Figure 11 The process shown will be described in detail. The load balancing method described in the inference method may specifically include the following steps:

[0147] Step S410: The service gateway receives an inference request.

[0148] For a detailed description of step S410, please refer to step S110, which will not be repeated here.

[0149] Step S420: Determine the fluctuation of the inference latency corresponding to the inference request.

[0150] For a detailed description of step S420, please refer to step S220, which will not be repeated here.

[0151] Step S430: If it is determined based on the fluctuation situation that the inference delay corresponding to the inference request is uneven, then the inference request is stored in the input queue of the distributed key-value pair database, and an idle inference server is determined from the plurality of inference servers as the target inference server.

[0152] Optionally, the fluctuation of inference latency corresponding to the inference request may include uneven inference latency. For scenarios with large fluctuations in inference latency (significantly affected by inference input), this embodiment uses a distributed key-value database as both an input / output queue and a task result cache, fully leveraging its advantages such as high concurrency, low latency, rich functionality, quick access, and easy code implementation, while ensuring that results can be returned within a single synchronous HTTP request.

[0153] For example, using Redis as a distributed key-value database, please refer to [link / reference]. Figure 12 , Figure 12 This diagram illustrates a framework of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uneven. For example... Figure 12 As shown, it includes the following components:

[0154] User: This refers to the client that issues the inference request and is the starting point of the entire process.

[0155] Service Gateway: Responsible for receiving user inference requests (Req) and connecting them to a distributed key-value database.

[0156] Distributed key-value database (KV-Cache): A key-value cache, organized by topic, containing an input queue and multiple output queues (output A, output B, etc.), used for storing and transmitting messages.

[0157] Inference Service: Reads request messages from the input queue, performs inference processing, and writes the results to the corresponding output queue.

[0158] In implementation, users (User A, User B, etc.) send inference requests (Req) to the service gateway. Upon receiving the request, the service gateway uses the XADD command to add messages (msgA, msgB, etc.) to the input queue of the KV-Cache. The inference server reads the request message from the input queue using the XREAD command and begins inference processing. After completing inference, the inference server uses the LPUSH command to write the processing results to the corresponding output queues (e.g., output A, output B). The service gateway uses the BRPOP command to read the inference results from the corresponding output queues (output A, output B) in a blocking manner, so as to return them to the user. This process utilizes the key-value cache's queue mechanism to achieve asynchronous processing of inference requests and return of results, which helps to cope with complex situations such as uneven inference latency and improves the system's processing capacity and stability.

[0159] Specifically, in this embodiment, if the fluctuation indicates uneven inference latency corresponding to the inference request, the inference request can be stored in the input queue of the distributed key-value pair database, and an idle inference server can be selected from multiple inference servers as the target inference server. Uneven inference latency for inference requests means significant differences in processing time between different inference tasks, which traditional load balancing methods struggle to handle. Storing inference requests in the input queue of the distributed key-value pair database utilizes the first-in-first-out (FIFO) characteristic of the queue to temporarily store inference requests, preventing a sudden surge of requests to the inference server. Simultaneously, by monitoring the status of each inference server (e.g., CPU and GPU utilization, number of tasks processed, etc.), an idle server can be selected from multiple inference servers as the target server, ensuring reasonable request allocation and avoiding uneven workload distribution.

[0160] In some implementations, if it is determined that the inference latency corresponding to the inference request is uneven based on the fluctuation, the inference request can be encapsulated in a prescribed format and stored in the input queue of a distributed key-value database using database commands.

[0161] In some implementations, multiple inference servers in the inference service cluster periodically report their resource usage, task load, and other status information to the registry center or database. Based on the status information reported by each inference server, idle inference servers are selected from the multiple inference servers as target inference servers according to a preset idle criterion. For example, inference servers with resource utilization rates below a threshold are selected.

[0162] Optionally, if there are multiple idle inference servers, an idle inference server can be selected as the target inference server in sequence or randomly, without limitation.

[0163] Please see Figure 13 , Figure 13 This application shows Figure 11 The flowchart shown illustrates step S430 of the load balancing method for inference. The following will focus on... Figure 13 The process shown will be described in detail, and the method may specifically include the following steps:

[0164] Step S431: If it is determined based on the fluctuation situation that the inference delay corresponding to the inference request is uneven, then check whether the distributed key-value pair database has cached the inference result corresponding to the inference request.

[0165] In some implementations, if it is determined that the inference latency corresponding to the inference request is uneven based on fluctuations, it is possible to detect whether the distributed key-value pair database caches the inference result corresponding to the inference request.

[0166] As an feasible approach, if it is determined that the inference latency corresponding to the inference request is uneven based on the fluctuation, the request ID corresponding to the inference request can be obtained, and the inference result corresponding to the inference request can be cached in the distributed key-value pair database based on the request ID.

[0167] Step S432: If it is detected that the distributed key-value pair database does not cache the reasoning result corresponding to the reasoning request, then the reasoning request is stored in the input queue of the distributed key-value pair database.

[0168] In some implementations, if the distributed key-value database does not cache the inference result corresponding to the inference request, it can be assumed that no inference processing has been performed on the inference task corresponding to the inference request before, or that the inference result obtained from the inference processing on the inference task corresponding to the inference request has not been cached before. Therefore, the inference request can be stored in the input queue of the distributed key-value database so that the inference result can be obtained through subsequent inference processing by the inference server.

[0169] Step S433: If it is detected that the distributed key-value pair database has a cached inference result corresponding to the inference request, then the cached inference result corresponding to the inference request is fed back.

[0170] In some implementations, if the distributed key-value pair database detects that the inference result corresponding to the inference request is cached, it can be assumed that inference processing has already been performed on the inference task corresponding to the inference request, and the obtained inference result has been cached in the distributed key-value pair database. Therefore, the cached inference result corresponding to the inference request can be directly fed back, thereby improving the processing efficiency of the inference task and reducing the system power consumption during the inference process.

[0171] Step S440: The inference request is read from the input queue of the distributed key-value pair database through the target inference server, and the inference request is processed to obtain the inference result.

[0172] In this embodiment, when the inference request is stored in the input queue of the distributed key-value pair database and the target inference server is determined from multiple inference servers, the inference request can be read from the input queue of the distributed key-value pair database through the target inference server, and the inference request can be processed by the target inference server to obtain the inference result.

[0173] In some implementations, when the inference request is stored in the input queue of the distributed key-value pair database and the target inference server is determined from multiple inference servers, the inference request can be pulled from the input queue by the target inference server, and the inference operation can be performed on the inference request by the target inference server to obtain the inference result.

[0174] Optionally, the inference request may correspond to input data. For example, if the inference request is to classify a photo, then the input data may be the photo itself.

[0175] As one feasible approach, storing inference requests in the input queue of a distributed key-value database may include: storing the inference request in the input queue of the distributed key-value database and forwarding the input data corresponding to the inference request to the distributed key-value database for caching.

[0176] As another feasible approach, storing inference requests in the input queue of a distributed key-value database can include: storing the inference requests in the input queue of the distributed key-value database and uploading the input data corresponding to the inference requests to an object storage service (OSS), such as S3 Service. An object storage service is a distributed storage system for massive data storage. By storing input and output data through an object storage service, the additional system load caused by a large amount of data passing through the gateway layer can be avoided, as well as the large key problem caused by directly storing data in a distributed key-value database.

[0177] In a scenario where inference requests are stored in the input queue of a distributed key-value database and the corresponding input data is uploaded to an object storage service, the target inference server can read inference requests from the input queue of the distributed key-value database and download input data from the object storage service. Then, it performs inference processing on the input data based on the inference requests to obtain inference results and output data.

[0178] Step S450: Store the inference result in the output queue of the distributed key-value database to provide feedback on the inference result.

[0179] In this embodiment, when the target inference server obtains an inference result by processing the inference request, it can store the inference result in the output queue of the distributed key-value pair database to provide feedback on the inference result.

[0180] In some implementations, when the target inference server obtains an inference result by processing the inference request, it can push the inference result-related information to the output queue of the distributed key-value pair database to provide feedback on the inference result.

[0181] As an implementable approach, for scenarios where inference requests are stored in the input queue of a distributed key-value pair database and the input data corresponding to the inference requests is forwarded to the distributed key-value pair database for caching, upon obtaining the inference results and output data, the target inference server can store the inference results in the output queue of the distributed key-value pair database and forward the output data to the distributed key-value pair database for caching. At this time, the service gateway can read the output data from the cache of the distributed key-value pair database after reading the inference results from the output queue of the distributed key-value pair database.

[0182] As another feasible approach, for scenarios where inference requests are stored in the input queue of a distributed key-value database and the corresponding input data is uploaded to an object storage service, the target inference server can store the inference results in the output queue of the distributed key-value database and upload the output data to the object storage service after obtaining the inference results and output data. At this time, the service gateway can download the output data from the object storage service after obtaining the inference results from the output queue of the distributed key-value database. This avoids the additional system load caused by a large amount of data passing through the gateway layer and also avoids the large key problem caused by directly storing data in the distributed key-value database.

[0183] Step S460: Cache the inference result in the distributed key-value database.

[0184] In this embodiment, after the inference request is processed by the target inference server to obtain the inference result, the inference result can be cached in a distributed key-value pair database. This allows the inference result to be retrieved directly from the cache of the distributed key-value pair database and fed back when the same inference request occurs again. This can improve the inference feedback effect and reduce the inference power consumption of the entire system.

[0185] One embodiment of this application provides a load balancing method for inference. The service gateway receives an inference request, determines the fluctuation of the inference latency corresponding to the request, and if the fluctuation determines that the inference latency is uneven, the inference request is stored in the input queue of a distributed key-value pair database. An idle inference server is selected from multiple inference servers as the target inference server. The target inference server reads the inference request from the input queue of the distributed key-value pair database, performs inference processing on the request, obtains the inference result, stores the inference result in the output queue of the distributed key-value pair database to provide feedback, and caches the inference result in the distributed key-value pair database. Compared to... Figure 6 The load balancing method for inference shown in this embodiment also addresses scenarios with uneven latency by using a distributed key-value database with cloud object storage and queues to act as a data transmission carrier. This allows for unlimited data transmission volume, thereby achieving global load balancing for the inference service and high resource utilization.

[0186] Please see Figure 14 , Figure 14 This illustration shows a flowchart of an embodiment of the present application in a scenario where the inference latency corresponding to the inference request is uneven. For example... Figure 14 As shown, taking Redis as an example of a distributed key-value database, the flowchart includes the following processes:

[0187] 1. User request initiation: The user sends an inference request to the API Gateway.

[0188] 2. Input data upload: After receiving the inference request, the service gateway uploads the input data to the S3 service (an object storage service) to prepare for the subsequent inference service to obtain the data.

[0189] 3. Cache Query: The service gateway initiates a query cache operation to the Redis cache to check if there is a directly usable cached result. If so, it can be returned directly to the user, skipping subsequent steps; otherwise, proceed to the next step.

[0190] 4. Request forwarding to the message queue: The service gateway forwards inference requests to the Redis message queue (input queue) to manage and schedule inference tasks.

[0191] 5. Inference Service Pull Request: The Inference Service pulls inference request messages from the message queue (input queue) to obtain the task information that needs to be processed.

[0192] 6. Download input data: The inference server downloads the previously uploaded input data from the S3 service in order to perform inference operations.

[0193] 7. Inference Execution: The inference server uses the downloaded input data to perform inference operations, and uses relevant algorithms and models to complete the inference task.

[0194] 8. Output data upload: The inference server uploads the inference output data to the S3 service to store the inference results data.

[0195] 9. Pushing results to the message queue: The inference server pushes the inference result information to the Redis message queue (output queue) to notify the service gateway that the result has been generated.

[0196] 10. Download output data: The service gateway downloads inference output data from the S3 service to obtain the final inference result.

[0197] 11. Result caching: The service gateway caches the inference results in the Redis cache, so that subsequent identical requests can directly obtain the results, thus improving response speed.

[0198] 12. Returning the result to the user: The service gateway returns the final inference result as a response to the user, completing the entire inference request processing flow.

[0199] Understandably, in the above process, data transmission between the caller and the inference service relies on both Object Storage Service (OSS) and Message Queue. After the caller completes the data upload, it only interacts with the service through the resource ID returned by OSS. This avoids the additional system load caused by a large amount of data passing through the gateway layer, and also avoids the large key problem caused by directly storing data in Redis.

[0200] After receiving a request, the gateway layer first performs a TLS proxy service at the security gateway, intercepting malicious attacks and unidentified APIs. Then comes a series of processes including rate limiting, interface authentication, and data processing. During the routing and forwarding phase, the protocol-converted request data is delivered to the corresponding input stream based on different business requirements and versions. An output list key is generated based on the distributed ID, and the server continuously waits for the inference service to input the results into the list. Each inference server is a consumer of a specific stream. When the current inference server is idle, it continuously waits for messages to be produced in the stream. Upon receiving a message, it decrypts and deserializes the data, extracts the output list name and business-related data, executes the algorithm model inference, and inputs the results into the output list.

[0201] In this embodiment, the caller first uploads data to the object storage service and obtains the resource identifier. Then, it calls the business API interface of the gateway layer to intercept, rate-limit, and version-based the request. Based on the inference request ID, it checks whether the corresponding inference result exists in the cache. If not, it performs communication protocol conversion and data protocol conversion according to the specific business, serializes the request data and metadata and link information, and generates an output queue name. After writing the data to the input queue, it continuously blocks and waits for the result of the corresponding output queue. Idle inference servers (data consumers) actively pull inference tasks from the stream, and actively write the result to the output queue after completing the inference. After receiving the result, the gateway layer can directly return the response of the current HTTP request and write the result to the cache at the same time.

[0202] Data structure design:

[0203] Key Naming Rules Input queue lb_{service}_in_{service version} Output queue lb_{service}_out_{date}_{uuid}

[0204] To prevent master-slave replication anomalies caused by large key deletion / expiration, which could impact Redis service, direct file / image data transfer is prohibited. The Value field is designed as follows:

[0205] field name illustrate x_start_time Message delivery timestamp x_in_timeout If current time - start_time > x_in_timeout, then no action is taken. x_http_header_version Service version number x_trace_info OTEL call chain information (JSON serialization) x_output_stream Output queue name body The internal request body contains all business fields.

[0206] To better illustrate the solutions of this application's embodiments, a specific case will be used below. In this case, Redis Cluster version 5.0.12 is used, AWS S3 service is used for object storage, API Gateway (supporting highly customized flow control and forwarding logic) is used for the gateway layer, and image super-resolution is used for inference service capabilities.

[0207] The image super-resolution service cluster mainly consists of two phases: Phase 1, which performs minor optimizations for high-resolution images; if the image resolution is low, it transitions to Phase 2 for fine-tuning and further enhancement of clarity. Phase 1 has relatively stable inference latency with minimal fluctuations, making it suitable for a global round-robin load balancing strategy. This phase uses a distributed database to store the weight list of inference instances. Phase 2 of the super-resolution process takes more than twice as long as Phase 1, resulting in a significant difference in inference time between low-resolution and high-resolution images. If forwarding is directly based on Nginx's load balancing algorithm (such as weighted round-robin or random), it can easily cause inference tasks to be blocked on GPU nodes processing low-resolution images when the request volume approaches the theoretical algorithm's limit. Therefore, a Redis queue is introduced between the gateway access layer and the inference service. Following the previously agreed-upon naming convention, the input queue and the output queue for the current request are named as follows:

[0208] name Input queue lb_superresolution_in_v1 Output queue lb_superresolution_out_20241224_abc

[0209] After completing preprocessing, the gateway layer serializes the request data and metadata and adds it to the end of `lb_superresolution_in_v1` using `xadd`. It then blocks and waits (`brpop`, timeout 30s) for the message `lb_superresolution_out_20241224_abc`. The superresolution service retrieves the message from the header of `lb_superresolution_in_v1` using `xread`, deserializes the data, performs inference, serializes the inference result, and passes it to `lb_superresolution_out_20241224_abc` (`lpush`, expiration time 1min). After retrieving the first data item from the list, the gateway layer directly returns the deserialized result.

[0210] Please see Figure 15 , Figure 15 The diagram shows a comparison of latency fluctuations before and after using the super-resolution inference service. It can be seen that the solution is significantly effective, with lower latency and smaller fluctuations, and can save a lot of GPU server resources under the same service performance indicators.

[0211] Please see Figure 16 , Figure 16 A block diagram of a load balancing device for inference provided in an embodiment of this application is shown. The following will focus on... Figure 16 The block diagram shown illustrates that the inference load balancing device 200 includes: an inference request receiving module 210, an inference server selection module 220, and an inference processing module 230, wherein:

[0212] The inference request receiving module 210 is used for the service gateway to receive inference requests.

[0213] The inference server selection module 220 is used to forward the inference request to the distributed key-value pair database and determine the target inference server from the multiple inference servers included in the inference service cluster, wherein the distributed key-value pair database serves as a load balancing server middleware between the service gateway and the inference service cluster.

[0214] Furthermore, the inference server selection module 220 includes: a fluctuation determination submodule and an inference server selection submodule, wherein:

[0215] The fluctuation determination submodule is used to determine the fluctuation of the inference latency corresponding to the inference request.

[0216] The inference server selection submodule is used to forward the inference request to the distributed key-value pair database based on the fluctuation situation, and to determine the target inference server from the plurality of inference servers.

[0217] Further, the inference server selection submodule includes: a first inference request forwarding unit and a first inference server selection unit, wherein:

[0218] The first inference request forwarding unit is configured to forward the inference request to the distributed key-value pair database if it is determined based on the fluctuation situation that the inference delay corresponding to the inference request is uniform.

[0219] The first inference server selection unit is used to determine the target inference server from the plurality of inference servers based on the load balancing information stored in the distributed key-value pair database, wherein the load balancing information includes the weights of the plurality of inference servers.

[0220] Further, the first inference server selection unit includes: a current weight acquisition subunit and a first inference server selection subunit, wherein:

[0221] The current weight acquisition subunit is used to traverse the multiple inference servers stored in the distributed key-value pair database and obtain the current weight of each of the multiple inference servers based on their respective effective weights.

[0222] The first inference server selection subunit is used to select the inference server with the highest current weight from the plurality of inference servers as the target inference server.

[0223] Furthermore, the first inference server selection unit further includes: a current weight update subunit, wherein:

[0224] The current weight update subunit is used to update the current weight of the target inference server stored in the distributed key-value pair database.

[0225] Furthermore, the inference server selection submodule includes: a second inference server selection unit, wherein:

[0226] The second inference server selection unit is configured to, if it is determined based on the fluctuation situation that the inference latency corresponding to the inference request is uneven, store the inference request in the input queue of the distributed key-value pair database, and determine an idle inference server from the plurality of inference servers as the target inference server.

[0227] Further, the second inference server selection unit includes: a cache detection subunit and a second inference server selection subunit, wherein:

[0228] The cache detection subunit is used to detect whether the distributed key-value pair database caches the inference result corresponding to the inference request if it is determined from the fluctuation situation that the inference latency corresponding to the inference request is uneven.

[0229] The second inference server selection subunit is used to store the inference request into the input queue of the distributed key-value pair database if it is detected that no inference result corresponding to the inference request is cached in the distributed key-value pair database.

[0230] Further, the second inference server selection unit includes: an inference result feedback subunit, wherein:

[0231] The inference result feedback subunit is used to feed back the cached inference result corresponding to the inference request if it is detected that the distributed key-value pair database has a cached inference result corresponding to the inference request.

[0232] Furthermore, the second inference server selection unit includes: an input data upload subunit, wherein:

[0233] The input data upload subunit is used to store the inference request into the input queue of the distributed key-value pair database and upload the input data corresponding to the inference request to the object storage service.

[0234] The inference processing module 230 is used to perform inference processing on the inference request through the target inference server and to return the inference result.

[0235] Further, the inference processing module 230 includes: an inference request sending submodule and a first inference processing submodule, wherein:

[0236] The inference request sending submodule is used to send the inference request to the target inference server through the service gateway.

[0237] The first inference processing submodule is used to perform inference processing on the inference request through the target inference server and to return the inference result.

[0238] Further, the inference processing module 230 includes: an inference request reading submodule and a second inference processing submodule, wherein:

[0239] The inference request reading submodule is used to read the inference request from the input queue of the distributed key-value pair database through the target inference server, and to perform inference processing on the inference request to obtain the inference result.

[0240] The second inference processing submodule is used to store the inference result into the output queue of the distributed key-value pair database to provide feedback on the inference result.

[0241] Furthermore, the inference processing module 230 includes: an input data download submodule, an output data upload submodule, and an output data download submodule, wherein:

[0242] The input data download submodule is used to read the inference request from the input queue of the distributed key-value pair database through the target inference server, and to download the input data from the object storage service, perform inference processing on the input data based on the inference request, and obtain the inference result and output data.

[0243] The output data upload submodule is used to store the inference results into the output queue of the distributed key-value pair database and upload the output data to the object storage service.

[0244] The input data download submodule is used by the service gateway to download the output data from the object storage service when the inference result is obtained from the output queue of the distributed key-value pair database.

[0245] Furthermore, the inference load balancing device 200 also includes:

[0246] The inference result caching module is used to cache the inference results in the distributed key-value pair database.

[0247] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0248] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0249] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0250] Please see Figure 17 This document illustrates a structural block diagram of an electronic device 100 provided in an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0251] The processor 110 may include one or more processing cores. The processor 110 connects to various parts of the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and by calling data stored in the memory 120. Optionally, the processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 110 and may be implemented separately using a communication chip.

[0252] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0253] Please see Figure 18 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 300 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0254] The computer-readable storage medium 300 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 300 has storage space for program code 310 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 310 may be compressed, for example, in a suitable form.

[0255] In summary, the inference load balancing method, apparatus, electronic device, and storage medium provided in this application embodiment allow the service gateway to receive an inference request, forward the inference request to a distributed key-value pair database, and determine a target inference server from among multiple inference servers included in the inference service cluster. The distributed key-value pair database acts as a load balancing server middleware between the service gateway and the inference service cluster. The target inference server performs inference processing on the inference request and returns the inference result. Therefore, by introducing a distributed key-value pair database as a load balancing server middleware between the service gateway and the inference service cluster, inference requests can be evenly distributed to each inference server to maximize the utilization of hardware resources.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A load balancing method for inference, characterized in that, The method includes: The service gateway received the inference request; The inference request is forwarded to a distributed key-value database, and a target inference server is determined from multiple inference servers included in the inference service cluster, wherein the distributed key-value database serves as a load balancing server middleware between the service gateway and the inference service cluster. The target inference server processes the inference request and returns the inference result.

2. The method according to claim 1, characterized in that, The step of forwarding the inference request to the distributed key-value database and determining the target inference server from the multiple inference servers included in the inference service cluster includes: Determine the fluctuation of the inference latency corresponding to the inference request; Based on the fluctuations, the inference request is forwarded to the distributed key-value database, and the target inference server is determined from the plurality of inference servers.

3. The method according to claim 2, characterized in that, The step of forwarding the inference request to the distributed key-value database based on the fluctuation situation, and determining the target inference server from the plurality of inference servers, includes: If it is determined based on the fluctuation situation that the inference delay corresponding to the inference request is uniform, then the inference request is forwarded to the distributed key-value pair database; Based on the load balancing information stored in the distributed key-value database, the target inference server is determined from the plurality of inference servers, wherein the load balancing information includes the weights of the plurality of inference servers.

4. The method according to claim 3, characterized in that, The step of determining the target inference server from the plurality of inference servers based on the load balancing information stored in the distributed key-value database includes: Iterate through the multiple inference servers stored in the distributed key-value database, and obtain the current weight of each of the multiple inference servers based on their respective effective weights. From the plurality of inference servers, the inference server with the highest current weight is selected as the target inference server.

5. The method according to claim 4, characterized in that, After selecting the inference server with the highest current weight from the plurality of inference servers as the target inference server, the method further includes: Update the current weights of the target inference server stored in the distributed key-value pair database.

6. The method according to any one of claims 3-5, characterized in that, The step of processing the inference request through the target inference server and feeding back the inference result includes: The inference request is sent to the target inference server through the service gateway; The target inference server processes the inference request and returns the inference result.

7. The method according to claim 2, characterized in that, The step of forwarding the inference request to the distributed key-value database based on the fluctuation situation, and determining the target inference server from the plurality of inference servers, includes: If it is determined based on the fluctuation that the inference latency corresponding to the inference request is uneven, then the inference request is stored in the input queue of the distributed key-value pair database, and an idle inference server is determined from the plurality of inference servers as the target inference server. The step of processing the inference request through the target inference server and feeding back the inference result includes: The target inference server reads the inference request from the input queue of the distributed key-value pair database, performs inference processing on the inference request, and obtains the inference result. The inference result is stored in the output queue of the distributed key-value database to provide feedback on the inference result.

8. The method according to claim 7, characterized in that, If, based on the fluctuations, it is determined that the inference latency corresponding to the inference request is uneven, then storing the inference request in the input queue of the distributed key-value database includes: If it is determined based on the fluctuation situation that the inference latency corresponding to the inference request is uneven, then it is detected whether the distributed key-value pair database has cached the inference result corresponding to the inference request; If it is detected that the distributed key-value pair database does not cache the inference result corresponding to the inference request, then the inference request is stored in the input queue of the distributed key-value pair database.

9. The method according to claim 8, characterized in that, The method further includes: If it is detected that the distributed key-value pair database has a cached inference result corresponding to the inference request, then the cached inference result corresponding to the inference request is returned.

10. The method according to any one of claims 7-9, characterized in that, The step of storing the inference request into the input queue of the distributed key-value database includes: The inference request is stored in the input queue of the distributed key-value database, and the input data corresponding to the inference request is uploaded to the object storage service. The step of processing the inference request through the target inference server and feeding back the inference result includes: The target inference server reads the inference request from the input queue of the distributed key-value pair database and downloads the input data from the object storage service. Based on the inference request, it performs inference processing on the input data to obtain the inference result and output data. The inference results are stored in the output queue of the distributed key-value database, and the output data is uploaded to the object storage service. When the service gateway obtains the inference result from the output queue of the distributed key-value database, it downloads the output data from the object storage service.

11. The method according to any one of claims 7-9, characterized in that, After storing the inference result in the output queue of the distributed key-value database to feed back the inference result, the method further includes: The inference results are cached in the distributed key-value database.

12. A load balancing device for inference, characterized in that, The device includes: The inference request receiving module is used by the service gateway to receive inference requests; An inference server selection module is used to forward the inference request to a distributed key-value pair database and determine a target inference server from multiple inference servers included in the inference service cluster, wherein the distributed key-value pair database serves as a load balancing server middleware between the service gateway and the inference service cluster. The inference processing module is used to perform inference processing on the inference request through the target inference server and to return the inference result.

13. An electronic device, characterized in that, The method includes a memory and a processor, the memory being coupled to the processor, the memory storing instructions, and when the instructions are executed by the processor, the processor performing the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-11.

Images