Load balancing method, apparatus, device, and medium

Abstract

The embodiment of the application provides a load balancing method, device, equipment and medium, the method comprises the following steps: as the target load balancer in the same geographical area range with the client, in response to the service data processing request sent by the client, the computing node list stored in the target load balancer is acquired; multiple computing nodes in the computing node list are in the same geographical area range with the target load balancer; according to the first performance index parameter and the active request number of each computing node in the computing node list, the second performance index parameter of each computing node is determined; according to the second performance index parameter, the target computing node is determined in the multiple computing nodes, and the service data processing request is forwarded to the target computing node; the service data processing result of the target computing node for the service data processing request is received, and the service data processing result is returned to the client. By implementing the embodiment of the application, the resource use efficiency of the computing node can be improved.

Description

Technical Field

[0001] This application relates to the field of cloud technology, and in particular to a load balancing method, apparatus, device, and medium. Background Technology

[0002] In current load balancing scenarios, the built-in load balancer of a Kubernetes (Kubernetes, an open-source container orchestration platform) cluster can use load balancing algorithms such as round-robin, weighted round-robin, random, least connections, and hash-based load balancing to distribute network or application traffic to the compute nodes within the Kubernetes cluster. However, when there are significant performance differences between compute nodes within the same Kubernetes cluster, or between compute nodes in different Kubernetes clusters, using the above load balancing algorithms can easily lead to excessively low resource utilization efficiency of the compute nodes. Summary of the Invention

[0003] This application provides a load balancing method, apparatus, device, and medium that can improve the resource utilization efficiency of computing nodes.

[0004] This application provides a load balancing method, which is executed by a target load balancer, wherein the target load balancer and the client are located in the same geographical area. The method includes:

[0005] In response to the business data processing request sent by the client, obtain the list of computing nodes stored in the target load balancer. The list of computing nodes includes multiple computing nodes and the first performance index parameters corresponding to each computing node. The multiple computing nodes and the target load balancer are located in the aforementioned geographical area. The first performance index parameter of each computing node is used to indicate the data processing performance level of a computing node.

[0006] The second performance metric parameter for each compute node is determined based on the first performance metric parameter for each compute node and the number of active requests for each compute node.

[0007] Based on the second performance index parameter of each computing node, the target computing node is determined among multiple computing nodes, and the business data processing request is forwarded to the target computing node so that the target computing node can execute the business data processing request.

[0008] Receive the business data processing results from the target computing node in response to the business data processing request, and return the business data processing results to the client.

[0009] One embodiment of this application provides a load balancing system, which includes a client, a target load balancer, and multiple computing nodes, wherein the client, the target load balancer, and the multiple computing nodes are located in the same geographical area.

[0010] The client is used to send business data processing requests to the target load balancer;

[0011] The target load balancer is used to respond to business data processing requests and obtain a list of computing nodes stored in the target load balancer. The list of computing nodes includes multiple computing nodes and a first performance index parameter corresponding to each computing node. The first performance index parameter of each computing node is used to indicate the data processing performance level of a computing node.

[0012] The target load balancer is also used to determine the second performance metric parameter of each compute node based on the first performance metric parameter of each compute node and the number of active requests for each compute node.

[0013] The target load balancer is also used to determine the target compute node among multiple compute nodes based on a second performance metric parameter for each compute node.

[0014] The target compute node is used to execute business data processing requests;

[0015] The target load balancer is also used to receive the business data processing results from the target computing node in response to the business data processing request, and return the business data processing results to the client;

[0016] The client is used to receive the results of business data processing.

[0017] One embodiment of this application provides a load balancing device, which is applied in a target load balancer, wherein the target load balancer and the client are located in the same geographical area. The device includes:

[0018] The request receiving module is used to respond to the business data processing request sent by the client, obtain the list of computing nodes stored in the target load balancer, the list of computing nodes includes multiple computing nodes and the first performance index parameters corresponding to each computing node, the multiple computing nodes and the target load balancer are located in the above-mentioned geographical area, and the first performance index parameter of each computing node is used to indicate the data processing performance level of a computing node.

[0019] The node selection module is used to determine the second performance index parameter of each computing node based on the first performance index parameter of each computing node and the number of active requests of each computing node.

[0020] The node selection module is also used to determine the target computing node among multiple computing nodes based on the second performance index parameter of each computing node, and forward the business data processing request to the target computing node so that the target computing node can execute the business data processing request.

[0021] The result return module is used to receive the business data processing results from the target computing node in response to the business data processing request, and return the business data processing results to the client.

[0022] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method provided in one aspect of this application.

[0023] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in one aspect of this application.

[0024] According to one aspect of this application, a computer program product is provided, which may include a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and executes the computer program, causing the computer device to perform the method provided in the above aspect.

[0025] In this embodiment, for a target load balancer located in the same geographical area as the client, it can respond to the client's business data processing request. Based on the first performance index parameters corresponding to multiple computing nodes stored in the target load balancer's computing node list, and the number of active requests for each computing node, it determines the second performance index parameter for each computing node. The computing node with the highest second performance index parameter can be used as the target computing node for issuing business data processing requests. The business data processing result of this target computing node can be sent to the target load balancer, which then returns the result to the client. By determining the real-time performance status of each computing node using its current first performance index parameters and the current number of active requests, and allocating business data processing requests to the target computing node with the lightest current load, the resource utilization efficiency of the computing nodes can be improved. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the structure of a load balancing system provided in an embodiment of this application;

[0027] Figure 2 This is a timing diagram of a load balancing method provided in an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of a computing node list provided in an embodiment of this application;

[0029] Figure 4 This is a functional schematic diagram of a load balancer provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram illustrating the principle of an endpoint discovery server provided in an embodiment of this application;

[0031] Figure 6 This is a flowchart illustrating the update of a first performance metric parameter of a network endpoint, as provided in an embodiment of this application.

[0032] Figure 7 This is a schematic diagram of the workflow for a business data processing request provided in an embodiment of this application. Figure 1 ;

[0033] Figure 8 This is a schematic diagram of the workflow for a business data processing request provided in an embodiment of this application. Figure 2 ;

[0034] Figure 9 This is a schematic diagram of the structure of a load balancing device provided in an embodiment of this application;

[0035] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0037] Kubernetes (K8s, an open-source container orchestration system) can be used to automate the deployment, scaling, and management of containerized applications. Kubernetes aims to provide a highly scalable and easy-to-manage platform for deploying, scaling, and managing containerized applications.

[0038] A federated Kubernetes cluster is an architecture that combines multiple independent Kubernetes clusters for management and resource coordination. The individual Kubernetes clusters within a federated cluster can be located in different geographical regions, data centers, or cloud environments. A geographical region can refer to a defined spatial area, such as a province, city, district, or street. The spatial scope indicated by this geographical region can be determined based on the geographical locations where the individual Kubernetes clusters within the federated cluster are deployed.

[0039] A load balancer (LB) is a device or software application that distributes network traffic or workloads to multiple servers (e.g., web servers, application servers, etc.) or resources according to specific rules. By rationally allocating requests, the response speed and availability of containerized applications deployed in a Kubernetes cluster can be improved; where containerized applications can refer to any application running in a Kubernetes cluster, this application does not limit the type of containerized application.

[0040] Deployment (a resource object): A resource object used to manage the lifecycle of stateless applications and ensure that applications always maintain the desired state. It allows users to automatically control the number of application replicas, perform downtime-free updates, scaling up, scaling down, and other operations in a declarative manner. In this embodiment, Deployment can be used to maintain the desired state of a group of Serving Pods (the smallest deployable and manageable computing unit in a Kubernetes cluster used to provide services externally, which can be called a container set) in a Kubernetes cluster, including the number of replicas, image versions, etc. Deployment can automatically perform rolling updates and rollbacks, ensuring that containerized applications deployed in the Kubernetes cluster remain available during the update process. Simultaneously, Deployment also has self-healing capabilities, capable of detecting the status of Serving Pods and automatically rebuilding abnormal Serving Pods, maintaining the high availability of containerized applications deployed in the Kubernetes cluster.

[0041] A Service provides a stable IP (Internet Protocol) address and DNS (Domain Name System) name to a group of Serving Pods, enabling other applications or users to access these Serving Pods. A Service can support simple load balancing strategies, distributing client business data processing requests to multiple Serving Pods to improve scalability and reliability.

[0042] Endpoint: Defines an accessible network endpoint. In a Kubernetes cluster, an endpoint can represent the IP address and port of its corresponding Serving Pod, providing a network abstraction without needing to know the specific information of the Serving Pod.

[0043] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a load balancing system provided in an embodiment of this application. The load balancing system may include, but is not limited to, a load balancer watcher, a domain name server, a client, a load balancer, an endpoint discovery server, and a collection of all containers (serving pods) located in the same geographical area as the load balancer. This application does not limit the number of load balancer watchers, clients, load balancers, endpoint discovery servers, and serving pods included in the load balancing system. The endpoint discovery server in the load balancing system can be represented as an EDS (Endpoint Discovery Service) server.

[0044] In this load balancing system, the load balancer, load balancer, endpoint discovery server, and Serving Pods can all be deployed on a federated Kubernetes cluster. The domain name server in this load balancing system can be deployed separately outside the Kubernetes system. For example, a federated Kubernetes cluster can include multiple Kubernetes clusters located in different geographical regions. Each Kubernetes cluster can run one or more Serving Pods, and a Serving Pod can contain one or more containers; therefore, a Serving Pod can be called a container collection. Serving Pods provide logical grouping of runtime environments for containers, facilitating resource allocation, scheduling, management, and inter-container communication. Different containers can run in isolation, and each container can run different applications; the applications running in containers can be called containerized applications.

[0045] A load balancer can be deployed for each Kubernetes cluster within a specific geographic region to manage all Serving Pods running in that cluster. All load balancers in this system can be dynamic load balancers. In other words, the load balancing system can include multiple load balancers, which can be deployed in different geographic regions.

[0046] In this load balancing system, a load balancer detector and an endpoint discovery server can be deployed for one load balancer; alternatively, a load balancer detector and an endpoint discovery server can be deployed for multiple load balancers; or, a load balancer detector and an endpoint discovery server can be deployed for all load balancers. In practical applications, the number of load balancer detectors and endpoint discovery servers included in the load balancing system can be determined based on their performance.

[0047] like Figure 1 As shown, the load balancing system includes load balancing detection 10a, domain name server 10b, client 10c, and multiple load balancers (e.g., Figure 1 The following description uses load balancers 10d, 10e, etc., endpoint discovery server 10f, and the corresponding Serving Pods for each load balancer as examples. A load balancer can be multiple resource objects (deployments). Endpoint discovery server 10f can be a single resource object (deployment), and it can also deploy a Service pointing to itself. Load balancer detector 10a can be a customized Kubernetes component, Bind (Berkeley Internet Name Domain, an open-source software implementing a DNS server), which manages the registration, deletion, and creation of load balancers by detecting their labels. It can be understood that when creating load balancers, a label can be set for each load balancer (for ease of understanding, this label can be called the first label). Different load balancers can be set to the same first label, such as load balancers with the same function or service. Load balancer detector 10a can manage (register, delete, etc.) all load balancers carrying a specific first label by detecting that first label.

[0048] A Serving Pod can be considered a workload, deployed as a resource object on a federated Kubernetes cluster, existing across multiple geographically dispersed physical Kubernetes clusters. Each Serving Pod in the federated Kubernetes cluster can be assigned a label (for ease of understanding, this label can be referred to as a secondary label). Multiple Serving Pods in the Kubernetes cluster can be assigned the same secondary label; for example, Serving Pods providing the same service can be assigned the same secondary label, such as image labeling, text labeling, video labeling, or game labeling. The load balancer's primary label and the Serving Pod's secondary label can be the same or different; this application does not impose restrictions on the setting of the primary and secondary labels. The endpoint discovery server 10f can detect cluster change events (Kubernetes events) of all Serving Pods carrying a specific secondary label by detecting that label. Among them, cluster change events are a type of resource object used as a message mechanism to record important activities or state changes in the lifecycle of Serving Pods within a K8s cluster. For example, they can be used to record changes in the state of Serving Pods such as creation, deletion, startup, shutdown, startup failure, and encountering errors. This application does not limit the type of cluster change events.

[0049] A Serving Pod can include multiple instances, such as instance 1, instance 2, etc. The Serving Pod can receive and execute business data processing requests through port 1. Sidecar containers within the Serving Pod can obtain runtime load metrics data from the Serving Pod through port 2 and transmit this load metric data to the endpoint discovery server 10f. An instance can be a group of processes running a containerized application; a process can refer to a program running within the Serving Pod, and this process can be the basic unit for resource allocation and scheduling by the operating system. Port 1 can be used to provide specific services to clients; for example, port 1 can be 8000 or other port numbers. Port 2 can be 8001 or other port numbers; this application does not limit this.

[0050] A sidecar container can be an auxiliary container running in the same Serving Pod as the business container (e.g., the main container used to execute business data processing requests). Business data processing requests can be used to request computing resources from compute nodes in the Kubernetes cluster for data processing. These requests can be any type of request initiated by the client, such as web page access requests, message sending requests, model inference service requests, database query requests, data storage requests, game session requests, data transfer requests, configuration update requests, etc. This application does not limit the type of business data processing request. The client can be a browser or an application client; this application does not limit the type of client.

[0051] Load metrics may include, but are not limited to: average throughput latency of the Serving Pod, request success rate and error rate within a given period, number of active requests, and average response time. Average throughput latency is a crucial metric for measuring the Serving Pod's ability and efficiency in executing business data processing requests. It represents the relationship between the number of successfully executed business data processing requests (throughput) per unit time and the average latency experienced by similar business data processing requests within the Serving Pod. Throughput, on the other hand, refers to the total number of business data processing requests a Serving Pod can execute per unit time.

[0052] The request execution success rate can refer to the ratio of the number of business data processing requests successfully executed by a Serving Pod within a certain time period (e.g., within the last 24 hours, last 12 hours, last hour, last minute, etc., which is not limited in this application). The request error rate can refer to the ratio of the number of business data processing requests that fail to execute by a Serving Pod within a certain time period to the total number of business data processing requests that the load balancer sends to the Serving Pod. Execution errors can include, but are not limited to: situations where the Serving Pod does not receive a business data processing request, situations where the execution of a business data processing request times out, situations where the execution of a business data processing request is interrupted, etc. For example, if the load balancer sends 10 business data processing requests to a Serving Pod in one minute, and the Serving Pod successfully executes 8 business data processing requests in one minute, then the request execution success rate of the Serving Pod in one minute can be expressed as 8 / 10 = 80%; the request error rate of the Serving Pod in one minute can be expressed as 2 / 10 = 20%.

[0053] The number of active requests refers to the number of business data processing requests that the Serving Pod is currently executing or waiting to execute. This includes the number of business data processing requests that the Serving Pod has received but has not yet returned a result. This number of active requests does not include the number of business data processing requests that the Serving Pod has already completed. The average response time refers to the average time spent by the Serving Pod in responding to (executing) all received business data processing requests. The time taken from receiving a business data processing request, executing it, and returning a result can be called the response time of that business data processing request.

[0054] In one or more embodiments, a Serving Pod may also include a LivenessProbe and an AvailabilityProbe. A LivenessProbe is a mechanism used to detect whether a Serving Pod is still running normally, ensuring that the application inside the container within the Serving Pod remains in a normal working state during operation. If a failure or unresponsiveness is detected in the Serving Pod, appropriate measures can be taken according to the configuration, such as restarting the Serving Pod. An AvailabilityProbe can be used to determine whether a Serving Pod is ready to receive traffic (e.g., business data processing requests). Unlike a LivenessProbe, an AvailabilityProbe can determine whether a Serving Pod has reached a state where it can execute business data processing requests, not just whether the Serving Pod is running normally. For example, a Serving Pod may have started and is running, but it may still be performing some initialization operations (e.g., loading cache, connecting to a database, etc.). Before these initialization operations are completed, the Serving Pod is not suitable for receiving business data processing requests, and the availability probe result in this case may be unavailable.

[0055] Endpoint discovery server 10f can detect cluster changes in the serving load within a federated Kubernetes cluster, acquire and delete endpoints, and update the first performance metric (weight) for each endpoint. An endpoint can refer to a Serving Pod detected by endpoint discovery server 10f that is maintaining normal operation, and the port used by that Serving Pod to receive business data processing requests, such as... Figure 1Port 1 is shown; the calculation method for the first performance metric parameter of the Endpoint will be described later. The Endpoint Discovery Server 10f can dynamically update the Endpoint and its first performance metric parameter based on the load metric data of each Serving Pod; subsequently, it can provide the updated Endpoint and updated first performance metric parameter (weight) to each load balancer via streaming remote procedure calls (Google Remote Procedure Call, gRPC). Each load balancer can access the Endpoint Discovery Server 10f through the EDS service without needing to know the specific information of the Serving Pod.

[0056] The load balancing detector 10a in the load balancing system can detect the first network address and first geographical location information of multiple load balancers, and transmit the detected first network address and first geographical location information of each load balancer to the domain name server 10b. Each load balancer corresponds to one first geographical location information and one first network address. In other words, the domain name server 10b can store the first network address and first geographical location information of all load balancers transmitted by the load balancing detector 10a. When the load balancing detector 10a detects a change in the number of load balancers (such as the creation or deletion of load balancers), it can dynamically update the first network address and first geographical location information and transmit the dynamically updated information to the domain name server 10b for storage.

[0057] Here, the first network address can refer to the IP address of each load balancer detected by the load balancer detector 10a, and the first geographical location information can refer to the geographical location of each load balancer detected by the load balancer detector 10a, such as the province, city, district, and street where the load balancer is located. For example, suppose the domain name server 10b has stored the first network address and first geographical location information of two load balancers, such as the first network address 1 and first geographical location information 1 of load balancer 10d, and the first network address 2 and first geographical location information 2 of load balancer 10e. When the load balancer detector 10a detects that a new load balancer B has been created, it can obtain the first network address 3 and first geographical location information 3 of load balancer B and transmit the first network address 3 and first geographical location information 3 to the domain name server 10b. The domain name server 10b can update its stored first network address and first geographical location information, such as storing the first network address 3 and first geographical location information 3.

[0058] Understandable Figure 1In the load balancing system shown, client 10c can refer to a device or software program that initiates a business data processing request. When client 10c is a software program, it can run on different types of terminal devices, such as smartphones, tablets, laptops, PDAs, desktop computers, wearable devices (e.g., smartwatches, smart bracelets), smart voice interaction devices, smart home appliances (e.g., smart TVs), in-vehicle devices, aircraft, and other electronic devices. This application does not limit the type of terminal device on which client 10c runs.

[0059] Figure 1 The domain name server 10b and endpoint discovery server 10f shown can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not limit the type of backend server.

[0060] In the embodiments of this application, Figure 1 Taking any business data processing request initiated by client 10c as an example, the processing procedure of such a request in the load balancing system is described. Client 10c can access domain name server 10b by generating a unique domain name. Domain name server 10b resolves the domain name of client 10c to determine the second geographical location information of client 10c. A domain name can refer to a character identifier used on the Internet to identify and locate a computer, network service, or other resource; the domain name of client 10c can also refer to a character identifier used to identify and locate client 10c.

[0061] Based on the first geographical location information and the first network address stored in the domain name server 10b, a second network address that is within the same geographical area as the second geographical location information is determined from among the first network addresses corresponding to multiple load balancers. This second network address is then returned to the client 10c. The load balancer corresponding to this second network address can be called the target load balancer. For example, when the second network address is... Figure 1 When the IP address of the load balancer 10e shown is given, the load balancer 10e can be referred to as the target load balancer.

[0062] In other words, based on the first geographical location information stored in the domain name server 10b and the second geographical location information of the client 10c, the target load balancer geographically closest to the client 10c can be determined. For example, when multiple load balancers are deployed in different provinces, the load balancer 10e located in the same province as the client 10c can be determined as the target load balancer; alternatively, when multiple load balancers are deployed in different cities, the load balancer 10e located in the same city as the client 10c can be determined as the target load balancer, and so on.

[0063] Alternatively, the domain name server 10b in the load balancing system can be replaced by other components, such as a database for storing the IP addresses and ports of load balancers in various geographical areas, and this application does not limit this.

[0064] After client 10c initiates a business data processing request, it can send the request to the target load balancer (such as load balancer 10e). This means that the request can be sent to the load balancer 10e, which is geographically closest to the client. This reduces network latency in the load balancing system and improves the response speed of the request.

[0065] After receiving a business data processing request from client 10c, load balancer 10e can select the lightest-loaded Serving Pod from all Serving Pods managed by load balancer 10e, based on the first performance metric parameters of the Endpoint transmitted from endpoint discovery server 10f to load balancer 10e, and the number of active requests to the Serving Pod indicated by the Endpoint. This lightest-loaded Serving Pod can be referred to as the target compute node. The business data processing request is sent to the target compute node via port 1. Upon receiving the request, the target compute node can execute the business data processing request and obtain the business data processing result. This result can be returned to load balancer 10e, which can then return it to client 10c. The process by which load balancer 10e selects the target compute node will be described in detail later.

[0066] In this embodiment, the Endpoint and its first performance metric parameter can be updated in real time and periodically to accurately reflect the current load of each Serving Pod. The Serving Pod can be registered with the corresponding load balancer to ensure that the load balancer holds the latest Serving Pod, thereby achieving optimal allocation of traffic (business data processing requests) and efficient utilization of Serving Pods.

[0067] In one or more embodiments, after the domain name server returns a second network address referring to the target load balancer to the client, the flow of the business data processing request initiated by the client can be referred to [reference needed]. Figure 2 , Figure 2 This is a timing diagram illustrating a load balancing method provided in an embodiment of this application. It can be understood that this load balancing method is executed interactively by a target load balancer, a client, and a target compute node. The target load balancer can refer to a load balancer located within the same geographical region as the client, and is determined for the client by a domain name server in the load balancing system. Multiple load balancers can be deployed in different geographical regions. The client can refer to any client initiating a business data processing request, and the target compute node can refer to a Serving Pod allocated by the target load balancer for the business data processing request initiated by the client. Figure 2 As shown, the load balancing method may include the following steps S101 to S108:

[0068] Step S101: The client sends a business data processing request to the target load balancer.

[0069] In this context, a business data processing request can refer to an instruction or message aimed at obtaining specific business function processing or services. A client can refer to a device or software application requesting resources or services, such as various application clients running on a terminal device, or a browser running on a terminal device. When a client needs to perform a certain business operation, it can send a business data processing request to the server, which can be a Serving Pod running in a federated Kubernetes cluster. This business data processing request can be a data query request, a data update request, a transaction processing request, a resource acquisition request, etc. This application does not limit the type of client or the type of business data processing request. In a load balancing scenario, the business data processing request sent by the client to the server can be sent to the target load balancer, which will then allocate a suitable Serving Pod for the business data processing request.

[0070] Step S102: The target load balancer obtains the list of computing nodes stored in the target load balancer.

[0071] The target load balancer can store a list of compute nodes, which includes the IP addresses of all active nodes located within the same geographical region as the target load balancer, as well as a first performance metric parameter for each active node. The target load balancer, the client, and all active nodes in the compute node list are located within the same geographical region. Each active node can correspond to an IP address and a first performance metric parameter. The first performance metric parameter of an active node can be used to indicate its data processing performance, such as its ability to handle current business data processing requests. The value range of the first performance metric parameter is [0,1]. Among all active nodes in the compute node list, a larger first performance metric parameter (closer to 1) indicates higher data processing performance and a greater likelihood of executing the current business data processing request; a smaller first performance metric parameter (closer to 0) indicates lower data processing performance and a lower likelihood of executing the current business data processing request. In this embodiment of the application, the active node in the compute node list can be called an Endpoint (network endpoint). An active node (Endpoint) can be represented as a Serving Pod that is in normal working order. The IP address of the active node can also be called the network endpoint address.

[0072] In one or more embodiments, the target load balancer may obtain the list of compute nodes in a manner that includes, but is not limited to: establishing a remote procedure call channel between the target load balancer and the endpoint discovery server, and obtaining, through this remote procedure call channel, the network endpoint addresses and first performance metric parameters of the network endpoint addresses sent by the endpoint discovery server. The network endpoint addresses and first performance metric parameters are transmitted from the endpoint discovery server to the target load balancer in a streaming data transmission mode, with each network endpoint address referring to an active node in the list of compute nodes. Based on all network endpoint addresses and all first performance metric parameters transmitted by the endpoint discovery server, a list of compute nodes corresponding to the target load balancer can be generated.

[0073] Among them, a remote procedure call channel is a communication method used to implement the remote procedure call mechanism in a distributed computing environment. Streaming data transmission mode is a data transmission method in which data is transmitted from the source to the destination in the form of a continuous stream, rather than in discrete, fixed-size data blocks (such as the entire file in file transfer). In this embodiment, the updated network endpoint address and first performance metric parameters calculated by the endpoint discovery server can be transmitted from the endpoint discovery server to the target load balancer in the form of a stream.

[0074] The network endpoint address transmitted by the endpoint discovery server to the target load balancer can refer to the IP address of an active node located within the same geographical region as the target load balancer. When the network endpoint address and the first performance metric parameter are updated, the endpoint discovery server can proactively transmit the updated network endpoint address's access address to the target load balancer. Optionally, the target load balancer can periodically retrieve updated network endpoint addresses and the first performance metric parameter from the endpoint discovery server, such as through a scheduled task or other methods. This application does not limit the method by which the target load balancer obtains updated network endpoint addresses and the first performance metric parameter.

[0075] It is understood that each of the multiple load balancers can store a list of compute nodes, and all load balancers can obtain their own list of compute nodes in the same way as the target load balancer; this will not be elaborated further here. In one or more embodiments, the endpoint discovery server can update the network endpoint address and its corresponding first performance metric parameter for each load balancer, and transmit the updated network endpoint address and first performance metric parameter to the corresponding load balancer in the form of a stream.

[0076] Please see Figure 3 , Figure 3 This is a schematic diagram of a computing node list provided in an embodiment of this application. Taking a target load balancer as an example, the computing node list corresponding to the target load balancer is described. Assume the computing node list corresponding to the target load balancer is as follows: Figure 3 The compute node list 20a is shown in the diagram. This compute node list 20a may include the IP addresses (network endpoint addresses) and first performance metric parameters corresponding to M active nodes, with one IP address corresponding to one first performance metric parameter. Here, M is a positive integer, and M can be used to represent the number of active nodes (Endpoints, representing Serving Pods that are maintaining normal operation) within the same geographical region as the target load balancer.

[0077] Understandably, among the Serving Pods located within the same geographical region as the target load balancer, some Serving Pods may not be in a normal working state (e.g., in a faulty state). These Serving Pods not in a normal working state can be referred to as inactive nodes. The compute node list 20a only contains the IP addresses and primary performance metric parameters of all active nodes corresponding to the target load balancer. The endpoint discovery server may not be able to detect inactive nodes.

[0078] The IP addresses and their corresponding first performance metric parameters in the compute node list 20a can be dynamically updated. For example, when the endpoint discovery server detects a new active node, it can transmit the IP address and first performance metric parameters (e.g., IP address M+1 and first performance metric parameter M+1) of the new active node to the target load balancer. At this time, the compute node list 20a can be updated with IP address M+1 and first performance metric parameter M+1. When the endpoint discovery server cannot detect the Serving Pod corresponding to IP address 3, it can delete IP address 3 and its corresponding first performance metric parameter 3 from the compute node list 20a based on the data transmitted by the endpoint discovery server. It is understood that the endpoint discovery server can obtain the load metric data of each active node in real time, dynamically update the active nodes and their corresponding first performance metric parameters, and transmit the dynamically updated IP addresses and their corresponding first performance metric parameters of the active nodes to the target load balancer. The target load balancer can then update the IP addresses and their corresponding first performance metric parameters in the compute node list 20a based on the data transmitted by the endpoint discovery server, thus allowing the target load balancer to know the real-time performance status of each active node.

[0079] Step S103: The target load balancer determines the second performance index parameter of each computing node based on the first performance index parameter of each computing node in the list of computing nodes and the number of active requests for each computing node.

[0080] The multiple computing nodes can be all active nodes in the computing node list (represented by IP addresses) or a subset of active nodes in the list. In other words, when the computing node list contains multiple active nodes, these multiple active nodes can include multiple computing nodes, meaning the number of multiple computing nodes is less than or equal to the number of active nodes in the computing node list. The multiple computing nodes can be randomly selected from the computing node list, or they can be selected from the list according to a certain selection strategy. This application does not limit the selection method for multiple computing nodes. The second performance indicator parameter can refer to the indicator parameter used by the target load balancer to measure the real-time performance status of multiple computing nodes. For example, it can be used to indicate the performance of a computing node in handling current business data processing requests under the current workload. The value range of the second performance indicator parameter is [0,1]. Among multiple computing nodes, a larger second performance indicator parameter, such as one closer to 1, indicates a better performance status and a lighter current workload for the computing node; a smaller second performance indicator parameter, such as one closer to 0, indicates a worse performance status and a heavier current workload for the computing node. This second performance metric parameter can serve as the basis for the target load balancer to allocate computing nodes for business data processing requests. It can distribute business data processing requests to the computing node with the lightest workload among multiple computing nodes, thereby improving the load balancing effect.

[0081] In one or more embodiments, the target load balancer can obtain the number of active requests for each active node in the compute node list, such as the number of active requests for the Serving Pod indicated by each IP address in the compute node list. Iterate through each active node in the compute node list, adding active nodes with an active request count less than a request count threshold to the node selection set. When the number of active nodes in the node selection set is greater than or equal to the node count threshold, randomly select multiple active nodes from the node selection set as selected compute nodes; at this point, the number of compute nodes equals the node count threshold.

[0082] The request quantity threshold can refer to the maximum number of requests used to determine whether an active node is in a busy state; the node quantity threshold can be used to represent the maximum number of selectable computing nodes; both the request quantity threshold and the node quantity threshold can be flexibly set according to the specific needs of the actual application scenario, and this application does not limit them.

[0083] For any active node in the computing node list (e.g., active node N1), determining whether it is in a busy state can include: if the number of active requests for active node N1 is greater than or equal to a request number threshold, then active node N1 is determined to be in a busy state; if the number of active requests for active node N1 is less than the request number threshold, then active node N1 is determined to be in an available state. Here, a busy state indicates that active node N1 currently has no idle resources to process business data processing requests, while an available state indicates that active node N1 currently has idle resources to process business data processing requests.

[0084] All active nodes in the node selection set are in an available state. When the node number threshold is N (N is a positive integer, such as N can be any positive integer less than or equal to M above), and the number of active nodes in the node selection set is greater than or equal to the node number threshold, N active nodes can be randomly selected from the node selection set as multiple computing nodes.

[0085] Optionally, when there are active nodes in the node selection set, and the number of active nodes in the node selection set is less than the node number threshold, all active nodes in the node selection set can be used as multiple computing nodes selected from the computing node list.

[0086] Optionally, when the node selection set is empty, meaning all active nodes in the compute node list are busy, multiple compute nodes cannot be selected temporarily. In this case, the target load balancer can add the business data processing request to the request waiting queue, waiting for active nodes in the compute node list to release idle resources to process the request. Once an active node in the compute node list is detected to be available, the business data processing request in the request waiting queue can be assigned to that active node for processing. This request waiting queue is a queue maintained by the target load balancer, used to store business request data that cannot be assigned temporarily due to the Serving Pod being busy. The queue follows a first-in, first-out (FIFO) data structure; business data processing requests added to the request waiting queue first are preferentially assigned to active nodes that are available.

[0087] After the target load balancer selects multiple compute nodes from the compute node list, it determines the request index value for each compute node based on the request skew parameter and the number of active requests for each compute node. This request skew parameter is used to balance the importance of the first performance metric parameter and the number of active requests. Based on the first performance metric parameter and the request index value for each compute node, the second performance metric parameter is determined for each compute node.

[0088] For each of the multiple computing nodes, the calculation process of its corresponding second performance index parameter can be shown in formula (1):

[0089]

[0090] Endpoint i w can represent the i-th computing node among multiple computing nodes, where i is a positive integer less than or equal to the number of computing nodes (e.g., N). The i-th computing node can be any one of the multiple computing nodes. r (Endpoint i ) can refer to the second performance metric parameter of the i-th computing node; f r (Endpoint i () can refer to the endpoint discovery server calculating and updating the first performance metric parameter for the i-th compute node.

[0091] `active_requests` represents the number of active requests currently active on the i-th compute node. `active_request_bias` represents the request bias parameter, which balances the importance of the primary performance metric and the number of active requests. It can be flexibly set according to the actual application scenario; for example, `active_request_bias` can be set to 1, or other values. (`active_requests+1`) active_request_bias This can represent the request index value corresponding to the number of active requests of the i-th compute node. Here, (active_requests+1) represents the sum of the number of active requests of the i-th compute node and the number of business data processing requests currently received by the target load balancer.

[0092] In one or more embodiments, the first performance metric parameter of each active node (Endpoint) in the compute node list is calculated and updated by the endpoint discovery server in the load balancing system. The endpoint discovery server then transmits all active nodes of the target load balancer and their corresponding first performance metric parameters to the target load balancer. The calculation method for the first performance metric parameter of each active node is similar. For ease of understanding, the i-th compute node (which can be denoted as Endpoint) will be referred to below. i Taking as an example, the calculation method of the first performance index parameter of active nodes is described.

[0093] Endpoint i The first performance index parameter f r (Endpoint i The calculation method for ) can be shown in formula (2):

[0094]

[0095] Among them, W init Endpoint i Initialization performance metrics parameters. latency avg Endpoint i The success_rate represents the average response time over a recent period (e.g., the last 24 hours, the last hour, the last minute, etc.). i The request success rate over a recent period, where max(·) represents the maximum value; both average response time and request success rate can be included in Endpoint. i In the load metric data, Endpoint i The load metric data can be obtained from this Endpoint i The sidecar container in the Serving Pod reports to the endpoint discovery server. C1, C2, C3, and C4 are constants and can be flexibly configured according to the specific needs of the actual application scenario. This application does not impose any restrictions on them.

[0096] When Endpoint i When the corresponding Serving Pod is first started, the auxiliary containers within that Serving Pod have not yet reported load metric data to the endpoint discovery server. In other words, there is no load metric data available at this time to dynamically calculate the Endpoint. i The first performance metric parameter is used, therefore the initial performance metric parameter of the Serving Pod can be used as the endpoint. iThe current primary performance metric.

[0097] Alternatively, if the performance differences among all active nodes within the same geographical region as the target load balancer are not significant, constant terms C1, C2, and Endpoint can be used. i Average response time avg Calculate the Endpoint based on the success rate (success_rate) of the request execution. i The current primary performance parameter f r (Endpoint i If the performance differences among all active nodes within the same geographical region as the target load balancer are too large, constant terms C3 and C4, as well as Endpoint, can be used. i Average response time avg Calculate the Endpoint based on the success rate (success_rate) of the request execution. i The current primary performance parameter f r (Endpoint i ).

[0098] In one or more embodiments, the endpoint discovery server can determine the performance differences between active nodes based on at least one of the following information: resource usage metrics, network performance metrics, and application performance metrics of each Serving Pod located within the same geographical region as the target load balancer. Resource usage metrics may include, but are not limited to, CPU (Central Processing Unit) utilization and memory usage. Network performance metrics may include, but are not limited to, network bandwidth and network latency. Application performance metrics may include, but are not limited to, response time and throughput.

[0099] For example, CPU utilization can be used to reflect the proportion of CPU resources used by a Serving Pod during operation. If the difference in CPU utilization between different Serving Pods is greater than a pre-set first threshold, it can be determined that the performance difference between active nodes is too large. If the difference in CPU utilization between different Serving Pods is less than the first threshold, it can be determined that the performance difference between active nodes is not significant. Memory usage can be used to represent the amount of memory used by a Serving Pod during operation. If some Serving Pods have memory usage close to their resource limits, while others have very low memory usage, it can be determined that the performance difference between active nodes is too large. If the memory usage of different Serving Pods is all within a certain range, it can be determined that the performance difference between active nodes is not significant.

[0100] Network bandwidth measures the speed of data transfer between Serving Pods; network latency represents the time it takes for data to be sent from one Serving Pod to another; response time is the time interval between receiving a business data processing request from a Serving Pod and returning the business data processing result; throughput refers to the number of business data processing requests or the amount of data that a Serving Pod can process per unit of time. Similar to the methods used for CPU utilization and memory usage, performance differences between active nodes can be determined by using at least one of the following: CPU utilization, memory usage, network bandwidth, network latency, response time, and throughput. Examples of each will not be provided here.

[0101] In one or more embodiments, the endpoint discovery server can obtain all endpoints within the same geographical region as the target load balancer by detecting the second tag carried by the Serving Pod. Based on all the obtained endpoints, it constructs a network endpoint list for the target load balancer. This EndpointList contains all endpoints detected by the endpoint discovery server for the target load balancer, i.e., all Serving Pods that are maintaining normal operation. The dynamic weights of all endpoints in the Endpoint List can be represented by a weight array (which can be called the Endpoints_weight array). Here, the dynamic weight can refer to the first performance metric parameter dynamically calculated by the endpoint discovery server based on the load metric data of each endpoint. That is, the initial performance metric parameters of each endpoint in the EndpointList do not need to be added to the Endpoints_weight array.

[0102] When Endpoint i When the corresponding Serving Pod is first started, this Endpoint i Initialization performance index parameter W init The determination process may include, but is not limited to: when all Endpoints in the Endpoint List have no dynamic weights, i.e., the Endpoints_weight array is empty, the weight can be determined based on the Endpoint. i The corresponding number of compute and memory resources for the Serving Pod, and the calculation of the Endpoint. i Initialization performance index parameter W init At this point, the initial performance parameter W init This can be represented as: parse_func(CPU limit )×CPU factor +parse_func(memory limit )×memory factor Among them, parse_func(·) is a parsing function, CPU limit Used to represent Endpoint i The constraints on the amount of CPU resources that the corresponding Serving Pod can use; for example, CPU limit It can be represented as Endpoint i The maximum amount of CPU resources that the corresponding Serving Pod is allowed to use for the UI. (CPU) factor Endpointi The number of compute resources corresponding to the Serving Pod, which can refer to the number of compute resources of the Kubernetes cluster as the Endpoint. i The actual CPU resources allocated to the corresponding Serving Pod can be represented by the number of CPU cores (e.g., 1 CPU core) or other methods. limit It can be used to represent Endpoint i The maximum amount of memory that the corresponding Serving Pod can use can be represented in bytes. factor Endpoint i The memory resources of the corresponding Serving Pod, which can refer to the Endpoint i The actual amount of memory used by the corresponding Serving Pod.

[0103] After the Kubernetes cluster has been running normally for a period of time, some endpoints in the Endpoint List will have dynamic weights, meaning the Endpoints_weight array will contain data. The average of the top 20% of the dynamic weights in the Endpoints_weight array can then be calculated and used as the endpoint weight. i The corresponding initial performance index parameter W init At this point, the initial performance parameter W init This can be represented as: avg(top 20% of sorted(Endpoints_weight)). Here, 20% is merely an example in this embodiment; in actual applications, it can be flexibly set according to actual needs, such as 10%, 15%, 30%, etc., and this application does not limit this. `sorted(·)` represents a sorting function that can be used to sort the dynamic weights in the `Endpoints_weight` array in descending order. `avg(·)` represents a function to calculate the average of the top 20% of the dynamic weights in the `Endpoints_weight` array. By using the above method to calculate the initial performance metric parameters for newly started Serving Pods, it is possible to ensure that Serving Pods quickly receive business data processing requests after a cold start, improving the resource utilization efficiency of Serving Pods.

[0104] Step S104: The target load balancer determines the target computing node among multiple computing nodes based on the second performance index parameter of each computing node.

[0105] Step S105: The target load balancer forwards the business data processing request to the target computing node.

[0106] In one or more embodiments, if multiple compute nodes are randomly selected from a node selection set, then after the target load balancer calculates the second performance index parameters corresponding to the multiple compute nodes, it can determine the compute node with the largest second performance index parameter as the target compute node. Here, the target compute node represents the Serving Pod that is in an available state and has the lightest load; then, the business data processing request initiated by the client can be forwarded to the target compute node.

[0107] In one or more embodiments, if the multiple compute nodes are multiple Endpoints (active nodes) randomly selected from a list of compute nodes, then the compute node corresponding to the largest second performance metric parameter can be determined as a candidate compute node. If the candidate compute node is in a busy state, such as when the Serving Pod corresponding to the candidate compute node is in a busy state, the business data processing request can be added to the request waiting queue of the target load balancer. When an available compute node is detected in the list of compute nodes, the available compute node can be determined as the target compute node, and the business data processing requests in the request waiting queue can be forwarded to the target compute node.

[0108] Specifically, the relationship between the number of active requests to a candidate computing node and a request threshold can be used to determine whether the candidate computing node is in a busy state. The method by which the target load balancer determines whether a candidate computing node is in a busy state can be found in step S103, which describes how to determine whether active node N1 is in a busy state; it will not be repeated here.

[0109] Step S106: The target computing node executes the business data processing request.

[0110] Step S107: The target computing node returns the business data processing result of the business data processing request to the target load balancer.

[0111] Specifically, the target computing node can be accessed through a specific port (such as...). Figure 1Port 1 (which could be port 8000 or another port number) receives business data processing requests forwarded by the target load balancer. The container in the Serving Pod corresponding to the target compute node is the actual environment for executing the business data processing requests. Specific applications can run within the container, executing the business data processing requests and obtaining the processing results. The processing results are then returned to the target load balancer via the specific port used to receive the requests.

[0112] The business data processing result can be the response of the target computing node to the business data processing request. For example, when the business data processing request is a query request, the business data processing result can be the query result; when the business data processing request is a configuration update request, the business data processing result can be the configuration update result; when the business data processing request is a model inference service request, the business data processing result can be the model output result, etc.

[0113] It is understood that the target computing node can execute the business data processing request in a container of the request type it receives, based on the request type of the business data processing request. Business data processing requests of different request types may be executed in different containers; different containers may run the same application or different applications, and this application does not limit this.

[0114] Step S108: The target load balancer returns the business data processing result to the client.

[0115] Specifically, after receiving the business data processing results returned by the target computing node, the target load balancer can return the business data processing results to the client, and the client can continue to perform subsequent data processing based on the business data processing results.

[0116] In this embodiment, the target load balancer in the load balancing system can interact with the load balancing detector, endpoint discovery server, clients, and Serving Pods within the system. The target load balancer can be dynamically detected by the load balancing detector using a first tag it carries. The load balancing detector can register the target load balancer's information (including its first network address and first geographical location information) with a domain name server for storage. The target load balancer can communicate with the endpoint discovery server, obtaining dynamically updated endpoints and their corresponding first performance metric parameters through a remote procedure call service using streaming data transmission, without needing to understand the internal details of the endpoint discovery server. The target load balancer can provide specific services to clients by exposing a specific port (e.g., port 8000), such as online inference capabilities for models, data query services, and configuration update services.

[0117] Please see Figure 4 , Figure 4 This is a functional schematic diagram of a load balancer provided in an embodiment of this application. Figure 4 As shown, taking the target load balancer as an example, the functions of the load balancer in the load balancing system are described. In addition to the functions mentioned above, the target load balancer can be dynamically detected by a load balancer detector, obtain dynamically updated endpoints and their corresponding first performance metric parameters from the endpoint discovery server through the endpoint discovery service client (EDS client), and allocate the lightest-load Serving Pod to business data processing requests initiated by clients. It can also provide functions such as health checks, retry, timeout, active reqqueue, and metrics.

[0118] The target load balancer can periodically send test requests to Serving Pods within the same geographical region as the target load balancer through a health check function. This ensures that the Serving Pods are functioning correctly and capable of responding to client-initiated business data processing requests. For example, suppose the target load balancer's compute node list includes compute node 'a', which can represent any Serving Pod within the same geographical region. Test requests can be sent to compute node 'a' at preset intervals. If the target load balancer does not receive a test response from compute node 'a' within the first preset response time, it determines that compute node 'a' is in a faulty state and can be removed from the compute node list. This ensures that all Serving Pods corresponding to Endpoints in the compute node list are functioning correctly and capable of responding to business data processing requests. In other words, the health check function guarantees that all Endpoints in the compute node list are functioning normally as Serving Pods, improving the response speed of business data processing requests.

[0119] The first preset response time can be the maximum response time pre-set for the test request. This first preset response time can be flexibly set according to the specific requirements of the actual application scenario, and this application does not limit it. If a test response result is received within the first preset response time, it means that the Serving Pod that returned the test response result is running normally; if a test response result is received within the first preset response time, it means that the Serving Pod that did not return a test response result has failed and needs to be removed from the compute node list.

[0120] By introducing a health check mechanism within the load balancer of the load balancing system, test requests (e.g., heartbeats or diagnostic requests) are periodically sent to the Serving Pods to determine their operational status. If a Serving Pod is detected to be faulty or experiencing performance issues, the load balancing system can remove it from the compute node list, preventing any impact on the overall service quality. Furthermore, the health check mechanism supports automatic recovery. Once a faulty or performance-abnormal Serving Pod recovers, the endpoint discovery server in the load balancing system can re-add it to the compute node list. This proactive detection and automatic management approach improves the reliability and stability of the load balancing system.

[0121] When a business data processing request fails to be sent to a Serving Pod, the target load balancer can automatically retry the request using the Retry function, forwarding it to another active Serving Pod. The target load balancer can also use the Timeout function to set the maximum time it can wait for a response from a Serving Pod (e.g., the Serving Pod corresponding to the target compute node), preventing business data processing requests from being suspended indefinitely and improving the reliability of the load balancing system. Specifically, the target load balancer can use the Timeout function to set two preset response durations, referred to as the second preset response duration and the third preset response duration. The second preset response duration can refer to the maximum time set for a single response for a business data processing request; the third preset response duration can refer to the total response duration set for a business data processing request; the third preset response duration is longer than the second preset response duration. For example, the second preset response duration can be set to 2 seconds, and the third preset response duration can be set to 5 seconds.

[0122] For example, after the target load balancer forwards a business data processing request to the target compute node, if the target load balancer does not receive the business data processing result from the target compute node within a second preset response time, it indicates that the target load balancer's request to send the business data processing request to the target compute node failed. The load balancer can then change the target compute node for the business data processing request in the compute node list, determine the new target compute node, and forward the business data processing request to the new target compute node. If the target load balancer does not receive the business data processing result from the business data processing request within a third preset response time, it can determine that the business data processing request has failed and return a failure message to the client.

[0123] In one or more embodiments, when all the ServingPods corresponding to all active nodes in the compute node list are in a busy state, the target load balancer can use the active req queue function to queue new business data processing requests into the request waiting queue, waiting for available Serving Pod resources to process them, that is, waiting for an available Serving Pod to execute the business data processing request in the request waiting queue.

[0124] The metrics feature in the target load balancer can be used to collect and provide load metric data for the Serving Pods. This load metric data can include statistics on the performance and traffic of the Serving Pods. For example, this load metric data may include, but is not limited to, the number of active requests, response time (average response time), request error rate, etc., which facilitates the detection and optimization of the target load balancer.

[0125] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the principle of an endpoint discovery server provided in an embodiment of this application. For example... Figure 5 As shown, the endpoint discovery server in a load balancing system may include a set of K8s cluster watchers, a streaming gRPC server, and a weight controller.

[0126] The endpoint discovery server can provide dynamically updated endpoints (network endpoints, or active nodes) and their corresponding primary performance metric parameters (weights) to multiple load balancers via a remote procedure call service that transmits data through streaming. Each load balancer's endpoint and its corresponding primary performance metric parameters can be transmitted to each load balancer in the form of a stream via the remote procedure call channel.

[0127] The number of Kubernetes cluster detectors included in the endpoint discovery server matches the number of Kubernetes clusters in the federated Kubernetes cluster. One Kubernetes cluster detector can correspond to one Kubernetes cluster, such as... Figure 5 As shown, the Kubernetes Cluster X detector can be used to detect all Serving Pods in a Kubernetes cluster. The Kubernetes Cluster X detector can detect cluster change events (Kubernetes events) for each Serving Pod using the second tag carried by the Serving Pod. Based on the event type of the cluster change event, it initializes the Serving Pod and updates the Endpoint and its corresponding first performance metric parameters. For example, when the Kubernetes Cluster X detector detects that a new Serving Pod has started in the Kubernetes cluster, it can apply initialization performance metric parameters to that Serving Pod. The method for determining these initialization performance metric parameters can be found in the previous description and will not be repeated here. This new Serving Pod can be transmitted as a network endpoint (or an active node, denoted as Endpoint) to the weight controller in the endpoint discovery server, dynamically updating the Endpoint.

[0128] The Serving weight controller in the endpoint discovery server works in conjunction with the sidecar in the ServingPod (which can be considered the actual workload) to dynamically calculate the first performance metric parameter of the ServingPod based on the load metric data generated by the instance services in the ServingPod (e.g., instance 1, instance 2, etc.). The load metric data can be reported to the weight controller in the endpoint discovery server by the sidecar in the ServingPod. The calculation method for the first performance metric parameter can be found in formula (2), which will not be elaborated here. The weight controller can dynamically calculate the first performance metric parameter of the Endpoint based on the load metric data reported by the sidecar. This weight controller can trigger updates through a streaming remote procedure call service, that is, through a streaming remote procedure call service, it transmits the dynamically updated Endpoint and its corresponding first performance metric parameter to each load balancer.

[0129] Optionally, communication between the endpoint discovery server and the auxiliary containers in the Serving Pod can be achieved through HTTP (Hypertext Transfer Protocol) or gRPC (Remote Procedure Call). When communication between the endpoint discovery server and the auxiliary containers is achieved through gRPC, the endpoint discovery server can update load metric data by pulling or pushing (i.e., the auxiliary containers actively report to the endpoint discovery server). This application does not limit the communication method between the endpoint discovery server and the Serving Pod.

[0130] The endpoint discovery server can detect cluster change events of Serving Pods in each K8s cluster through the K8s cluster detector to obtain the service running status in the Serving Pods; or the Serving Pods can actively report load metric data to the endpoint discovery server. This application does not limit the detection method of the endpoint discovery server.

[0131] The following description uses the target load balancer in a load balancing system as an example to illustrate the interaction between the endpoint discovery server and the target load balancer. The endpoint discovery server can be used to determine multiple active nodes (which can be denoted as Endpoints) corresponding to the target load balancer, and the first performance metric parameters corresponding to each active node. These multiple active nodes may include the aforementioned multiple compute nodes. The endpoint discovery server can also be used to transmit the multiple active nodes (including multiple compute nodes) and the first performance metric parameters of each active node to the target load balancer via a remote procedure call channel between the target load balancer and the endpoint discovery server.

[0132] Endpoint discovery servers can detect cluster change events in Kubernetes clusters located within the same geographical region as the target load balancer. Based on the event type of the cluster change event, they identify multiple active nodes (including multiple compute nodes) corresponding to the target load balancer. They then acquire load metric data for each of these active nodes and determine the primary performance metric parameters for each node based on this data.

[0133] In one or more embodiments, the calculation method for the first performance index parameter of each active node may include: obtaining load index data corresponding to multiple active nodes (including multiple computing nodes), determining computing nodes with empty load index data as first candidate nodes, and determining computing nodes with non-empty load index data as second candidate nodes; determining the initial performance index parameter of the first candidate node, and determining the initial performance index parameter as the first performance index parameter of the first candidate node; obtaining the average response time and request execution success rate of the second candidate node from the load index data of the second candidate node, and determining the first performance index parameter of the second candidate node based on the average response time and request execution success rate.

[0134] In this model, the first candidate node can represent all newly launched Serving Pods in the Kubernetes cluster corresponding to the target load balancer. The auxiliary containers within these newly launched Serving Pods have not yet reported load metric data to the endpoint discovery server, meaning their load metric data is empty. The second candidate node can represent Serving Pods in the Kubernetes cluster corresponding to the target load balancer that have been running normally for some time. The auxiliary containers within these running Serving Pods can continuously report load metric data to the endpoint discovery server, and their load metric data is not empty at this stage. The first performance metric parameter of the first candidate node can use the initial performance metric parameter, while the first performance metric parameter of the second candidate node is a dynamically weighted parameter calculated based on the load metric data. Because the load metric data is constantly being updated and reported, the first performance metric parameters of each active node also change dynamically.

[0135] In one or more embodiments, the calculation method for initial performance metric parameters may include, but is not limited to: when the second candidate node is empty, it indicates that all Serving Pods in the K8s cluster within the same geographical region as the target load balancer have just been started and have no load metric data yet, making it impossible to calculate the dynamic weight of the first candidate node, such as when the Endpoints_weight array of the target load balancer is empty; then, the initial performance metric parameters of the first candidate node can be determined based on the number of computing resources and memory resources of the first candidate node. When the second candidate node is not empty, it indicates that some Serving Pods have been continuously reporting load metric data, and the dynamic weight of the second candidate node can be calculated, such as when the Endpoints_weight array of the target load balancer has data; then, based on the first performance metric parameters of the second candidate node, a third candidate node is determined from the second candidate nodes, the average value of the first performance metric parameters of the third candidate node is determined as the mean value of the metric parameters, and this mean value of the metric parameters is determined as the initial performance metric parameters of the first candidate node.

[0136] The third candidate node can refer to some or all of the second candidate nodes selected from those whose load metric data is not empty, where the number of second candidate nodes is greater than or equal to the number of third candidate nodes. For example, the Serving Pods corresponding to the top 20% of the first performance metric parameters (which are dynamic weights calculated based on the load metric data) in the Endpoints_weight array can be used as third candidate nodes. The average value of the first performance metric parameters of the third candidate nodes (which can be called the mean of the first performance metric parameters) is calculated, and this average value can be used as the initial performance metric parameters of the first candidate nodes. The calculation method of the first performance metric parameters of each active node can be found in the relevant description in step S103, and will not be repeated here.

[0137] In this embodiment of the application, the dynamics involved in the target load balancer in the load balancing system can include two dynamics: one is the dynamic change of the Endpoint, and the other is the dynamic change of the first performance metric parameter of the Endpoint. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart illustrating the update process of a first performance metric parameter of a network endpoint, as provided in an embodiment of this application. Figure 6 As shown, the update process for the first performance metric parameter of a network endpoint may include the following steps S201 to S215:

[0138] S201, Endpoint discovery server continuously monitors cluster change events.

[0139] S202, determine whether the event type of the cluster change event is "increase".

[0140] S203, determine whether the event type of the cluster change event is deletion.

[0141] S204, Delete the network endpoint (which can be called an active node, denoted as Endpoint) corresponding to the container collection Serving Pod.

[0142] S205, the event type of cluster change events is any type other than addition and deletion.

[0143] S206, Update the first performance metric parameter of the Endpoint corresponding to the Serving Pod.

[0144] S207, calculate the initial performance metrics parameters for the newly added Serving Pod.

[0145] S208 adds the newly added Serving Pod as an Endpoint to the active node array.

[0146] In this embodiment of the application, a set of K8s cluster detectors in the endpoint discovery server detect cluster change events of the workload Serving Pod and process the network endpoint and the corresponding first performance metric parameters according to the cluster change events.

[0147] For example, when the endpoint discovers that the server has detected a cluster change event, the event type of the cluster change event can be determined. If the event type is "addition", it means that a new container set Serving Pod has been added to the K8s cluster. Then, the initial performance index parameters of the newly added Serving Pod can be calculated, and the newly added Serving Pod can be added as an Endpoint to the active node array. The calculation method of the initial performance index parameters can be found in the relevant description in step S103, which will not be repeated here.

[0148] The active node array is similar to the aforementioned network endpoint list; both represent collections of all endpoints obtained by the endpoint discovery server for the target load balancer. The difference lies in their data structures: the active node array stores endpoints as an array, while the endpoint list stores endpoints as a list. In practical applications, different data structures can be used to store endpoints, and this application does not impose any restrictions on this.

[0149] If the event type is not "add", then it can be determined whether the event type is "delete". If the event type is "delete", it means that one or more Serving Pods have been deleted from the Kubernetes cluster, and the Endpoints corresponding to these deleted Serving Pods can then be deleted. If the event type is another type besides "add" or "delete", such as "modify" or "start", then the first performance metric parameters of the existing Endpoints can be updated.

[0150] S209 triggers the target load balancer to pull the dynamically updated array of active nodes.

[0151] S210, determine whether the scheduled fetch time has been reached.

[0152] S211, retrieve the first performance metric parameter of the Endpoint from the updated active node array.

[0153] S212, determine whether the retrieval was successful.

[0154] S213, Delete the Endpoint that failed to be fetched.

[0155] S214, Update the first performance metric parameter of the successfully pulled Endpoint.

[0156] S215, determine whether a stop command has been received.

[0157] In this embodiment, the weight controller in the endpoint discovery server can be a scheduled task. This scheduled task periodically scans the endpoints in the active node array and calculates dynamically changing first performance metric parameters based on load metric data reported by auxiliary containers in the workload Serving Pod, and updates the corresponding endpoints. When the Kubernetes cluster detector and weight controller (scheduled task) are working effectively, an update can be triggered. The endpoint discovery server can then transmit the updated endpoints and their corresponding first performance metric parameters to the target load balancer until it receives a stop command, at which point it stops transmitting the endpoints and their corresponding first performance metric parameters to the target load balancer.

[0158] For example, once the Kubernetes cluster detector is working effectively, it can trigger the target load balancer to pull the dynamically updated array of active nodes and determine if the scheduled pull time has arrived, such as checking if the weight controller (scheduled task) has reached its scheduled task trigger time. If the scheduled pull time has arrived, the target load balancer can pull the first performance metric parameter of the Endpoint in the updated array of active nodes. If the pull fails, the failed Endpoint can be deleted from the compute node list; if the pull succeeds, the first performance metric parameter of the successfully pulled Endpoint can be updated in the compute node list until a stop command is received.

[0159] In this embodiment, a set of K8s cluster detectors and weight controllers in the endpoint discovery server can dynamically adjust the first performance metric parameters of each Serving Pod. The dynamically adjusted Endpoint (the Serving Pod that is in normal working condition) is transmitted to the corresponding load balancer in real time. This allows the load balancer to allocate the currently received business data processing requests to the Serving Pod with the lightest load based on the real-time performance status of the Serving Pod. This not only optimizes the utilization efficiency of Serving Pod resources but also improves the adaptability of the load balancing system to changes in the workload of Serving Pods, reduces the response time of business data processing requests, and enhances the user experience.

[0160] Please see Figure 7 , Figure 7 This is a schematic diagram of the workflow for a business data processing request provided in an embodiment of this application. Figure 1 ;like Figure 7 As shown, the workflow for processing business data requests may include the following steps S301 to S309:

[0161] S301, client-side pre-calculation completed.

[0162] Specifically, in practical applications, the client can perform some pre-processing of data locally. After completing the pre-computation, the client accesses the domain name server by generating a unique domain name. For example, in the data annotation scenario of a network model, the client can initialize the data to be annotated (text, images, videos, audio, etc.), such as initially classifying the data into long and short texts, e-books, film scripts, news texts, etc.; or it can process the data in batches, such as dividing the data into batches according to pre-defined rules and annotating each batch independently; or it can represent the data as vectors, such as converting the data into embedded feature vectors. The process of the client initializing the data can be called the pre-computation process.

[0163] S302, the domain name server accesses the unique domain name generated by the client to determine the target load balancer.

[0164] Specifically, the domain name server can access the unique domain name generated by the client. By analyzing this domain name, the client's geographical location information (referred to as secondary geographical location information) can be determined. Using the geographical location information of each load balancer stored on the domain name server (referred to as primary geographical location information) and the client's secondary geographical location information, the target load balancer with the closest geographical location to the client can be selected from among multiple load balancers.

[0165] Understandably, the primary geographical location and network address (referred to as the primary network address) of each load balancer can be registered with the domain name server through a load balancer detector. After the domain name server determines the target load balancer for the client, it can automatically return the primary network address of the target load balancer as the secondary network address to the client.

[0166] S303, the client sends a business data processing request to the target load balancer.

[0167] S304, the target load balancer calculates the second performance metric parameter for multiple compute nodes.

[0168] S305 selects the target computing node to be distributed based on the second performance index parameter.

[0169] S306, determine whether the target computing node is busy.

[0170] S307, Business data processing requests enter the request waiting queue.

[0171] S308, the target computing node executes the business data processing request and obtains the business data processing result of the request.

[0172] S309 returns the business data processing results layer by layer upwards.

[0173] In this embodiment, after receiving the second network address returned by the domain name server, the client can initiate a business data processing request to the target load balancer. After receiving the business data processing request, the target load balancer can randomly select multiple computing nodes (which are essentially Serving Pods that are maintaining normal working status) from the list of computing nodes corresponding to the target load balancer. The second performance index parameters corresponding to the multiple computing nodes can be calculated according to the method shown in formula (1), and the computing node with the largest second performance index parameter is determined as the target computing node.

[0174] The decision to send the data processing request directly to the target compute node can be based on its current load status. For example, the number of active requests to the target compute node can be used to determine if it is busy. If the target compute node is not busy and is available, the data processing request can be sent directly to it. After the target compute node completes the data processing request, it receives the result and returns it upwards layer by layer. This upward layering can refer to the target compute node returning the result to the target load balancer, which then returns the result to the client.

[0175] Optionally, if the target compute node is busy, the business data processing request can enter the request waiting queue of the target load balancer until there is an available compute node in the compute node list, and then the business data processing request in the request waiting queue is sent to the available compute node for processing.

[0176] Optionally, after the target load balancer selects a target compute node, if the target compute node is in a busy state, the business data processing request can be sent to the target compute node and enter the request waiting queue of the target compute node, waiting for the target compute node to release idle resources, and then the target compute node that releases idle resources can process the business data processing request.

[0177] In other words, the target load balancer and each Serving Pod can each maintain a request waiting queue, such as the target load balancer and the target compute node each maintaining their own request waiting queue. When the target compute node is busy, the target load balancer can handle the currently received business data processing requests in two ways: One is to not forward the business data processing request to the target compute node, but temporarily add the request to the target load balancer's request waiting queue, and then reselect a compute node (an available compute node) to execute the request. The other is to forward the business data processing request to the target compute node, which then adds the request to its own request waiting queue until it releases idle resources before executing the request. This application embodiment does not limit the processing method for business data processing requests.

[0178] Please see Figure 8 , Figure 8 This is a schematic diagram of the workflow for a business data processing request provided in an embodiment of this application. Figure 2 ;like Figure 8 As shown, the workflow for processing business data requests may include the following steps S401 to S410:

[0179] S401, client-side pre-calculation completed.

[0180] S402, the domain name server accesses the unique domain name generated by the client to determine the target load balancer.

[0181] S403, the client sends a business data processing request to the target load balancer.

[0182] S404, traverse the list of active nodes in the compute node list (which can be called network endpoints, denoted as Endpoint).

[0183] S405, determine whether all active nodes in the compute node list are busy.

[0184] S406, Business data processing requests enter the request waiting queue.

[0185] S407, the target load balancer calculates the second performance metric parameters for multiple compute nodes.

[0186] S408 selects the target computing node to be distributed based on the second performance index parameter.

[0187] S409, the target computing node executes the business data processing request and obtains the business data processing result of the request.

[0188] S410 returns the business data processing results layer by layer upwards.

[0189] In this embodiment of the application, the specific implementation process of steps S401 to S403 can be found in [reference needed]. Figure 7 The descriptions of steps S301 to S303 in the corresponding embodiments will not be repeated here.

[0190] After receiving a business data processing request, the target load balancer can iterate through each active node (Endpoint) in its compute node list to determine if the Endpoint is busy. If all Endpoints in the compute node list are busy, the business data processing request can be directly added to the request waiting queue maintained by the target load balancer, waiting for an available Endpoint in the compute node list to execute the request. For example, when Endpoint1 in the compute node list changes from busy to available, the business data processing request in the request waiting queue can be sent to the Serving Pod corresponding to Endpoint1 for execution, obtaining the business data processing result. In this scenario, the target load balancer does not need to calculate the second performance metric parameter; it only needs to check the available Endpoints in the compute node list, which improves the rational use of resources by the target load balancer.

[0191] If some Endpoints in the compute node list are available, multiple compute nodes can be randomly selected from all available Endpoints. The target compute node can be determined using the method described above for selecting Endpoints based on the second performance metric parameter. Since the busy status has already been determined, the target compute node is now available. The target load balancer can forward business data processing requests to the target compute node for direct processing, obtain the business data processing result, and return the result upwards layer by layer.

[0192] In one or more embodiments, the load balancing system can be applied to the training data annotation scenario of large models. Because the raw data input sources are complex and lack structured labels during the creation of large model training datasets, a significant amount of manpower and resources are required for processing. The load balancing system involved in this application embodiment can be integrated into a federated Kubernetes cluster to deploy online inference services for cross-regional raw data and annotate large-scale cross-regional data. When the load balancing system is applied to the annotation load of large model dataset creation, it can efficiently utilize various idle resources in the deployed Kubernetes cluster, accelerating the iterative process of creating large model training datasets. In an experiment deploying a Caption model (an image description generation model) to perform text descriptions on image and video data, under the same throughput requirements, the load balancing system provided in this application embodiment can reduce the number of CPU cores from 1.73 million to 580,000.

[0193] Optionally, this load balancing system can be widely applied to application scenarios requiring high concurrency and low latency, such as cloud computing services, content delivery networks (CDNs), online games, large e-commerce platforms, and real-time communication systems. Through deep integration with cloud platforms, container orchestration systems (such as Kubernetes), service discovery mechanisms, and network infrastructure, it can achieve elastic scaling of resources, automatic failover, and high availability of services. Furthermore, this load balancing system can be integrated into microservice architectures, edge computing, and IoT systems to build agile, efficient, and scalable distributed application systems.

[0194] In this embodiment, the endpoint discovery server can dynamically update the Serving Pods that are maintaining normal operation and their corresponding first performance metric parameters based on the load metric data reported by the auxiliary containers in each Serving Pod, and periodically transmit this data to the corresponding load balancer, such as the target load balancer. The target load balancer can determine the real-time performance status of each compute node by using the current first performance metric parameters and the current number of active requests for each compute node, and allocate business data processing requests to the target compute node with the lightest load, thereby improving the resource utilization efficiency of the compute nodes.

[0195] It is understood that the specific implementation of this application may involve the geographical location information of the client used by the user, the configuration information of the user in the client, domain name access records, etc. When the above embodiments of this application are applied to specific products or technologies, permission or consent from relevant institutions or departments or the user himself is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions.

[0196] Please see Figure 9 , Figure 9 This is a schematic diagram of a load balancing device provided in an embodiment of this application. It can be understood that this load balancing device is applied in a target load balancer, and the target load balancer and the client are located in the same geographical area. Figure 9 As shown, the load balancing device 1 may include:

[0197] The request receiving module 91 is used to respond to the business data processing request sent by the client, obtain the list of computing nodes stored in the target load balancer, the list of computing nodes includes multiple computing nodes and the first performance index parameters corresponding to the multiple computing nodes, and the multiple computing nodes are located in the same geographical area as the target load balancer.

[0198] The node selection module 92 is used to determine the second performance index parameter of each computing node based on the first performance index parameter of each computing node and the number of active requests of each computing node.

[0199] The node selection module 92 is also used to determine the target computing node among multiple computing nodes based on the second performance index parameter of each computing node, and forward the business data processing request to the target computing node so that the target computing node can execute the business data processing request.

[0200] The result return module 93 is used to receive the business data processing result of the target computing node in response to the business data processing request, and return the business data processing result to the client.

[0201] In one or more embodiments, the load balancing device 1 further includes: a weight transmission module 94;

[0202] Weight transmission module 94 is used to establish a remote procedure call channel between the target load balancer and the endpoint discovery server;

[0203] The weight transmission module 94 is also used to obtain multiple network endpoint addresses sent by the endpoint discovery server and the first performance index parameters of the multiple network endpoint addresses through the remote procedure call channel. The multiple network endpoint addresses and the first performance index parameters of the multiple network endpoint addresses are transmitted from the endpoint discovery server to the target load balancer in a streaming data transmission mode. One network endpoint address refers to one computing node.

[0204] The weight transmission module 94 is also used to create an association for each network endpoint address and the first performance index parameter of each network endpoint address, generate a list of computing nodes for the target load balancer, and store the list of computing nodes.

[0205] In one or more embodiments, when the node selection module 92 determines the second performance indicator parameter of each computing node based on the first performance indicator parameter of each computing node and the number of active requests for each computing node, it specifically performs the following steps:

[0206] Multiple compute nodes are identified in the compute node list, and the request deviation parameter is obtained. Based on the request deviation parameter and the number of active requests for each compute node, the request index value of each compute node is determined. The request deviation parameter is used to balance the importance of the primary performance indicator parameter and the number of active requests.

[0207] The second performance metric parameter for each compute node is determined based on the first performance metric parameter for each compute node and the request index value for each compute node.

[0208] In one or more embodiments, the list of computing nodes includes multiple active nodes, and the multiple active nodes include multiple computing nodes;

[0209] When the node selection module 92 determines multiple computing nodes in the computing node list, it specifically performs the following steps:

[0210] Get the number of active requests for each active node in the compute node list;

[0211] Iterate through each active node in the list of computing nodes and add active nodes with fewer than the number of active requests to the node selection set.

[0212] When the number of active nodes in the node selection set is greater than or equal to the node number threshold, multiple computing nodes are selected from the node selection set; the number of multiple computing nodes is equal to the node number threshold.

[0213] In one or more embodiments, when the node selection module 92 determines a target computing node from multiple computing nodes based on the second performance index parameter of each computing node, and forwards the business data processing request to the target computing node, it specifically performs the following steps:

[0214] Among multiple computing nodes, the computing node corresponding to the largest second performance index parameter is determined as the candidate computing node;

[0215] If the candidate compute node is busy, the business data processing request will be added to the request waiting queue of the target load balancer.

[0216] When an available compute node is detected in the compute node list, the available compute node is identified as the target compute node, and the business data processing requests in the request waiting queue are forwarded to the target compute node.

[0217] In one or more embodiments, the load balancing device 1 further includes: a node status determination module 95;

[0218] The node status determination module 95 is used to determine that the candidate computing node is in a busy state if the number of active requests of the candidate computing node is greater than or equal to the request number threshold.

[0219] The node status determination module 95 is also used to determine that the candidate computing node is in an available state if the number of active requests of the candidate computing node is less than the request number threshold.

[0220] In one or more embodiments, the load balancing device 1 further includes: a health check module 96;

[0221] The health check module 96 is used to send test requests to multiple computing nodes in the computing node list at preset intervals, including computing node a.

[0222] The health check module 96 is also used to determine that computing node a is in a fault state if the target load balancer does not receive the test response result of computing node a within the first preset response time, and to delete computing node a in a fault state from the computing node list.

[0223] In one or more embodiments, the load balancing device 1 further includes: a timeout processing module 97;

[0224] The timeout processing module 97 is used to, if the target load balancer does not receive the business data processing result of the target computing node for the business data processing request within the second preset response time, change the target computing node for the business data processing request in the computing node list, determine the changed target computing node, and forward the business data processing request to the changed target computing node.

[0225] The timeout processing module 97 is also used to determine that the business data processing request has failed if the target load balancer does not receive the business data processing result of the business data processing request within the third preset response time. The module returns a failure message to the client, and the third preset response time is longer than the second preset response time.

[0226] According to one embodiment of this application, the foregoing Figure 2 The load balancing method shown can be described by the relevant steps involved. Figure 9 The load balancing device 1 shown is used by various modules to perform this task. For example, Figure 2 The step S102 shown can be performed by Figure 9 The request receiving module 91 shown is used to execute this. Figure 2 Steps S103-S105 shown can be derived from... Figure 9The node selection module 92 shown is used for execution. Figure 2 Steps S107 and S108 shown can be derived from... Figure 9 The results shown are returned to module 93 for execution, etc.

[0227] According to one embodiment of this application, Figure 9 The various modules in the load balancing device 1 shown can be individually or entirely combined into one or more modules, or some of the modules can be further divided into at least two functionally smaller units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by at least two units, or the function of at least two modules can be implemented by one module. In other embodiments of this application, the load balancing device 1 may also include other modules or units. In practical applications, these functions can also be implemented with the assistance of other modules, and can be implemented collaboratively by at least two modules. The beneficial effects of using the same method will not be repeated here.

[0228] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 10 As shown, the computer device 1000 can be a terminal device or a server; this application does not limit this. For ease of understanding, the following description uses a server as an example. The computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 10 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0229] The network interface 1004 in the computer device 1000 can also provide network communication functionality. Figure 10In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0230] In response to the business data processing request sent by the client, obtain the list of computing nodes stored in the target load balancer. The list of computing nodes includes multiple computing nodes and the first performance index parameters corresponding to each computing node. The multiple computing nodes are located in the same geographical area as the target load balancer. The first performance index parameter of each computing node is used to indicate the data processing performance level of a computing node.

[0231] The second performance metric parameter for each compute node is determined based on the first performance metric parameter for each compute node and the number of active requests for each compute node.

[0232] Based on the second performance index parameter of each computing node, the target computing node is determined among multiple computing nodes, and the business data processing request is forwarded to the target computing node so that the target computing node can execute the business data processing request.

[0233] Receive the business data processing results from the target computing node in response to the business data processing request, and return the business data processing results to the client.

[0234] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 2 , Figures 6 to 8 The description of the load balancing method in any embodiment can also be performed as described above. Figure 9 The description of the load balancing device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0235] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the load balancing device 1 mentioned above. The computer program includes computer instructions, and when the processor executes the computer instructions, it can execute the aforementioned... Figure 2 , Figures 6 to 8The description of the load balancing method in any of the embodiments is omitted here. Similarly, the beneficial effects of using the same method are also omitted here. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed and executed on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0236] Furthermore, it should be noted that this application also provides a computer program product, which may include a computer program that can be stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, causing the computer device to perform the aforementioned... Figure 2 , Figures 6 to 8 The description of the load balancing method in any of the embodiments is redundant and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0237] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different media content, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0238] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0239] The methods and related apparatuses provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0240] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0241] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A load balancing method, characterized in that, The method is executed by a target load balancer located in the same geographical region as the client. The method includes: In response to the business data processing request sent by the client, a list of computing nodes stored in the target load balancer is obtained. The list of computing nodes includes multiple computing nodes and a first performance index parameter corresponding to each of the multiple computing nodes. The multiple computing nodes and the target load balancer are located in the geographical area. The first performance index parameter of each computing node is used to indicate the data processing performance of a computing node. Based on the first performance metric parameter of each computing node and the number of active requests of each computing node, the second performance metric parameter of each computing node is determined. Based on the second performance index parameter of each computing node, a target computing node is determined among the plurality of computing nodes, and the business data processing request is forwarded to the target computing node so that the target computing node executes the business data processing request; The system receives the business data processing result from the target computing node in response to the business data processing request, and returns the business data processing result to the client.

2. The method according to claim 1, characterized in that, The method further includes: Establish a remote procedure call channel between the target load balancer and the endpoint discovery server; Through the remote procedure call channel, multiple network endpoint addresses sent by the endpoint discovery server and the first performance index parameters of the multiple network endpoint addresses are obtained. The multiple network endpoint addresses and the first performance index parameters of the multiple network endpoint addresses are transmitted from the endpoint discovery server to the target load balancer in a streaming data transmission mode. One network endpoint address refers to one computing node. Create an association relationship for each network endpoint address and a first performance metric parameter of each network endpoint address, generate a list of compute nodes for the target load balancer, and store the list of compute nodes.

3. The method according to any one of claims 1 to 2, characterized in that, The step of determining the second performance indicator parameter of each computing node based on the first performance indicator parameter of each computing node and the number of active requests of each computing node includes: The plurality of computing nodes are determined in the list of computing nodes, and the request deviation parameter is obtained. Based on the request deviation parameter and the number of active requests of each computing node, the request index value of each computing node is determined. The request deviation parameter is used to balance the importance of the first performance index parameter and the number of active requests. The second performance index parameter of each computing node is determined based on the first performance index parameter of each computing node and the request index value of each computing node.

4. The method according to claim 3, characterized in that, The list of computing nodes includes multiple active nodes, and the multiple active nodes include the multiple computing nodes; Determining the plurality of computing nodes in the list of computing nodes includes: Get the number of active requests for each active node in the list of computing nodes; Iterate through each active node in the list of computing nodes and add active nodes whose number of active requests is less than the request number threshold to the node selection set; When the number of active nodes in the node selection set is greater than or equal to the node number threshold, the plurality of computing nodes are determined in the node selection set, and the number of the plurality of computing nodes is equal to the node number threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining a target computing node among the plurality of computing nodes based on the second performance index parameters of each computing node, and forwarding the business data processing request to the target computing node, includes: Among the plurality of computing nodes, the computing node corresponding to the largest second performance index parameter is determined as a candidate computing node; If the candidate computing node is busy, the business data processing request is added to the request waiting queue of the target load balancer. When a computing node in an available state is detected in the computing node list, the computing node in an available state is determined as the target computing node, and the business data processing request in the request waiting queue is forwarded to the target computing node.

6. The method according to claim 5, characterized in that, The method further includes: If the number of active requests to the candidate computing node is greater than or equal to the request number threshold, then the candidate computing node is determined to be in the busy state. If the number of active requests to the candidate computing node is less than the request number threshold, then the candidate computing node is determined to be in the available state.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: According to a preset interval, test requests are sent to the plurality of computing nodes in the computing node list, including computing node a; If the target load balancer does not receive the test response result of computing node a within the first preset response time, it determines that computing node a is in a fault state and deletes computing node a in a fault state from the computing node list.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If the target load balancer does not receive the business data processing result from the target computing node for the business data processing request within the second preset response time, then in the computing node list, the target computing node for the business data processing request is replaced, the replaced target computing node is determined, and the business data processing request is forwarded to the replaced target computing node. If the target load balancer does not receive the business data processing result of the business data processing request within the third preset response time, it determines that the business data processing request has failed and returns the execution failure message of the business data processing request to the client. The third preset response time is longer than the second preset response time.

9. A load balancing system, characterized in that, The load balancing system includes a client, a target load balancer, and multiple computing nodes, wherein the target load balancer, the client, and the multiple computing nodes are located in the same geographical area. The client is used to send a business data processing request to the target load balancer; The target load balancer is used to respond to the business data processing request and obtain a list of computing nodes stored in the target load balancer. The list of computing nodes includes the plurality of computing nodes and a first performance index parameter corresponding to each of the plurality of computing nodes. The first performance index parameter of each computing node is used to indicate the data processing performance level of a computing node. The target load balancer is further configured to determine the second performance index parameter of each computing node based on the first performance index parameter of each computing node and the number of active requests of each computing node. The target load balancer is further configured to determine the target computing node among the plurality of computing nodes based on the second performance index parameter of each computing node; The target computing node is used to execute the business data processing request; The target load balancer is also used to receive the business data processing result of the target computing node in response to the business data processing request, and return the business data processing result to the client. The client is used to receive the results of the business data processing.

10. The system according to claim 9, characterized in that, The load balancing system also includes a load balancing detector, a domain name server, and multiple load balancers, including the target load balancer. The load balancing detector is used to detect the first network address of each load balancer and the first geographical location information of each load balancer, and transmit the first network address and the first geographical location information to the domain name server. The domain name server is used to resolve the domain name of the client and determine the client's second geographical location information; The domain name server is further configured to determine, based on the first geographical location information and the first network address, a second network address that is located within the same geographical area as the second geographical location information from among the first network addresses of the plurality of load balancers, and return the second network address to the client, wherein the second network address is used to indicate the target load balancer.

11. The system according to any one of claims 9 to 10, characterized in that, The load balancing system also includes an endpoint discovery server; The endpoint discovery server is used to determine multiple compute nodes of the target load balancer, and a first performance metric parameter for each compute node; The endpoint discovery server is also used to transmit the plurality of compute nodes, and the first performance index parameters of each compute node, to the target load balancer through a remote procedure call channel between the target load balancer and the endpoint discovery server.

12. The system according to any one of claims 9 to 11, characterized in that, When the endpoint discovery server determines the multiple compute nodes of the target load balancer and the first performance metric parameter of each compute node, it specifically performs the following steps: Detect cluster change events, and determine the multiple computing nodes of the target load balancer based on the event type of the cluster change events; Obtain the load index data of each computing node, and determine the first performance index parameter of each computing node based on the load index data of each computing node.

13. The system according to any one of claims 9 to 12, characterized in that, When the endpoint discovery server determines the first performance metric parameter of each computing node based on the load metric data of each computing node, it specifically performs the following steps: The computing nodes with empty load index data are determined as the first candidate nodes, and the computing nodes with non-empty load index data are determined as the second candidate nodes. Determine the initial performance index parameters of the first candidate node, and set the initial performance index parameters as the first performance index parameters of the first candidate node; From the load metric data of the second candidate node, obtain the average response time and request execution success rate of the second candidate node, and determine the first performance metric parameter of the second candidate node based on the average response time and the request execution success rate.

14. The system according to claim 13, characterized in that, When the endpoint discovery server determines the initialization performance metric parameters of the first candidate node, it specifically performs the following steps: When the second candidate node is empty, the initial performance index parameters of the first candidate node are determined based on the number of computing resources and memory resources of the first candidate node. When the second candidate node is not empty, a third candidate node is determined from the second candidate nodes according to the first performance index parameter of the second candidate node, the average value of the first performance index parameter of the third candidate node is determined as the index parameter mean, and the index parameter mean is determined as the initial performance index parameter of the first candidate node.

15. A load balancing device, characterized in that, The device is used in a target load balancer, where the target load balancer and the client are located in the same geographical area. The device includes: The request receiving module is used to respond to the business data processing request sent by the client, obtain the list of computing nodes stored in the target load balancer, the list of computing nodes includes multiple computing nodes and first performance index parameters corresponding to the multiple computing nodes respectively, the multiple computing nodes and the target load balancer are located in the geographical area, and the first performance index parameter of each computing node is used to indicate the data processing performance level of a computing node. The node selection module is used to determine the second performance index parameter of each computing node based on the first performance index parameter of each computing node and the number of active requests of each computing node. The node selection module is further configured to determine a target computing node among the plurality of computing nodes based on the second performance index parameter of each computing node, and forward the business data processing request to the target computing node so that the target computing node executes the business data processing request; The result return module is used to receive the business data processing result of the target computing node in response to the business data processing request, and return the business data processing result to the client.

16. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1 to 8.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1 to 8.

18. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 8.

Images