Load balancing systems and load balancing methods

Abstract

This invention provides a load balancing system and a load balancing method. The system includes at least one backend server and a load balancer. The backend server includes at least two processors and at least two network interface cards (NICs), with each processor communicating with at least two NICs via a system bus. The load balancer, communicating with the backend server, responds to a first request sent by a client, determines a first backend server and a first processor of the first backend server, and forwards the first request to the first NIC corresponding to the first processor. If the first backend server determines that the processing time of the first request by the first processor exceeds a preset time, it forwards a second request from the first request to a second processor via the first NIC. The second processor is a processor within the first backend server and is connected to the first NIC. The load balancing system provided by this invention achieves both hardware-level and software-level load balancing.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a load balancing system and a load balancing method. Background Technology

[0002] With the increasing complexity of network applications and the rapid growth of data volume, server processing power and computing intensity have also increased accordingly, making it increasingly difficult for a single server device to handle the load. Furthermore, to improve server network throughput and data processing efficiency, modern servers generally use high-performance Network Interface Cards (NICs) as the network access interface. However, in current technical architectures, servers are often configured with only a single NIC to handle all network traffic.

[0003] While a single network interface card (NIC) can provide a certain amount of network bandwidth, in high-concurrency scenarios, all network traffic entering and leaving the server needs to pass through this NIC, which can easily lead to bandwidth shortages and thus affect overall network performance. Summary of the Invention

[0004] This invention provides a load balancing system and load balancing method to address the shortcomings of existing technologies where a single server device is insufficient to handle high-concurrency scenarios, and where a single network card on the server leads to bandwidth constraints, thus affecting overall network performance. The invention achieves load balancing at both the hardware and software levels to ensure reliability and stability in high-concurrency scenarios.

[0005] The present invention provides a load balancing system, comprising: a load balancer and at least one back-end server communicatively connected to the load balancer, wherein the back-end server includes at least two processors and at least two network interface cards (NICs), and each of the processors is connected to the at least two NICs via a system bus.

[0006] The load balancer is used to respond to a first request sent by the client, determine a first backend server and a first processor of the first backend server, and forward the first request to the first network interface card corresponding to the first processor.

[0007] The first backend server is further configured to, when it is determined that the processing time of the first processor in processing the first request exceeds a preset time, forward the second request in the first request to the second processor through the first network card, wherein the second processor is the processor in the first backend server and is connected to the first network card.

[0008] According to a load balancing system provided by the present invention, the load balancer is further used for:

[0009] In response to a first request sent by the client, the request type corresponding to the first request is determined, and candidate backend servers matching the request type are selected from at least one of the backend servers based on the service type of at least one of the backend servers.

[0010] Based on a pre-configured load balancing strategy, a first backend server and a first processor of the first backend server are determined from the candidate backend servers.

[0011] This invention provides a load balancing system, wherein the load balancer is further used for:

[0012] Monitor the status of the first backend server and the first network interface card;

[0013] If it is determined that the first network interface card (NIC) is in an unavailable state, the first request is forwarded to the second NIC, which includes other NICs in the first backend server besides the first NIC, or NICs of other backend servers besides the first backend server.

[0014] This invention provides a load balancing system, further comprising: a system service processor, wherein the system service processor is communicatively connected to the load balancer and at least one of the back-end servers;

[0015] The system service processor is used for:

[0016] Acquire monitoring data of the load balancing system, the monitoring data including the first performance and first status of the load balancer and the second performance and second status of each processor in each of the backend servers;

[0017] Based on the monitoring data, the load balancing system is adjusted, including at least one of adjusting the load balancing strategy and adjusting the number of backend servers.

[0018] This invention provides a load balancing system, wherein the system service processor is further configured to:

[0019] Configure the virtual IP address of the load balancer so that the client can access it based on the virtual IP address;

[0020] Configure the backend server list of the load balancer, wherein the backend server list includes the IP address, port number and service type of at least one backend server;

[0021] Configure the health check protocol and parameters corresponding to the backend servers in the backend server list;

[0022] Configure the load balancing strategy of the load balancer, wherein the load balancing strategy includes at least one of round-robin mechanism, weight allocation, minimum number of connections, and fastest response time.

[0023] This invention provides a load balancing system, wherein the load balancing strategy includes:

[0024] Determine the backend server pointed to by the first pointer among the candidate backend servers that match the first request sent by the current client, and take the backend server pointed to by the first pointer as the first backend server that matches the first request.

[0025] Determine the processor pointed to by the second pointer in the current first backend server, and use the processor pointed to by the second pointer as the first processor matched by the first request;

[0026] When the first processor receives the first request, the first pointer points to the next backend server in a first polling order, and the second pointer points to the next processor in the first backend server in a second polling order.

[0027] This invention provides a load balancing system, wherein the load balancing strategy includes:

[0028] Determine the backend server with the fewest connections among the candidate backend servers that match the first request sent by the current client, and use the backend server with the fewest connections as the first backend server that matches the first request.

[0029] Determine the processor corresponding to the minimum number of threads in the current first backend server, and use the processor corresponding to the minimum number of threads as the first processor to match the first request.

[0030] This invention provides a load balancing system, wherein the load balancing strategy includes:

[0031] Determine the backend server with the shortest first response time among the candidate backend servers that match the first request sent by the client, and use the backend server with the shortest first response time as the first backend server that matches the first request.

[0032] The processor corresponding to the shortest second response time among the first backend servers is determined, and the processor corresponding to the shortest second response time is used as the first processor to match the first request.

[0033] This invention provides a load balancing system, wherein the first backend server is further configured to:

[0034] Upon receiving the first request, the first processor starts a timer;

[0035] If the timer determines that the processing time of the first request by the first processor has reached a preset duration, the current processing progress of the first request is determined.

[0036] If there are multiple pending tasks in the current first request, the multiple pending tasks are parsed to determine at least one target pending task that can be processed independently among the multiple pending tasks.

[0037] A second request is generated based on at least one of the target tasks to be processed;

[0038] Identify at least one candidate processor in the current first backend server, excluding the first processor, wherein the candidate processor is connected to the first network interface card;

[0039] A second processor is determined from the candidate processors based on the load state of at least one of the candidate processors;

[0040] The second request in the first request is forwarded to the second processor through the first network card.

[0041] The present invention also provides a load balancing method, the method being applied to a load balancing system as described in any of the preceding claims, the method comprising:

[0042] The load balancer responds to the first request sent by the client, determines the first backend server and the first processor of the first backend server, and forwards the first request to the first network interface card corresponding to the first processor.

[0043] If the first backend server determines that the processing time of the first processor in processing the first request exceeds a preset time, the second request in the first request is forwarded to the second processor through the first network card. The second processor is the processor in the first backend server and is connected to the first network card.

[0044] According to one provided load balancing method, it also includes:

[0045] The first backend server responds to the first request sent by the client, determines the request type corresponding to the first request, and filters out candidate backend servers that match the request type from at least one backend server based on the service type of at least one backend server.

[0046] Based on a pre-configured load balancing strategy, a first backend server and a first processor of the first backend server are determined from the candidate backend servers, wherein the load balancing strategy includes at least one of round-robin mechanism, weight allocation, minimum number of connections, and fastest response time.

[0047] According to one provided load balancing method, it also includes:

[0048] The load balancer monitors the status of the first backend server and the first network interface card.

[0049] If it is determined that the first network interface card (NIC) is in an unavailable state, the first request is forwarded to the second NIC, which includes other NICs in the first backend server besides the first NIC, or NICs of other backend servers besides the first backend server.

[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the load balancing methods described above.

[0051] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the load balancing method as described above.

[0052] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the load balancing methods described above.

[0053] The load balancing system and method provided by this invention establish a communication connection between the load balancer and the backend servers. This allows the load balancer to determine which backend server and which processor on that server to forward the client's first request to. This achieves load balancing at the software level. Furthermore, each backend server is equipped with at least two processors and at least two network interface cards (NICs), with each processor connected to at least two NICs via a system bus. Specifically, when the processing time of the first request by the first processor exceeds a preset time, a portion of the second request within the first request can be forwarded to the second processor. This allows requests to be transferred between different processors via NICs, avoiding overload of a single processor and further achieving load balancing at the hardware level, thus enhancing the system's reliability and stability. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is one of the structural schematic diagrams of the load balancing system provided by the present invention.

[0056] Figure 2 This is a schematic diagram of the network card configuration and connection of the backend server provided by the present invention.

[0057] Figure 3 This is the second schematic diagram of the load balancing system provided by the present invention.

[0058] Figure 4 This is a flowchart illustrating the load balancing method provided by the present invention.

[0059] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention.

[0060] Figure label:

[0061] 10: Load balancer; 11: Server pool; 20: Backend server; 21: Processor; 22: Network interface card; 30: System service processor. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0063] It should be noted that the load balancing system and load balancing method provided in this application embodiment can be applied to various high-density server clusters, so that both hardware-level load balancing and software-level load balancing can be achieved based on the load balancing system and load balancing method provided in this application embodiment.

[0064] Figure 1 This is one of the structural schematic diagrams of the load balancing system provided by the present invention; for example... Figure 1 As shown, the system includes a load balancer 10, and the server pool 11 of the load balancer 10 includes at least one backend server 20 that is communicatively connected to the load balancer.

[0065] Here, the main function of the load balancer 10 is to distribute network traffic (such as HTTP requests) entering the system to the backend servers 20 in the server pool 11, so as to optimize resource utilization, improve response speed and fault tolerance.

[0066] Server pool 11 is a collection containing at least one backend server 20 that collectively handles requests forwarded by load balancer 10. The backend servers 20 in server pool 11 can be physical servers, virtual machines, or containers, and they are configured to handle the same type of requests (e.g., web servers may all be running the same web application). They can also be configured to handle different types of requests. For example, requests handling a large number of compute-intensive tasks might be assigned to backend servers with high-performance CPUs and memory, while requests handling a large number of I / O-intensive tasks might be assigned to backend servers with high-speed storage and network interfaces.

[0067] Specifically, each backend server 20 includes at least two processors 21 and at least two network interface cards (NICs) 22, and each processor 21 is connected to at least two NICs 22 via a system bus. The NIC types of the at least two NICs 22 in each backend server 20 can be the same or different, and there is no restriction on this.

[0068] Processor 21 is the core computing unit of backend server 20, responsible for executing instructions, processing data, and running programs, such as CPU (Central Processing Unit) and other chips that perform specific processing tasks (such as GPU, DSP, etc.). These processors 21 contain multiple cores (CPU Cores), each of which can execute instructions independently, thereby achieving parallel processing.

[0069] The system bus is the channel for transmitting data and information between various components within the backend server 20. In this embodiment, each processor 21 is connected to at least two network interface cards (NICs) 22 via the system bus, meaning that the processor 21 can communicate directly with the NICs 22 without going through other intermediate devices. The system bus can be of various types, such as PCI (Peripheral Component Interconnect), PCI-X, PCIe, etc.

[0070] It should be noted that in this embodiment, multiple links are routed from each processor 21 to different network interface cards (NICs) 22, thus achieving multi-path network connectivity. When one link or NIC fails, the system can continue operating through another link, improving network connectivity reliability. Furthermore, traffic allocation on different links can be dynamically adjusted according to application requirements, providing greater flexibility.

[0071] For reference Figure 3 A backend server includes two processors: CPU0 and CPU1, and two network interface cards (NICs): a 100G PCIe NIC and a 100G OCP NIC. Here, an x8 PCIe link can be connected from CPU0 and CPU1 to each of the 100G PCIe NICs, and another x8 PCIe link can be connected from CPU0 and CPU1 to each of the 100G OCP NICs. This configuration allows each CPU to directly access two different NICs.

[0072] Furthermore, since different CPLD versions may support different link configurations and bandwidth allocations, this embodiment, in addition to supporting the aforementioned PCIe link allocation, also requires updating the CPLD firmware on the motherboard to a version that supports this configuration. This ensures that the PCIe links can effectively allocate bandwidth according to actual needs and avoid resource conflicts.

[0073] In addition, the network interface card (NIC) firmware is the software running on the NIC that controls its hardware functions and network communication. Therefore, the NIC firmware also needs to be upgraded to this configuration version to ensure that the NIC can correctly identify and process traffic from multiple CPUs.

[0074] Specifically, the load balancer 10 is used to respond to a first request sent by the client, determine a first backend server and a first processor of the first backend server, and forward the first request to the first network interface card corresponding to the first processor.

[0075] The client (such as a browser, API client, etc.) sends a first request to the load balancer 10. This request may be an HTTP request, a database query, or other types of network request. After receiving the request from the client, the load balancer 10 will decide how to distribute this first request according to the predefined load balancing strategy.

[0076] Specifically, load balancer 10 selects a backend server 20 from server pool 11 to handle the first request. This selected backend server 20 is referred to as the first backend server.

[0077] After selecting the first backend server, the load balancer 10 needs to further determine which processor 21 on that server will actually handle the request. For example, it may select a processor 21 from the first backend server based on the current load, processing capacity, or other configuration priority rules of the processor 21 on the first backend server. This selected processor 21 is referred to as the first processor.

[0078] It should be understood that in this embodiment, each processor 21 is connected to at least two network cards 22 through the system bus. Therefore, in this embodiment, after selecting the first processor, it is also necessary to further select the first network card corresponding to the first processor.

[0079] In one example, the load balancer 10 is also configured with detailed information about each network interface card (NIC) 22 on each backend server 20, including the IP address of each NIC 22, the processor it is connected to, and related load information. Therefore, in this embodiment, the load balancer 10 evaluates the current load of each NIC of the first processor (e.g., network traffic, packet processing rate, error rate, etc.). It also evaluates the current load of other processors connected to each NIC, and finally selects the first NIC by combining the current load of each NIC and the current load of other processors connected to each NIC (e.g., CPU utilization, number of processes and threads, etc.). For example, a NIC with low network traffic and a low error rate, and whose connected processors also have sufficient remaining processing power to handle requests, may be selected as the first NIC.

[0080] Finally, load balancer 10 forwards the request to the first network interface card (NIC) corresponding to the first processor on the first backend server responsible for processing the first request. Specifically, via a network protocol (such as TCP / IP), load balancer 10 sends the data packet of the first request to the IP address and port of the first NIC.

[0081] Specifically, the first backend server is further configured to forward the second request in the first request to the second processor through the first network card when it is determined that the processing time of the first processor in processing the first request exceeds a preset time. The second processor is the processor in the first backend server and is connected to the first network card.

[0082] When the first backend server receives the first request distributed through the load balancer 10 and determines that the first processor should handle the request, the first backend server will monitor the processing, especially the time required for the first processor to process the first request.

[0083] If, during processing, the first backend server finds that the first processor takes longer than a preset time to process the first request (such as a preset time set based on system performance requirements, user experience standards, or historical data), in order to reduce the burden on the first processor and speed up the processing of the first request, the first backend server will forward a part of the first request, namely the second request, to the second processor through the first network card.

[0084] Here, the second request may be a subtask of the first request, a fragment that can be executed independently, or a request that is associated with the first request but can be processed in parallel.

[0085] Here, the second processor is another processor on the first backend server, and it is also connected to the first network interface card (NIC). In other words, the first NIC serves multiple processors simultaneously, providing hardware-level redundancy and load balancing capabilities.

[0086] The system provided in this embodiment establishes a communication connection between the load balancer and the backend servers. This allows the load balancer to determine which backend server and which processor on that backend server to forward the client's first request to. This achieves load balancing at the software level. Furthermore, each backend server is equipped with at least two processors and at least two network interface cards (NICs), with each processor connected to at least two NICs via a system bus. Specifically, when the processing time of the first request by the first processor exceeds a preset time, a portion of the second request within the first request can be forwarded to the second processor. This allows requests to be transferred between different processors via NICs, avoiding overload on a single processor and further achieving load balancing at the hardware level, thus enhancing the system's reliability and stability.

[0087] In some embodiments, the load balancer 10 is further configured to:

[0088] In response to a first request sent by the client, the request type corresponding to the first request is determined, and candidate backend servers matching the request type are selected from at least one backend server 20 based on the service type of at least one backend server 20.

[0089] Based on a pre-configured load balancing strategy, a first backend server and a first processor of the first backend server are determined from the candidate backend servers. The load balancing strategy includes at least one of round-robin mechanism, weight allocation, minimum number of connections, and fastest response time.

[0090] Specifically, load balancer 10 will first determine the request type of the first request. This can be done based on factors such as the URL path, HTTP type (e.g., GET, POST), or specific fields in the request header.

[0091] Once the request type is determined, the load balancer 10 filters candidate backend servers that match the request type based on the service type provided by at least one backend server 20. For example, if the first request is a video streaming request, the load balancer 10 will filter backend servers 20 that are configured with video streaming services.

[0092] After selecting candidate backend servers, load balancer 10 will further apply pre-configured load balancing strategies to determine the primary backend server from among the candidate backend servers. These load balancing strategies may include, but are not limited to, the following:

[0093] Polling mechanism: Requests are distributed to backend servers in turn in a sequential manner.

[0094] Weighting: Assign different weights to each backend server based on its processing capacity or resource status, and then distribute requests according to the weight ratio.

[0095] Least Connections: Select the backend server with the fewest current connections (i.e., the lightest load) to handle the request.

[0096] Fastest response time: Select the backend server with the shortest response time based on the average response time of the backend server over a period of time or real-time performance monitoring data.

[0097] Finally, the load balancer 10 will determine a backend server from the candidate backend servers as the first backend server according to the load balancing strategy, and may further determine the first processor on that backend server.

[0098] The system provided in this embodiment not only ensures that requests are distributed to backend servers capable of handling that type of request, but also optimizes resource utilization through intelligent load balancing strategies, thereby improving the overall performance and reliability of the system.

[0099] In some embodiments, the load balancer 10 is further configured to:

[0100] Monitor the status of the first backend server and the first network interface card;

[0101] If it is determined that the first network interface card (NIC) is in an unavailable state, the first request is forwarded to the second NIC, which includes other NICs in the first backend server besides the first NIC, or NICs of other backend servers besides the first backend server.

[0102] Here, the load balancer 10 continuously monitors the status of the first backend server and its connected first network interface card (NIC), including but not limited to the connection status, data traffic, error rate, packet loss rate, and response time of the first NIC. Using this information, the load balancer 10 can assess the health status of the first NIC.

[0103] During the monitoring process, if the load balancer 10 determines whether the first network card is in an unavailable state, then the unavailable state means that the first network card can no longer receive and process requests from the load balancer 10, and it is also unable to return the corresponding results to the load balancer 10.

[0104] When the first network interface card (NIC) is detected to be unavailable, the load balancer 10 selects a suitable second NIC to continue processing the first request that should have been forwarded through the first NIC. If the first backend server has multiple NICs, the load balancer 10 can select one of the working NICs as the second NIC. If all NICs on the first backend server are unavailable, or to achieve higher availability and load balancing, the load balancer 10 can also select NICs from other backend servers offering the same service type as the second NIC.

[0105] Finally, load balancer 10 forwards the first request, which should have been forwarded through the first network interface card (NIC), to the selected second NIC. This ensures that even if the first NIC fails, the client's first request can still be processed promptly, thus guaranteeing system continuity and user experience.

[0106] The system provided in this embodiment ensures that the load balancer can continue to provide services even if some hardware or network components fail, thereby improving the reliability and failover capability of the entire system.

[0107] In some embodiments, reference Figure 3 As shown, the load balancing system also includes a system service processor 30, which is communicatively connected to the load balancer 10 and at least one of the back-end servers 20.

[0108] The system service processor 30 is used for:

[0109] Acquire monitoring data of the load balancing system, the monitoring data including the first performance and first status of the load balancer 10 and the second performance and second status of each processor 21 in each of the backend servers 20;

[0110] Based on the monitoring data, the load balancing system is adjusted, including at least one of adjusting the load balancing strategy and adjusting the number of backend servers 20.

[0111] In this embodiment, the load balancing system further includes a system service processor 30. Specifically, the system service processor 30 periodically or in real-time collects monitoring data from the load balancer 10 and the backend server 20 in the load balancing system. This data includes, but is not limited to:

[0112] The first performance and first status of load balancer 10: such as the performance indicators of load balancer 10, such as CPU utilization, memory usage, network throughput, number of connections, error rate, etc., as well as its own health status (such as whether it is online, whether there are fault reports, etc.).

[0113] Second performance and second status of each processor 21 in each backend server 20: such as performance metrics (e.g., CPU utilization, memory usage, processing queue length, etc.) and status information (e.g., whether overloaded, whether an error has occurred, etc.) of each processor 21.

[0114] Based on the collected monitoring data, the load balancing monitoring module can dynamically adjust the load balancing strategy. For example, if the processor load of a backend server is too high, the load balancing monitoring will reduce the traffic subsequently allocated to that backend server to distribute requests more efficiently.

[0115] Furthermore, the system service processor 30 can increase or decrease the number of backend servers based on the overall system load. If the system load consistently exceeds the processing capacity of the current backend servers, or if the number of available servers is insufficient due to the failure of some backend servers, the load balancing monitoring module can trigger an automatic scaling mechanism to add new backend servers to the load balancer's server pool. Conversely, if the processing load decreases for an extended period, it will automatically reduce the number of backend servers in the load balancer's server pool to conserve resources. Alternatively, when a backend server failure is detected, the load balancer will be notified so that it can be removed from the service pool.

[0116] The system provided in this embodiment ensures that the load balancing system can self-optimize based on real-time performance data and status information, thereby maintaining high availability, high performance, and efficient resource utilization.

[0117] In some embodiments, the system service processor 30 is further configured to:

[0118] Configure the virtual IP address of the load balancer 10 so that the client can access it based on the virtual IP address;

[0119] Configure the backend server list of the load balancer 10, wherein the backend server list includes the IP address, port number and service type of at least one backend server;

[0120] Configure the health check protocol and parameters corresponding to the backend servers in the backend server list;

[0121] Configure the load balancing strategy of the load balancer, wherein the load balancing strategy includes at least one of round-robin mechanism, weight allocation, minimum number of connections, and fastest response time.

[0122] The system service processor 30 allows administrators to configure one or more virtual IP addresses for the load balancer 10. Clients access services through these virtual IP addresses without needing to know the actual IP addresses of the backend servers 20. The virtual IP addresses provide a unified entry point, enabling the load balancer 10 to transparently distribute client requests to different backend servers 20.

[0123] The system service processor 30 also allows administrators to customize a set of backend servers and assign IP addresses, port numbers, and service types (such as HTTP, HTTPS, FTP, etc.) to each backend server 20. In this way, the load balancer 10 can correctly forward requests to the appropriate backend servers 20 and network interface cards based on this information in the backend server list.

[0124] Furthermore, to ensure that only healthy backend servers 20 can receive requests, the load balancer configuration module allows administrators to configure health check protocols (such as HTTP GET requests, TCP connection tests, etc.) and parameters (such as request path, expected response code, timeout, etc.) for each backend server in the backend server list. Through these configurations, the load balancer 10 can automatically detect the health status of the backend servers 20 and remove them from the backend server list when a failure is detected.

[0125] Here, the load balancing strategy determines how load balancer 10 distributes client requests to backend servers 20. The load balancer configuration module provides administrators with a variety of load balancing strategies to choose from, including but not limited to:

[0126] Polling mechanism: Requests are distributed sequentially to each backend server and its corresponding processor.

[0127] Weighting: Different weights are assigned to backend servers and processors based on their processing capabilities or other factors, and then requests are distributed according to the weight ratio.

[0128] Least Connections: Distribute requests to the backend server with the fewest current connections and the corresponding processor to balance the load on each backend server.

[0129] Fastest response time: Measure the response time of each backend server and processor in real time, and distribute requests to the fastest responding backend server and corresponding processor.

[0130] The system provided in this embodiment, through these configurations, ensures that the load balancer can effectively distribute client requests according to established rules and strategies, while maintaining the health and efficient operation of the backend server cluster.

[0131] In some embodiments, the load balancing strategy includes:

[0132] Determine the backend server pointed to by the first pointer among the candidate backend servers that match the first request sent by the current client, and take the backend server pointed to by the first pointer as the first backend server that matches the first request.

[0133] Determine the processor pointed to by the second pointer in the first backend server and use the processor pointed to by the second pointer as the first processor matched by the first request;

[0134] When the first processor receives the first request, the first pointer points to the next backend server in a first polling order, and the second pointer points to the next processor in the first backend server in a second polling order.

[0135] Here, the first round-robin order refers to the round-robin order of the backend servers. When load balancer 10 receives a request from a client, it selects a backend server 20 to handle the request according to the first round-robin order.

[0136] Specifically, each service type is configured with a list containing all available backend servers. When determining which candidate backend services in this list match the first request sent by the client, the first pointer selects the first backend server according to the round-robin order of this list. Here, the second round-robin order is the round-robin order for the processors 21 within each backend server 20. Once the first backend server is selected, the load balancer 10 selects a first processor on that first backend server to handle the request, according to the second round-robin order.

[0137] Furthermore, once it is determined that the selected first processor can handle the first request, the first pointer will automatically point to the next backend server 20 according to the first polling order, in order to prepare for the next request. At the same time, the second pointer will also point to the next processor 21 among the first backend servers according to the second polling order.

[0138] The system provided in this embodiment uses a round-robin load balancing strategy to distribute requests evenly to different backend servers and processors, thereby avoiding overload of any single server or processor and improving the overall performance and response time of the system.

[0139] In some embodiments, the load balancing strategy includes:

[0140] Determine the backend server with the fewest connections among the candidate backend servers that match the first request sent by the current client, and use the backend server with the fewest connections as the first backend server that matches the first request.

[0141] Determine the processor corresponding to the minimum number of threads in the current first backend server, and use the processor corresponding to the minimum number of threads as the first processor to match the first request.

[0142] Here, the current connection count refers to the number of connections that backend server 20 is currently processing and / or waiting to be processed. For example, load balancer 10 can continuously monitor the current connection count of each backend server 20 by periodically sending health check requests.

[0143] When a new first request arrives, load balancer 10 compares the number of connections of all candidate backend servers and selects the first backend server with the fewest active connections to handle the first request.

[0144] In addition, after selecting the first backend server, the load balancer 10 needs to select the first processor with the fewest threads to handle the first request, based on the number of threads of each processor 21 inside the monitored first backend server.

[0145] Here, the current number of threads for each processor 21 can be determined based on the historical requests sent by the load balancer 10 to each processor 21 and the processing progress of those historical requests. For example, when the load balancer 10 sends a request to a processor 21, the thread count of that processor 21 is incremented by 1, and when the result corresponding to the request is returned by that processor 21, the thread count of that processor 21 is decremented by 1, and so on.

[0146] The system provided in this embodiment can achieve finer-grained load balancing within the backend server by simultaneously considering the number of connections to the backend server and the number of threads to the processor. This not only balances the load between backend servers but also balances the load of each processor within the backend server.

[0147] In some embodiments, the load balancing strategy includes:

[0148] Determine the backend server with the shortest first response time among the candidate backend servers that match the first request sent by the client, and use the backend server with the shortest first response time as the first backend server that matches the first request.

[0149] The processor corresponding to the shortest second response time among the first backend servers is determined, and the processor corresponding to the shortest second response time is used as the first processor to match the first request.

[0150] Here, the initial response time of each backend server 20 can be determined based on the historical average response time. The historical average response time can be obtained through long-term monitoring and statistical analysis of the response time of the backend server 20 in processing all requests.

[0151] Specifically, the load balancer 10 can internally maintain a data list to store and update the first response time of each backend server 20, and query this data list when a new first request is received to select the first backend server based on the first response time.

[0152] Here, the second response time of processor 21 can be determined based on the workload status of processor 21, including but not limited to indicators such as CPU utilization, memory usage, and number of threads. For example, each backend server 20 can periodically report the load status of each processor within it to the load balancer 10.

[0153] After determining the first backend server, the first processor corresponding to the shortest second response time is selected to process the first request based on the second response time of each processor 21 in the first backend server.

[0154] The system provided in this embodiment can reduce client waiting time and improve service response speed by selecting the backend server and processor with the shortest response time to process requests.

[0155] In some embodiments, the first backend server is further configured to:

[0156] Upon receiving the first request, the first processor starts a timer;

[0157] If the timer determines that the processing time of the first request by the first processor has reached a preset duration, the current processing progress of the first request is determined.

[0158] If there are multiple pending tasks in the current first request, the multiple pending tasks are parsed to determine at least one target pending task that can be processed independently among the multiple pending tasks.

[0159] A second request is generated based on at least one of the target tasks to be processed;

[0160] Identify at least one candidate processor in the current first backend server, excluding the first processor, wherein the candidate processor is connected to the first network interface card;

[0161] A second processor is determined from the candidate processors based on the load state of at least one of the candidate processors;

[0162] The second request in the first request is forwarded to the second processor through the first network card.

[0163] When the first processor receives the first request, the first backend server starts a timer. The purpose of this timer is to monitor the time required to process the first request. This timer can use either forward or reverse timing; there is no restriction on which.

[0164] Using a timer, the first backend server can monitor and determine the processing time for the first request by the first processor. If this processing time reaches a preset duration, the first processor will further check the processing progress of the current first request.

[0165] If the first request contains multiple pending tasks, and the processing time has exceeded the preset time, the first backend server will further parse these pending tasks. It will identify at least one pending task that can be processed independently. In other words, it will identify a pending task that can be executed independently without relying on other tasks to complete.

[0166] Based on the identified tasks that can be processed independently, the first server generates a second request. This second request contains tasks that can be separated from the first request and processed in parallel, thereby reducing the complexity and processing time of a single request.

[0167] Next, the first backend server will further identify at least one candidate processor connected to the first network interface card (NIC) in addition to the first processor, so that the request can be forwarded to the other processor through the first NIC.

[0168] It should be noted that, compared to direct forwarding between processors, forwarding via network interface card (NIC) does not require a specific communication interface or protocol between processors. Furthermore, compared to direct forwarding which increases the load on the first processor, in this embodiment, NIC forwarding allows the network hardware to take over the task, reducing the processor's burden.

[0169] Finally, based on the load status of the candidate processors (such as CPU utilization, memory usage, number of current processing tasks, etc.), the most suitable second processor is determined from the candidate processors to handle the second request, and the second request is forwarded to the second processor for processing through the first network card.

[0170] The system provided in this embodiment can dynamically allocate and adjust resources based on the network card configuration of the backend server when processing complex or time-consuming requests, ensuring that the task can be completed in the shortest possible time, thereby avoiding overload of a single processor and further achieving load balancing at the hardware level, thus enhancing the reliability and stability of the system.

[0171] Based on the above embodiments, it can be seen that the load balancing system provided in this application has the following advantages:

[0172] Improved network bandwidth utilization: Distributing network traffic across multiple backend servers effectively improves network bandwidth utilization.

[0173] Network performance optimization: Reduced response time for user requests and improved overall service performance.

[0174] Enhanced system reliability: Service interruptions caused by single-point network interface card (NIC) failures were avoided, improving system reliability and stability. By implementing NIC load balancing technology, network traffic bottlenecks were successfully resolved, improving service stability and availability.

[0175] The load balancing method provided by the present invention is described below. The load balancing method described below and the load balancing system described above can be referred to in correspondence.

[0176] Figure 4 This is a flowchart illustrating the load balancing method provided by the present invention; the method is executed by the load balancing system provided in the above embodiments; as follows Figure 4 As shown, the method includes:

[0177] Step 410: Through the load balancer, respond to the first request sent by the client, determine the first backend server and the first processor of the first backend server, and forward the first request to the first network card corresponding to the first processor;

[0178] Step 420: If the first backend server determines that the processing time of the first processor in processing the first request exceeds a preset time, the second request in the first request is forwarded to the second processor through the first network card, wherein the second processor is the processor in the first backend server and is connected to the first network card.

[0179] Here, the client (such as a browser, API client, etc.) sends an initial request to the load balancer. This request may be an HTTP request, a database query, or other types of network requests. After receiving the request from the client, the load balancer will decide how to distribute this initial request according to the predefined load balancing strategy.

[0180] Specifically, the load balancer selects a backend server from the server pool to handle this first request. This selected backend server is called the first backend server.

[0181] After selecting the primary backend server, the load balancer needs to further determine which processor on that server will actually handle the request. For example, it may select a processor from the primary backend server based on the current load, processing capacity, or other configuration priority rules. This selected processor is called the primary processor.

[0182] It should be understood that in this embodiment, each processor is connected to at least two network cards through the system bus. Therefore, in this embodiment, after selecting the first processor, it is also necessary to further select the first network card corresponding to the first processor.

[0183] In one example, the load balancer is also configured with detailed information about each network interface card (NIC) on each backend server, including the IP address of each NIC, the processor it is connected to, and related load information. Therefore, in this embodiment, the load balancer evaluates the current load of each NIC on the first processor (e.g., network traffic, packet processing rate, error rate, etc.). It also evaluates the current load of other processors connected to each NIC, and finally selects the first NIC by combining the current load of each NIC and the current load of other processors connected to each NIC (e.g., CPU utilization, number of processes and threads, etc.). For example, a NIC with low network traffic and a low error rate, and whose connected processors have sufficient remaining processing power to handle requests, might be selected as the first NIC.

[0184] Finally, the load balancer forwards the request to the first network interface card (NIC) corresponding to the first processor on the first backend server responsible for processing the first request. Specifically, using a network protocol (such as TCP / IP), the load balancer sends the data packet of the first request to the IP address and port of the first NIC.

[0185] When the first backend server receives the first request distributed by the load balancer and determines that the first processor should handle the request, the first backend server will monitor the processing, especially the time required for the first processor to process the first request.

[0186] If, during processing, the first backend server finds that the first processor takes longer than a preset time to process the first request (such as a preset time set based on system performance requirements, user experience standards, or historical data), in order to reduce the burden on the first processor and speed up the processing of the first request, the first backend server will forward a part of the first request, namely the second request, to the second processor through the first network card.

[0187] Here, the second request may be a subtask of the first request, a fragment that can be executed independently, or a request that is associated with the first request but can be processed in parallel.

[0188] Here, the second processor is another processor on the first backend server, and it is also connected to the first network interface card (NIC). In other words, the first NIC serves multiple processors simultaneously, providing hardware-level redundancy and load balancing capabilities.

[0189] The method provided in this embodiment establishes a communication connection between the load balancer and the backend servers, allowing the load balancer to determine which backend server and which processor on that server to forward the client's first request to. This achieves load balancing at the software level. Furthermore, each backend server is equipped with at least two processors and at least two network interface cards (NICs), with each processor connected to at least two NICs via a system bus. Specifically, when the processing time of the first request by the first processor exceeds a preset time, a portion of the second request within the first request can be forwarded to the second processor. This allows requests to be transferred between different processors via NICs, avoiding overload on a single processor and further achieving load balancing at the hardware level, thus enhancing the system's reliability and stability.

[0190] In some embodiments, step 410 is specifically implemented based on the following steps:

[0191] The first backend server responds to the first request sent by the client, determines the request type corresponding to the first request, and filters out candidate backend servers that match the request type from at least one backend server based on the service type of at least one backend server.

[0192] Based on a pre-configured load balancing strategy, a first backend server and a first processor of the first backend server are determined from the candidate backend servers. The load balancing strategy includes at least one of round-robin mechanism, weight allocation, minimum number of connections, and fastest response time.

[0193] Specifically, the load balancer first determines the request type of the first request. This can be based on factors such as the URL path, HTTP type (e.g., GET, POST), or specific fields in the request header.

[0194] Once the request type is determined, the load balancer filters candidate backend servers that match the request type based on the service type provided by at least one backend server. For example, if the first request is a video streaming request, the load balancer will filter backend servers configured with video streaming services.

[0195] After selecting candidate backend servers, the load balancer further applies pre-configured load balancing strategies to determine the primary backend server from among the candidates. These load balancing strategies may include, but are not limited to, the following:

[0196] Polling mechanism: Requests are distributed to backend servers in turn in a sequential manner.

[0197] Weighting: Assign different weights to each backend server based on its processing capacity or resource status, and then distribute requests according to the weight ratio.

[0198] Least Connections: Select the backend server with the fewest current connections (i.e., the lightest load) to handle the request.

[0199] Fastest response time: Select the backend server with the shortest response time based on the average response time of the backend server over a period of time or real-time performance monitoring data.

[0200] Finally, the load balancer will select a backend server from the candidate backend servers as the first backend server according to the load balancing strategy, and may further determine the first processor on that backend server.

[0201] The method provided in this embodiment not only ensures that requests are distributed to backend servers capable of handling that type of request, but also optimizes resource utilization through intelligent load balancing strategies, thereby improving the overall performance and reliability of the system.

[0202] In some embodiments, step 410 is followed by the following steps:

[0203] The load balancer monitors the status of the first backend server and the first network interface card.

[0204] If it is determined that the first network interface card (NIC) is in an unavailable state, the first request is forwarded to the second NIC, which includes other NICs in the first backend server besides the first NIC, or NICs of other backend servers besides the first backend server.

[0205] Here, the load balancer continuously monitors the status of the first backend server and its connected first network interface card (NIC), including but not limited to the NIC's connection status, data traffic, error rate, packet loss rate, and response time. Using this information, the load balancer can assess the health of the first NIC.

[0206] During the monitoring process, the load balancer will determine whether the first network interface card (NIC) is in an unavailable state. Here, an unavailable state means that the first NIC can no longer receive and process requests from the load balancer, nor can it return the corresponding results to the load balancer.

[0207] When the first network interface card (NIC) is detected to be unavailable, the load balancer will select a suitable second NIC to continue processing the first request that should have been forwarded through the first NIC. If the first backend server has multiple NICs, the load balancer can select one of the working NICs as the second NIC. If all NICs on the first backend server are unavailable, or to achieve higher availability and load balancing, the load balancer can also select a NIC from another backend server offering the same service type as the second NIC.

[0208] Finally, the load balancer forwards the first request, which should have been forwarded through the first network interface card (NIC), to the selected second NIC. This ensures that even if the first NIC fails, the client's first request can still be processed promptly, thus guaranteeing system continuity and user experience.

[0209] The method provided in this embodiment ensures that the system can continue to provide services even if some hardware or network components fail, thereby improving the reliability and failover capability of the entire system.

[0210] The method provided by this invention is executed by the system based on the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0211] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a load balancing method, which includes: responding to a first request sent by a client through a load balancer, determining a first backend server and a first processor of the first backend server, and forwarding the first request to a first network interface card (NIC) corresponding to the first processor; and, if the first backend server determines that the processing time of the first request by the first processor exceeds a preset time, forwarding a second request in the first request to a second processor through the first NIC, wherein the second processor is a processor in the first backend server and is connected to the first NIC.

[0212] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0213] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the load balancing method provided by the above methods, the method including: responding to a first request sent by a client through a load balancer, determining a first backend server and a first processor of the first backend server, and forwarding the first request to a first network interface card corresponding to the first processor; through the first backend server, if it is determined that the processing time of the first processor in processing the first request exceeds a preset time, forwarding a second request in the first request to a second processor through the first network interface card, wherein the second processor is a processor in the first backend server and is connected to the first network interface card.

[0214] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the load balancing method provided by the above methods. The method includes: responding to a first request sent by a client through a load balancer, determining a first backend server and a first processor of the first backend server, and forwarding the first request to a first network interface card (NIC) corresponding to the first processor; and, if the first backend server determines that the processing time of the first request by the first processor exceeds a preset time, forwarding a second request in the first request to a second processor through the first NIC, wherein the second processor is a processor in the first backend server and is connected to the first NIC.

[0215] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A load balancing system, characterized in that, include: A load balancer and at least one backend server communicatively connected to the load balancer, the backend server comprising at least two processors and at least two network interface cards (NICs), and each of the processors being connected to the at least two NICs via a system bus; The load balancer is used to respond to a first request sent by the client, determine a first backend server and a first processor of the first backend server, and forward the first request to the first network interface card corresponding to the first processor. The first backend server is further configured to, when it is determined that the processing time of the first processor in processing the first request exceeds a preset time, forward the second request in the first request to the second processor through the first network card, wherein the second processor is the processor in the first backend server and is connected to the first network card; It also includes: a system service processor, which is connected in communication with the load balancer and at least one of the back-end servers; The system service processor is used for: Configure the virtual IP address of the load balancer so that the client can access it based on the virtual IP address; Configure the backend server list of the load balancer, wherein the backend server list includes the IP address, port number and service type of at least one backend server; Configure the health check protocol and parameters corresponding to the backend servers in the backend server list; Configure the load balancing strategy of the load balancer.

2. The load balancing system according to claim 1, characterized in that, The load balancer is also used for: In response to a first request sent by the client, the request type corresponding to the first request is determined, and candidate backend servers matching the request type are selected from at least one of the backend servers based on the service type of at least one of the backend servers. Based on a pre-configured load balancing strategy, a first backend server and a first processor of the first backend server are determined from the candidate backend servers.

3. The load balancing system according to claim 1, characterized in that, The load balancer is also used for: Monitor the status of the first backend server and the first network interface card; If it is determined that the first network interface card (NIC) is in an unavailable state, the first request is forwarded to the second NIC, which includes other NICs in the first backend server besides the first NIC, or NICs of other backend servers besides the first backend server.

4. The load balancing system according to claim 1, characterized in that, The system service processor is also used for: Acquire monitoring data of the load balancing system, the monitoring data including the first performance and first status of the load balancer and the second performance and second status of each processor in each of the backend servers; Based on the monitoring data, the load balancing system is adjusted, including at least one of adjusting the load balancing strategy and adjusting the number of backend servers.

5. The load balancing system according to claim 2 or 1, characterized in that, The load balancing strategy includes: Determine the backend server pointed to by the first pointer among the candidate backend servers that match the first request sent by the current client, and take the backend server pointed to by the first pointer as the first backend server that matches the first request. Determine the processor pointed to by the second pointer in the first backend server and use the processor pointed to by the second pointer as the first processor matched by the first request; When the first processor receives the first request, the first pointer points to the next backend server in a first polling order, and the second pointer points to the next processor in the first backend server in a second polling order.

6. The load balancing system according to claim 2 or 1, characterized in that, The load balancing strategy includes: Determine the backend server with the fewest connections among the candidate backend servers that match the first request sent by the current client, and use the backend server with the fewest connections as the first backend server that matches the first request. Determine the processor corresponding to the minimum number of threads in the current first backend server, and use the processor corresponding to the minimum number of threads as the first processor to match the first request.

7. The load balancing system according to claim 2 or 1, characterized in that, The load balancing strategy includes: Determine the backend server with the shortest first response time among the candidate backend servers that match the first request sent by the client, and use the backend server with the shortest first response time as the first backend server that matches the first request. The processor corresponding to the shortest second response time among the first backend servers is determined, and the processor corresponding to the shortest second response time is used as the first processor to match the first request.

8. The load balancing system according to claim 1, characterized in that, The first backend server is also used for: Upon receiving the first request, the first processor starts a timer; If the timer determines that the processing time of the first request by the first processor has reached a preset duration, the current processing progress of the first request is determined. If there are multiple pending tasks in the current first request, the multiple pending tasks are parsed to determine at least one target pending task that can be processed independently among the multiple pending tasks. A second request is generated based on at least one of the target tasks to be processed; Identify at least one candidate processor in the current first backend server, excluding the first processor, wherein the candidate processor is connected to the first network interface card; A second processor is determined from the candidate processors based on the load state of at least one of the candidate processors; The second request in the first request is forwarded to the second processor through the first network card.

9. A load balancing method, characterized in that, The method is applied to the load balancing system as described in any one of claims 1 to 8, and the method includes: The load balancer responds to the first request sent by the client, determines the first backend server and the first processor of the first backend server, and forwards the first request to the first network interface card corresponding to the first processor. If the first backend server determines that the processing time of the first processor in processing the first request exceeds a preset time, the second request in the first request is forwarded to the second processor through the first network card. The second processor is the processor in the first backend server and is connected to the first network card.

Images