High-concurrency network connection load balancing method based on operating system

Abstract

A high-concurrency network connection load balancing method based on an operating system comprises the steps that an application layer is controlled to submit an IP-CPU binding request, and the binding request is transmitted to a kernel layer from the application layer through a kernel interface; binding information of the binding request comprises a target IP address and a target CPU core; when the kernel layer receives a binding request, binding information is stored in a global hash table; wherein the key of the global hash table is the hash value of the target IP address, and the value of the global hash table is the target CPU core; receiving a network message request, and analyzing a request IP address of the network message request; and determining a request CPU core corresponding to the hash value of the request IP address through the global hash table, wherein the request CPU core is the CPU core for processing the network message request. According to the invention, accurate load scheduling and resource balanced utilization are realized, and the stability and reliability of the system in a high-concurrency environment are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of load balancing, and particularly relates to a high-concurrency network connection load balancing method based on an operating system. Background Art

[0002] With the efforts of researchers and the expansion of market opportunities, domestic operating systems have gradually grown and matured. While making progress and breakthroughs, domestic operating systems still face many development challenges and bottlenecks. In current mainstream desktop distributions, load balancing technology generally adopts RPS and RFS, and network packets are randomly distributed to a certain CPU in a multi-core CPU through a hash algorithm to achieve load balancing. As follows: The load balancing strategies of domestic operating systems are mainly divided into hardware and software: (1) RPS / RFS in software: It relies on five-tuple hashing to randomly distribute packets and cannot perform targeted optimization for critical service IPs (such as high-frequency trading and video streams), resulting in uncontrollable latency.

[0003] (2) RSS of network card in hardware: The network card hardware distributes traffic to multiple queues of the network card according to the quadruple of the received packet (the calculation method can be specified). Although this method can specify the hash strategy, it still cannot bind the receive queue to a specific IP.

[0004] However, traditional load balancing strategies are not transparent to users, and users cannot manually set the correspondence between CPU cores and traffic. The traditional load balancing method based on a hash table has "unconscious" allocation in high-concurrency network connections, and in traffic scenarios with more complex priorities, it cannot achieve a finer-grained CPU core load division. Summary of the Invention

[0005] In order to overcome the above defects, the present invention provides a high-concurrency network connection load balancing method based on an operating system to solve problems such as unconscious allocation in load balancing and inability to perform a finer-grained load division.

[0006] The present invention provides a high-concurrency network connection load balancing method based on an operating system, including the following steps: Controlling the application layer to submit an IP-CPU binding request, and transferring the binding request from the application layer to the kernel layer through a kernel interface; the binding information in the binding request includes a target IP address and a target CPU core; When the kernel layer receives the binding request, storing the binding information in a global hash table; wherein, the key of the global hash table is the hash value of the target IP address, and the value of the global hash table is the target CPU core; Receive a network packet request and parse the request IP address of the network packet request; determine the requested CPU core corresponding to the hash value of the request IP address through the global hash table, and the requested CPU core is the CPU core that processes the network packet request.

[0007] A further improvement of the present invention is that before storing the binding information into the global hash table, it further includes: Check whether there is a node with the hash value of the target IP address as the key in the global hash table; If it exists, update the node with the hash value of the target IP address as the key with the target CPU core as the value; If it does not exist, store it in the hash table with the hash value of the target IP address as the key and the target CPU core as the value.

[0008] A further improvement of the present invention is that the hash value of the target IP address is obtained by calling the jhash() kernel interface with the target IP address as a parameter.

[0009] A further improvement of the present invention is that parsing the request IP address of the network packet request specifically is: Extract the header information of the network packet request and obtain it by calling the netfi_receive_skb_internal() kernel interface with the header information as a parameter.

[0010] A further improvement of the present invention is that determining the requested CPU core corresponding to the hash value of the request IP address through the global hash table includes: When there is a node with the hash value of the request IP address as the key in the global hash table, obtain the target CPU corresponding to the node with the hash value of the request IP address as the key as the corresponding requested CPU core, and distribute the network packet request to the receive queue of the requested CPU core.

[0011] A further improvement of the present invention is that when there is no node with the hash value of the request IP address as the key in the global hash table, distribute the network packet request to the receive queue of the CPU core that meets the preset conditions.

[0012] A further improvement of the present invention is that when the distribution of the network packet request to the receive queue of the requested CPU core fails, distribute the network packet request to the receive queue of the CPU core that meets the preset conditions.

[0013] The beneficial effects of the present invention: Through the IP-CPU binding method, it is possible to specify the CPU according to the IP, achieving precise load scheduling and balanced utilization of resources, reducing the overload of a single CPU, lowering the risk of system failures caused by uneven loads, and improving the stability and reliability of the system in a high-concurrency environment. In the case of more complex priorities, the network traffic can be divided into finer granularities according to the IP, replacing the original method of processing network packets in units of processes, making the processing of network traffic more targeted and more adaptable in terms of processing details. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0015] Figure 1 It is a logical architecture diagram of an embodiment of the present invention; Figure 2 It is another logical architecture diagram of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] In order to further understand the technical solutions and beneficial effects of the present invention, the following will describe in detail the technical solutions of the present invention and the beneficial effects they produce in conjunction with the drawings.

[0017] Abbreviations and Definitions of Key Terms: RPS (Receive Packet Steering): A Linux kernel mechanism that distributes network packets to multiple-core CPUs for processing through a hash algorithm to achieve load balancing.

[0018] RFS (Receive Flow Steering): An extension based on RPS that optimizes CPU affinity by tracking the flow processing location, but it takes effect only after the first packet is processed.

[0019] RSS (receive side scaling): Receive-side scaling. The network card hardware distributes traffic to multiple queues of the network card according to the quadruple of the received packet (the calculation method can be specified).

[0020] CPU core: When Linux manages multiple CPU cores, it uses physical numbers to distinguish different CPUs, and the numbers are the CPU cores, also known as CPU logical numbers.

[0021] Socket: It is an intermediate software abstraction layer for communication between the application layer and the TCP / IP protocol family in Linux. Socket provides a set of interfaces that enable application programs to communicate over the network through these interfaces.

[0022] Please combine Figure 1 and Figure 2 As shown, the present invention is completed through the interaction between the application layer and the kernel layer. The application layer is responsible for initiating requests, and the kernel layer receives and processes the requests.

[0023] In this embodiment, on the domestic D2000 processor, the Galaxy Kylin V10 version is installed. By specifying the binding relationship between the IP and the CPU through the user layer interface, precise directional transmission of specific IP address packets is achieved, meeting the personalized requirements of different services for computing resources. For example, for a service IP address that requires high computing performance, it can be bound to a CPU with stronger performance to improve service processing efficiency.

[0024] Specifically as follows: S1. The user layer or the application layer (also called the application program) sets the binding relationship between the destination IP address and the destination CPU core through the upper layer interface, and calls the socket interface to complete the setting (i.e., corresponding to [[ID=P16]] Figure 1 in (1) calling the interface to set the binding relationship between the ip and the cpu, and Figure 2 in setsockopt() to pass the IP-CPU binding information). An example of the socket interface is as follows: setsockopt(fd, SOL_SOCKET, SO_CPU_BIND, &bind_info, sizeof(bind_info)); Among them, the meanings of each parameter in the interface are as follows: fd: Represents the socket handle to be set; SOL_SOCKET: Represents setting the socket option; SO_CPU_BIND: A newly defined macro in this embodiment, representing setting the IP-CPU binding relationship of the socket; bind_info: Represents the binding information that the application layer needs to pass to the kernel layer. The kernel layer establishes an IP-CPU hash table based on this information. The definition structure of bind_info is as follows: struct ip_cpu_bind { char ip_addr[INET_ADDRSTRLEN]; / / Destination IP address int cpu_id; / / Destination CPU core, that is, the bound CPU logical core number (0-based) }; After the call is completed, the target IP address in the bind_info passed by the application layer will be bound to the target CPU core therein.

[0025] S2. After the kernel layer receives the binding request passed by the application layer, it calls the jhash() kernel interface to calculate the hash value corresponding to the target IP address of the binding information in the binding request, and uses this hash value as the subscript of the global hash table ip_cpu_bind_table, that is, the key of the global hash table. Before storing the target CPU core as the value in the global hash table, check whether there is a node with the hash value of the target IP address as the key in the global hash table; if it exists, update the node with the hash value of the target IP address as the key with the target CPU core as the value; if it does not exist, store it in the hash table with the hash value of the target IP address as the key and the target CPU core as the value (that is, corresponding to Figure 1 in (2) storing the ip address into the hash table, and the table entry is the cpu number, and Figure 2 in jhash() calculating the IP hash value and maintaining the global hash table ip_cpu_bind_table). The final structure is: ip_cpu_bind_table[hash value of the target IP address] = target CPU core.

[0026] S3. When the network interface card receives a network packet from an external request, the hardware device interrupt triggers the kernel layer network protocol stack processing flow. The network packet request is first parsed by the driver layer to extract the header information of the second and third layers, and the netfi_receive_skb_internal() interface in the kernel layer is called. In this interface, the request IP address of the network packet request is parsed, and the jhash() interface is called to calculate the hash value of the request IP address. The global hash table is queried through this hash value to obtain the request CPU core bound to the hash value of the request IP address (corresponding to Figure 1 in (3) querying the cpu number according to the destination ip in the packet, and the destination ip in the packet refers to the request IP address; and Figure 2 in netfi_receive_skb_internal() parsing the destination IP and jhash() calculating the hash value).

[0027] When there is a node with the hash value of the request IP address as the key in the global hash table, obtain the target CPU core corresponding to the node with the hash value of the request IP address as the key as the request CPU core, and call the enqueue_to_backlog() interface to distribute the network packet request to the receive queue of the request CPU core (corresponding to Figure 1 in (4) putting the data packet on the receive queue of the corresponding cpu, andFigure 2 In the query global hash table, enqueue_to_backlog() sends the packet to the corresponding CPU receive queue), and thereafter, network packets with the same requested IP address will be sent to this CPU. enqueue_to_backlog() is an existing interface in the kernel layer, and the interface example is as follows: enqueue_to_backlog(skb, cpu); Among them, the meanings of the parameters in the interface are as follows: skb: represents the received sk_buff packet, that is, the network packet; cpu: represents the corresponding cpu number in the ip_cpu_bind_table global table, that is, the requested CPU core; It should be noted that when the return value of the enqueue_to_backlog() interface is 1, it means that the network packet fails to join the receive queue of the requested CPU core, indicating that the receive queue of this requested CPU core is full. In this case, following the default processing principle of Linux, it is distributed to the CPU cores that meet the preset conditions for processing, and the CPU cores that meet the preset conditions can be the local CPU cores (corresponding to Figure 1 in (5) put into the current cpu receive queue).

[0028] In another case of this embodiment, when there is no node with the hash value of the requested IP address as the key in the global hash table, in this case, following the default processing principle of Linux, it is distributed to the CPU cores that meet the preset conditions for processing, and the CPU cores that meet the preset conditions can be the local CPU cores.

[0029] The beneficial effects of the present invention are as follows: By setting a dedicated interface in the user layer / application layer, allowing the user layer / application layer to actively specify the correspondence between the target IP address and the target CPU core, the active control of load distribution is realized.

[0030] Using the global hash table to store the binding information of IP and CPU, querying the global hash table during the packet processing process, and determining the requested CPU core for packet forwarding according to the query result, realizing the accurate directional sending of packets.

[0031] Designed according to the characteristics of domestic operating systems, it has good compatibility with domestic operating systems and can give full play to the advantages of domestic operating systems. At the same time, the design of the user layer / application layer interface makes this method have strong scalability. The user layer / application layer can flexibly adjust the binding strategy of IP and CPU according to different service requirements to adapt to the changing network application scenarios.

[0032] Precise load scheduling and balanced resource utilization reduce the overload of a single CPU, lower the risk of system failures caused by uneven loads, and improve the stability and reliability of the system in a high-concurrency environment. In cases where priorities are more complex, such as when resource allocation needs to be performed for multiple IPs, network packets are no longer processed on a per-process basis. Instead, network traffic is divided into finer granularities based on IPs, giving full play to the role of policy customization.

[0033] Although the present invention has been described using the above preferred embodiments, it is not intended to limit the scope of protection of the present invention. Any person skilled in the art, without departing from the spirit and scope of the present invention, making various changes and modifications to the above embodiments still falls within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. A high-concurrency network connection load balancing method based on an operating system, characterized in that, Including: Controlling the application layer to submit an IP-CPU binding request, and passing the binding request from the application layer to the kernel layer through a kernel interface; The binding information of the binding request includes a target IP address and a target CPU core; When the kernel layer receives the binding request, storing the binding information into a global hash table; wherein, the key of the global hash table is the hash value of the target IP address, and the value of the global hash table is the target CPU core; Receiving a network packet request, and parsing the request IP address of the network packet request; determining the request CPU core corresponding to the hash value of the request IP address through the global hash table, and the request CPU core is the CPU core for processing the network packet request.

2. The high-concurrency network connection load balancing method based on an operating system according to claim 1, wherein Before storing the binding information into the global hash table, it further includes: Searching in the global hash table to check if there is a node with the hash value of the target IP address as the key; If it exists, updating the node with the hash value of the target IP address as the key with the target CPU core as the value; If it does not exist, storing it into the hash table with the hash value of the target IP address as the key and the target CPU core as the value.

3. A high-concurrency network connection load balancing method based on an operating system according to any one of claims 1 or 2, characterized in that The hash value of the target IP address is obtained by calling the jhash() kernel interface with the target IP address as a parameter.

4. A high-concurrency network connection load balancing method based on an operating system according to claim 1, characterized in that, Specifically, parsing the request IP address of the network packet request means: Extracting the header information of the network packet request, and obtaining it by calling the netfi_receive_skb_internal() kernel interface with the header information as a parameter.

5. A high-concurrency network connection load balancing method based on an operating system according to claim 1, characterized in that, Determining the request CPU core corresponding to the hash value of the request IP address through the global hash table includes: When there is a node with the hash value of the request IP address as the key in the global hash table, obtaining the target CPU corresponding to the node with the hash value of the request IP address as the key as the corresponding request CPU core, and distributing the network packet request to the receive queue of the request CPU core.

6. The high-concurrency network connection load balancing method based on an operating system according to claim 5, characterized in that When there is no node with the hash value of the request IP address as the key in the global hash table, distributing the network packet request to the receive queue of a CPU core meeting a preset condition.

7. A high-concurrency network connection load balancing method based on an operating system according to claim 5, characterized in that, When the distribution of the network packet request to the receive queue of the request CPU core fails, distributing the network packet request to the receive queue of a CPU core meeting a preset condition.

Images