An openstack load balancing data monitoring method, system, storage medium and device

Abstract

The application provides a kind of openstack load balancing data monitoring method, comprising the following steps: creating amphora virtual machine, and according to the CPU number of amphora virtual machine, the configuration of the number of multiple queues, the number of haproxy processes and the number of statistics sockets of the network card of virtual machine;After the start of haproxy process, the statistical data in socket is filtered and collected one by one;After collecting data corresponding to the socket of each process, the data of total new connection number and maximum new connection number are outputted;And when the data needs to be monitored, the latest statistical data is read from the storage file, and returned to the user.The application is helpful to real-time acquisition of the statistical data of each haproxy process and integration processing, and the reaction of accurate load balancing statistical data.

Description

Technical Field

[0001] This invention relates to the field of data monitoring technology, and in particular to a method, system, storage medium, and device for monitoring OpenStack load balancing data. Background Technology

[0002] In today's rapidly developing internet era, various intelligent products powered by the cloud have entered our daily lives. Despite the rapid adoption of cloud computing technology, the ability to reliably distribute workloads across multiple cloud platforms, data centers, and hybrid infrastructures remains a long-standing deficiency. This results in uneven workload distribution and degraded application performance, which can be avoided by better managing workloads globally. Currently, the load balancing project Octavia in OpenStack uses single-process HAProxy by default. However, with increasing network traffic, the performance requirements for load balancing are also rising. Enabling HAProxy's multi-process mode and mapping it to each CPU core can fully utilize CPU performance and significantly improve HAProxy's load balancing capabilities.

[0003] However, when HAProxy is running in multiple processes, using a single socket to collect load balancing data can lead to statistical distortions, such as in the count of new connections per second and the maximum number of new connections. Furthermore, it's impossible to obtain data for each process individually. The Octavia project currently only supports single-socket statistics, which results in data errors when multiple processes are running. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, system, storage medium and device for monitoring OpenStack load balancing data, so as to solve the shortcomings of existing OpenStack load balancing data monitoring methods and systems.

[0005] To achieve the above objectives, this invention provides a method for monitoring OpenStack load balancing data, comprising the following steps:

[0006] Create an Amphora virtual machine, and configure the number of queues, HAProxy processes, and the number of sockets for the virtual machine's network interface card according to the number of CPUs in the Amphora virtual machine;

[0007] After the haproxy process starts, it filters and collects statistical data from each socket one by one.

[0008] After collecting data from the sockets corresponding to each process, the data is aggregated, and the total number of newly established connections and the maximum number of newly established connections are output; and

[0009] When monitoring data is needed, the latest statistics are read from the storage file and returned to the user.

[0010] In some embodiments, the command to create an Amphora virtual machine is issued through the OpenStack user interface.

[0011] In some embodiments, the number of multiple queues, the number of haproxy processes, and the number of sockets counted are equal.

[0012] In some embodiments, the specific method for collecting statistical data in the socket is to use the socat and grep commands to filter and redirect the statistical data in the socket.

[0013] In some embodiments, when monitoring data is needed, load balancing statistics are obtained through a URL interface. Parameters are added to the URL to obtain the target data. After receiving the URL request, Octavia parses the requested data fields. It checks whether the socket file exists and whether there is statistical data. If the socket file does not exist or there is no statistical data, it returns no available data. If the socket file exists and statistical data exists, it collects and extracts data for each socket according to the query fields. The collected data is summed to obtain a total value, and the data, along with the data from individual processes, is assembled and returned.

[0014] In some embodiments, the parameters include the following:

[0015] Rate: The number of new connections created by each process and the total number of new connections created by all processes, i.e., the number of new connections created in the past second;

[0016] Rmax: The maximum number of newly established connections and the total maximum number of connections in the history of each process;

[0017] Bin, Bout: Total number of bytes received and total number of bytes sent;

[0018] Scur: Current session count, etc., can be found in hatop statistics.

[0019] Another aspect of the present invention provides a system for monitoring OpenStack load balancing data, comprising:

[0020] The module is used to create Amphora virtual machines and configure the number of queues, HAProxy processes, and the number of sockets for the virtual machine's network interface card based on the number of CPUs in the Amphora virtual machine.

[0021] The data collection module is used to filter and collect statistical data from each socket after the haproxy process starts.

[0022] The output module is used to collect data from the sockets corresponding to each process, summarize the data, and output the total number of newly established connections and the maximum number of newly established connections; and

[0023] The read module is used to read the latest statistics from the storage file when monitoring data is needed and return it to the user.

[0024] In another aspect, the present invention provides a computer-readable storage medium storing computer program instructions that, when executed, implement any of the methods described above.

[0025] In another aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, performs any of the methods described above.

[0026] The present invention has at least the following beneficial technical effects:

[0027] 1. It can obtain and integrate the statistical data of each HAProxy process in real time, reflecting accurate load balancing statistics;

[0028] 2. The cloud environment based on OpenStack provides a load balancing statistics system. In scenarios using HAProxy multi-process, it can obtain accurate statistics and statistics for each HAProxy process, enabling accurate monitoring of load balancing and thus a more accurate judgment of business volume and system performance.

[0029] 3. Resolved the issues of inaccurate statistics for a single socket and the inability to obtain statistics for each process. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of an OpenStack load balancing data monitoring method provided according to an embodiment of the present invention;

[0032] Figure 2This is a schematic diagram of an OpenStack load balancing data monitoring system provided according to an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a computer-readable storage medium for implementing a method for monitoring OpenStack load balancing data according to an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the hardware structure of a computer device for performing an OpenStack load balancing data monitoring method according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0036] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two different entities or different parameters with the same name. Therefore, "first" and "second" are merely for convenience of expression and should not be construed as limiting the embodiments of the present invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as other steps or units inherent in a process, method, system, product, or device that includes a series of steps or units.

[0037] Based on the above objectives, a first aspect of the present invention provides an embodiment of a method for monitoring OpenStack load balancing data. Figure 1 This diagram illustrates an embodiment of the OpenStack load balancing data monitoring method provided by the present invention. Figure 1 As shown, the embodiments of the present invention include the following steps:

[0038] Step S10: Create an Amphora virtual machine, and configure the number of multiple queues, the number of HAProxy processes, and the number of sockets to be counted for the virtual machine's network card according to the number of CPUs in the Amphora virtual machine;

[0039] Step S20: After the haproxy process starts, it filters and collects statistical data from each socket.

[0040] Step S30: After collecting data from the socket corresponding to each process, summarize the data and output the total number of newly established connections and the maximum number of newly established connections.

[0041] Step S40: When monitoring data is required, read the latest statistical data from the storage file and return it to the user.

[0042] Load balancing refers to distributing operations across multiple operational units for execution, thereby collectively completing tasks and improving overall efficiency. OpenStack is a cloud computing open-source infrastructure project. Octavia is an open-source, carrier-scale load balancing solution designed to work with OpenStack. Amphora is a component of the Octavia project, serving as the load balancing execution unit. Haproxy is free and open-source software written in C, providing high availability, load balancing, and application proxying based on TCP and HTTP; it is the load balancing software currently used by the Octavia project. A socket is an intermediate software abstraction layer for communication between the application layer and the TCP / IP protocol suite. It is a set of interfaces through which users can communicate and retrieve data. Network interface card (NIC) multi-queueing is a technique to address network I / O bandwidth QoS (Quality of Service) issues. NIC multi-queue drivers bind each queue to different cores via interrupts, thereby resolving the single-core CPU bottleneck when network I / O bandwidth increases, improving network forwarding efficiency and bandwidth performance. Hatop is an open-source data statistics tool for HAProxy that can monitor HAProxy's new connection count, maximum new connection count, and number of bytes processed.

[0043] In some preferred embodiments, the instructions to create the Amphora virtual machine are issued through the OpenStack user interface.

[0044] In some preferred embodiments, the number of multiple queues, the number of haproxy processes, and the number of counted sockets are equal.

[0045] In some preferred embodiments, the specific method for collecting statistical data in the socket is to use the socat and grep commands to filter and redirect the statistical data in the socket.

[0046] In a specific embodiment, the OpenStack load balancing data monitoring method of the present invention may include:

[0047] 1) In a multi-process HAProxy environment, adapt to the number of CPU cores of the Amphora virtual machine, configure the number of network card queues of the Amphora virtual machine, the number of HAProxy processes, and count the number of sockets and store them in a directory, and bind each process to a specified process ID;

[0048] 2) Obtain data from each socket using socat, and perform filtering and selection as needed;

[0049] 3) Periodically traverse each socket, collect and organize the expected data, and return it to the client.

[0050] In some preferred embodiments, when monitoring data is required, the following operations are performed:

[0051] a) Obtain load balancing statistics through the URL interface. Add parameters to the URL to obtain the target data. After receiving the URL request, Octavia parses the requested data fields.

[0052] Request type: GET

[0053] Request parameters:

[0054] Rate: The number of new connections created by each process and the total number of new connections created by all processes, i.e., the number of new connections created in the past second.

[0055] Rmax: The maximum number of newly established connections and the total maximum number of connections in the history of each process.

[0056] Bin, Bout: Total number of bytes received and total number of bytes sent.

[0057] Scur: Current session count, etc., can be found in hatop statistics.

[0058] b. After receiving the URL request, Octavia parses the data fields in the request.

[0059] c. Check if the socket file exists and if there are statistics. If the socket file does not exist or there are no statistics, return that no data is available.

[0060] d. If the socket file exists and statistical data exists, then data collection and interception processing are performed on each socket according to the query fields.

[0061] e. The collected data are summed to obtain the total value, and the data from each individual process is then assembled and returned.

[0062] like Figure 1As shown, this invention proposes a method for monitoring OpenStack load balancing data. During cluster setup, the connection status of each node is constantly monitored. When a node's heartbeat fails to connect, a session backup module is triggered to record all session content during subsequent cluster setup, as well as the information of the node that lost connection at that moment, and saves the timestamp. When the node is detected again, the backed-up operation session information is automatically synchronized to that node to complete the missing operations. After completion, the cluster checks the integrity of its functions through a detection module. If the newly recovered node affects the normal operation of the cluster, the operation is automatically rolled back, prioritizing the integrity of the functions.

[0063] The aforementioned method for monitoring OpenStack load balancing data allows for real-time acquisition and integration of statistical data for each HAProxy process, resulting in accurate load balancing statistics. This invention provides a load balancing statistics method for OpenStack cloud environments. In scenarios using multiple HAProxy processes, obtaining accurate statistical data and statistics for each HAProxy process enables precise monitoring of load balancing, leading to a more accurate assessment of business volume and system performance. It also solves the problems of inaccurate statistics for individual sockets and the inability to obtain statistics for each process.

[0064] A second aspect of this invention also provides a system for monitoring OpenStack load balancing data. Figure 2 This diagram illustrates an embodiment of the OpenStack load balancing data monitoring system provided by the present invention. Figure 2 As shown, an OpenStack load balancing data monitoring system includes: a creation module 10, used to create Amphara virtual machines and configure the number of multi-queues, HAProxy processes, and the number of sockets to be counted for the virtual machine's network interface card based on the number of CPUs in the Amphara virtual machine; a collection module 20, used to filter and collect statistical data from each socket after the HAProxy process starts; an output module 30, used to collect data from the sockets corresponding to each process, summarize the data, and output the total number of newly established connections and the maximum number of newly established connections; and a reading module 40, used to read the latest statistical data from the storage file when monitoring data is needed and return it to the user.

[0065] The system using the OpenStack load balancing data monitoring described above can perform the following operations:

[0066] Step S10: Create an Amphora virtual machine, and configure the number of multiple queues, the number of HAProxy processes, and the number of sockets to be counted for the virtual machine's network card according to the number of CPUs in the Amphora virtual machine;

[0067] Step S20: After the haproxy process starts, it filters and collects statistical data from each socket.

[0068] Step S30: After collecting data from the socket corresponding to each process, summarize the data and output the total number of newly established connections and the maximum number of newly established connections.

[0069] Step S40: When monitoring data is required, read the latest statistical data from the storage file and return it to the user.

[0070] The aforementioned OpenStack load balancing data monitoring system can acquire and integrate statistical data for each HAProxy process in real time, providing accurate load balancing statistics. This invention provides a load balancing statistics system based on OpenStack in a cloud environment. In scenarios using multiple HAProxy processes, it acquires accurate statistical data and statistics for each HAProxy process, enabling accurate monitoring of load balancing and thus a more accurate assessment of business volume and system performance. It solves the problems of inaccurate statistics for individual sockets and the inability to obtain statistical data for each process.

[0071] A third aspect of the present invention also provides a computer-readable storage medium. Figure 3 A schematic diagram of a computer-readable storage medium for implementing an OpenStack load balancing data monitoring method according to an embodiment of the present invention is shown. Figure 3 As shown, the computer-readable storage medium 3 stores computer program instructions 31, which can be executed by a processor. When the computer program instructions 31 are executed, the above method is implemented as follows:

[0072] Step S10: Create an Amphora virtual machine, and configure the number of multiple queues, the number of HAProxy processes, and the number of sockets to be counted for the virtual machine's network card according to the number of CPUs in the Amphora virtual machine;

[0073] Step S20: After the haproxy process starts, it filters and collects statistical data from each socket.

[0074] Step S30: After collecting data from the socket corresponding to each process, summarize the data and output the total number of newly established connections and the maximum number of newly established connections.

[0075] Step S40: When monitoring data is required, read the latest statistical data from the storage file and return it to the user.

[0076] It should be understood that, where there is no conflict, all the embodiments, features and advantages described above for the method of OpenStack load balancing data monitoring according to the present invention are equally applicable to the system and storage medium for OpenStack load balancing data monitoring according to the present invention.

[0077] A fourth aspect of the present invention also provides a computer device, including a memory 402 and a processor 401, wherein the memory stores a computer program that, when executed by the processor, implements the method of any of the above embodiments.

[0078] like Figure 4 The diagram shown is a hardware structure schematic of an embodiment of a computer device for executing the OpenStack load balancing data monitoring method provided by the present invention. Figure 4 Taking the computer device shown as an example, this computer device includes a processor 401 and a memory 402, and may also include an input device 403 and an output device 404. The processor 401, memory 402, input device 403, and output device 404 can be connected via a bus or other means. Figure 4 Taking a bus connection as an example, input device 403 can receive input digital or character information, as well as generate key signal inputs related to user settings and function control of the OpenStack load balancing data monitoring system. Output device 404 may include display devices such as a display screen.

[0079] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the OpenStack load balancing data monitoring method in this embodiment. Memory 402 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by using the OpenStack load balancing data monitoring method, etc. In addition, memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 402 may optionally include memory remotely located relative to processor 401, and these remote memories can be connected to the local module via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] The processor 401 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 402, thereby implementing the OpenStack load balancing data monitoring method described in the above method embodiment.

[0081] Finally, it should be noted that the computer-readable storage medium (e.g., memory) described herein can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. By way of example, and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which can act as external cache memory. By way of example, and not limitation, RAM can be obtained in various forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage devices disclosed herein are intended to include, but are not limited to, these and other suitable types of memory.

[0082] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0083] The various exemplary logic blocks, modules, and circuits described herein can be implemented or performed using the following components designed to perform the functions herein: general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP, and / or any other such configuration.

[0084] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0085] It should be understood that, as used herein, the singular form "a" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the associatedly listed items. The embodiment numbers disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0086] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for monitoring OpenStack load balancing data, characterized in that, Includes the following steps: Create an Amphora virtual machine, and configure the number of queues, HAProxy processes, and the number of sockets for the virtual machine's network interface card according to the number of CPUs in the Amphora virtual machine; After the haproxy process starts, it filters and collects statistical data from each socket one by one. After collecting data from the socket corresponding to each process, the specific method for collecting statistical data from the socket is to use the `socat` and `grep` commands to filter and redirect the statistical data from the socket, summarize it, and output the total number of newly established connections and the maximum number of newly established connections; and When monitoring data is needed, the latest statistics are read from the storage file and returned to the user. Specifically, load balancing statistics are obtained via a URL interface when monitoring data is required. Parameters are added to the URL to retrieve the target data. After receiving the URL request, Octavia parses the requested data fields. The function checks if the socket file exists and if there are statistical data. If the socket file does not exist or there are no statistical data, it returns "No data available." If the socket file exists and there are statistical data, it performs data collection and truncation processing on each socket based on the query fields. The collected data are summed to obtain the total value, and the data from each individual process is then assembled and returned.

2. The method according to claim 1, characterized in that, The command to create the Amphora virtual machine is issued through the OpenStack interface.

3. The method according to claim 1, characterized in that, The number of queues, the number of HAProxy processes, and the number of sockets counted are equal.

4. The method according to claim 1, characterized in that, The parameters include the following: Rate: The number of new connections created by each process and the total number of new connections created by all processes, i.e., the number of new connections created in the past second; Rmax: The maximum number of newly established connections and the total maximum number of connections in the history of each process; Bin, Bout: Total number of bytes received and total number of bytes sent; Scur: Current session count, etc., can be found in hatop statistics.

5. A system for monitoring OpenStack load balancing data, characterized in that, include: The module is used to create Amphora virtual machines and configure the number of queues, HAProxy processes, and the number of sockets to be counted for the virtual machine's network interface card (NIC) based on the number of CPUs in the Amphora NIC. After the HAProxy process starts, it filters and collects statistical data from each socket. Specifically, the statistical data from the socket is collected by using the socat and grep commands to filter and redirect the statistical data from the socket, and the NIC's number of queues, HAProxy processes, and the number of sockets to be counted are configured. The data collection module is used to filter and collect statistical data from each socket after the haproxy process starts. The output module is used to collect data from the sockets corresponding to each process, summarize the data, and output the total number of newly established connections and the maximum number of newly established connections; and The read module is used to read the latest statistical data from the storage file when monitoring data is needed and return it to the user. Specifically, when monitoring data is needed, load balancing statistics are obtained through a URL interface. Parameters are added to the URL to obtain the target data. After receiving the URL request, Octavia parses the requested data fields. The function checks if the socket file exists and if there are statistical data. If the socket file does not exist or there are no statistical data, it returns "No data available." If the socket file exists and there are statistical data, it performs data collection and truncation processing on each socket based on the query fields. The collected data are summed to obtain the total value, and the data from each individual process is then assembled and returned.

6. A computer-readable storage medium, characterized in that, The system stores computer program instructions that, when executed, implement the method as described in any one of claims 1-4.

7. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, which, when executed by the processor, performs the method as described in any one of claims 1-4.

Images