MaxScale container automatic deployment method and system based on k8s
By implementing the automatic deployment method of MaxScale containers in Kubernetes environment, dynamically adjusting container locations, predicting traffic peaks, optimizing transmission protocols and monitoring performance, the performance loss problems caused by MaxScale read and write forwarding are solved, and database write performance and system scalability are significantly improved.
Patent Information
- Application Number
- CN202411834118.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-05-23
AI Technical Summary
MaxScale will cause performance loss when performing read, write and forwarding. Cloud manufacturers need to find ways to reduce performance loss and test performance loss in different scenarios.
The automatic deployment method of MaxScale container based on k8s is adopted to dynamically adjust the location of the MaxScale container by monitoring node performance indicators in real time; a traffic prediction module is embedded in the MaxScale container to predict write request traffic peaks based on timing data analysis and machine learning; an optimized transmission protocol and data encryption algorithm are used to ensure the security of cross-node transmission; a cluster performance is monitored in real time and resource and container scheduling is dynamically adjusted according to load changes.
It significantly improves database write performance, reduces network latency, enhances the scalability and fault tolerance of the system, solves the problem of large performance losses, and optimizes system performance.
Smart Images
Figure CN120029632A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of cloud computing technology, and in particular, relates to a k8s-based MaxScale container automatic deployment method and system. Background Art
[0002] Kubernetes is a container-centric infrastructure that can schedule and run containers on physical clusters or virtual machine clusters. It provides an open source platform for automatic deployment, expansion, and management of containers, and meets some common requirements of applications in production environments: application instance replicas, automatic horizontal expansion, naming and discovery, load balancing, rolling upgrades, resource monitoring, etc.
[0003] MaxScale is a database proxy that forwards database statements to one or more database servers. It extends the high availability, scalability, and security of database servers while simplifying application development by separating them from the underlying database infrastructure. MaxScale is designed to transparently provide load balancing and high availability capabilities to applications. MaxScale has a scalable and flexible architecture with plug-in components that support different protocols and routing methods. When the master library fails, MaxScale can achieve automatic database failover (automatically select one of the slave libraries as the master library, and other slave libraries automatically point to the new master library for replication).
[0004] Although MaxScale has many advantages and is one of the classic solutions used by many cloud vendors as a database proxy, read and write forwarding through MaxScale will inevitably bring a layer of performance loss. How to reduce performance loss and test performance loss in different scenarios are issues that cloud vendors cannot ignore. Summary of the invention
[0005] In view of this, the present application aims to propose a k8s-based MaxScale container automatic deployment method and system to solve the problem of how to use MaxScale to perform read and write forwarding, which leads to large performance loss.
[0006] To achieve the above purpose, the technical solution of this application is implemented as follows:
[0007] In a first aspect, the present application provides a MaxScale container automatic deployment method based on k8s, including:
[0008] Based on the performance indicator data of each node, the location of the MaxScale container is dynamically adjusted by monitoring the load and performance of the cluster nodes in real time;
[0009] A traffic prediction module is embedded in the MaxScale container to predict the next write request traffic peak based on time series data analysis and machine learning.
[0010] Optimized transmission protocols and data encryption algorithms based on container networks to ensure secure transmission of data packets across nodes;
[0011] Monitor cluster performance in real time and dynamically adjust resources and container scheduling based on load changes.
[0012] In the second aspect, based on the same inventive concept, the present application also provides a k8s-based MaxScale container automatic deployment system, including
[0013] The performance monitoring module is configured to dynamically adjust the position of the MaxScale container by monitoring the load and performance of the cluster nodes in real time based on the performance indicator data of each node;
[0014] The traffic prediction module is configured to embed the traffic prediction module inside the MaxScale container and predict the next write request traffic peak based on time series data analysis and machine learning;
[0015] The cross-node network optimization module is configured to optimize the transmission protocol and data encryption algorithm based on the container network to ensure the secure transmission of data packets transmitted across nodes;
[0016] The cluster monitoring and alarm module is configured to monitor cluster performance in real time and dynamically adjust resource and container scheduling according to load changes.
[0017] In a third aspect, based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.
[0018] In a fourth aspect, based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in the first aspect.
[0019] Compared with the prior art, the k8s-based MaxScale container automatic deployment method and system described in this application has the following beneficial effects:
[0020] The present application describes a k8s-based MaxScale container automatic deployment method and system, which can significantly improve database write performance, reduce network latency, and enhance the scalability and fault tolerance of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0022] Figure 1 This is a flow chart of a method for automatic deployment of MaxScale containers based on k8s described in an embodiment of the present application;
[0023] Figure 2 This is a schematic diagram of the structure of a k8s-based MaxScale container automatic deployment system described in an embodiment of the present application;
[0024] Figure 3 This is a schematic diagram of the hardware structure of the electronic device described in the embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0027] The goal of this application is to optimize the performance of MaxScale and MySQL databases deployed in the Kubernetes environment, especially the write performance. The current bottleneck occurs when MaxScale forwards write requests to the MySQL main container, especially when MaxScale and MySQL containers are distributed on different physical machines, network latency significantly affects performance, resulting in large performance loss. To this end, this application aims to improve the processing efficiency of write requests, enhance the scalability of the system, solve the problem of large performance loss, and optimize system performance through intelligent container scheduling, traffic prediction and read-write separation, and cross-node network optimization.
[0028] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0029] See also Figure 1 As shown, this embodiment provides a MaxScale container automatic deployment method based on k8s, which specifically includes the following steps:
[0030] Step S101: According to the performance indicator data of each node monitored in real time, the position of the MaxScale container is dynamically adjusted by monitoring the load and performance of the cluster nodes in real time.
[0031] Specifically, in this embodiment, the position of the MaxScale container is dynamically adjusted by real-time monitoring of the load and performance of the cluster nodes, thereby avoiding performance loss caused by cross-physical machine communication between the MaxScale and MySQL main containers.
[0032] In some implementations, a real-time performance monitoring component is deployed in the Kubernetes cluster to periodically collect performance indicator data of each node, wherein the performance indicator data includes at least CPU usage, memory usage, network latency, and number of database connections;
[0033] The load of the current node is determined based on the performance indicator data, and the traffic forwarding path of the MaxScale container is dynamically adjusted based on the load balancer pre-configured in k8s to dynamically adjust the location of the MaxScale container.
[0034] Specifically, in this embodiment, based on the deployment of Kubernetes native monitoring tools (such as Prometheus) and a customized performance collection module, indicators such as CPU, memory, disk IO, network latency, etc. are monitored on each node. The performance data of each node is regularly reported to the cluster monitoring system in the Kubernetes cluster, and the load of the node is evaluated through these data.
[0035] Integrate a k8s-based load balancing tool to support traffic scheduling between the MaxScale container and the MySQL main container. According to the real-time performance indicators of the node, the load balancer can dynamically adjust the traffic forwarding path of the MaxScale container and then select the most suitable location for deploying the MaxScale container.
[0036] Step S102: embed a traffic prediction module in the MaxScale container to predict the peak of write request traffic in the next phase based on time series data analysis and machine learning.
[0037] Specifically, in this embodiment, in the traditional database architecture, read-write separation usually relies on traffic distribution at the database level, but this approach may not be able to fully meet the performance requirements in high-load scenarios, especially in the case of high concurrency of write requests. The synchronization between the database master and slave and the transfer of the MaxScale container will become bottlenecks.
[0038] This application can optimize the pressure on the database by intelligently separating read requests and write requests. In this solution, optimization is performed for high-concurrency write requests. Before the write requests arrive, a pre-judgment is performed. If the number of write requests is large, the location of the MaxScale container is dynamically adjusted to avoid network delays across physical machines.
[0039] In some implementations, a traffic prediction module is embedded in the MaxScale container, so the traffic prediction module analyzes the fluctuation trend of traffic based on the historical write request volume of the database and the time series, and predicts the load situation in the next stage;
[0040] The MaxScale container adopts a write request buffering strategy during peak hours to buffer write requests;
[0041] Read requests and write requests are processed separately. Write requests are forwarded to the MySQL master database through the MaxScale container, and read requests are diverted to the MySQL backup database based on the load situation.
[0042] Specifically, in this embodiment, a traffic prediction module is added to the MaxScale container. The module collects historical request data, analyzes the trend and peak hours of write requests (such as the peak hours of each day), and predicts the write request traffic within a certain period of time in the future.
[0043] In this embodiment, traffic prediction uses time series data analysis and machine learning algorithms (such as regression analysis, time series prediction, etc. The above algorithms are mature algorithms, which are not improved in this embodiment and will not be further described here) to identify traffic fluctuation patterns and predict future loads.
[0044] The prediction module analyzes the traffic fluctuation trend based on the historical write request volume of the database and the time series (such as hourly and daily) to predict the load situation in the short term. The module will analyze the traffic data of the past few days and calculate the traffic change trend in the short term.
[0045] Based on the predicted traffic data, the MaxScale container can prioritize writing requests to the MySQL master container with a lower load when the volume of write requests is large. At this point, the system can schedule the MaxScale container to the physical machine where the master is located as needed to reduce latency across physical machines. To further improve write performance, the MaxScale container can use a write request cache strategy to buffer write requests when the load is too high, reduce the pressure of instantaneous writes, and gradually forward requests to the MySQL master.
[0046] Write requests are forwarded to the MySQL master database through MaxScale, and read requests can be diverted to the MySQL backup database based on the load situation. MaxScale can intelligently identify the database load and automatically adjust the routing of read and write traffic, thereby reducing the burden on the master database.
[0047] Step S103: Optimize the transmission protocol and data encryption algorithm based on the container network to ensure the secure transmission of data packets transmitted across nodes.
[0048] The purpose of this step is to improve the efficiency of cross-physical machine communication, ensure data security during the communication process, and reduce performance loss caused by network latency.
[0049] In some implementations, a network plug-in that supports the QUIC protocol is integrated into the container network, and a lightweight encryption algorithm is used to encrypt data during communication between containers.
[0050] Specifically, in this embodiment, by integrating a network plug-in that supports the QUIC protocol in the container network, the QUIC protocol has lower latency and higher transmission efficiency in cross-node communication compared to the traditional TCP protocol. QUIC can reduce handshake delays when establishing a connection and support mechanisms such as fast retransmission and flow control.
[0051] In addition, the QUIC protocol can also optimize the use of bandwidth, so that network traffic across physical machines can be transmitted more efficiently, especially improving throughput under high load conditions.
[0052] When communicating between containers, a lightweight encryption algorithm (such as an encryption scheme based on AES, which is a mature scheme and will not be described in detail here) is used to encrypt data to ensure security. The encryption algorithm should consider hardware acceleration (such as AES-NI) to reduce performance loss during the encryption process. For data packets transmitted across nodes, ensure secure transmission through the QUIC protocol instead of relying on the traditional TCP+TLS method, further reducing the impact of encryption operations on performance.
[0053] Step S104: monitor cluster performance in real time, and dynamically adjust resource and container scheduling according to load changes.
[0054] The purpose of this step is to continuously monitor performance data during system operation and dynamically optimize according to load conditions to ensure that the system is always efficient and stable.
[0055] In some implementations, the performance data of all nodes, Pods, and containers in the Kubernetes cluster are monitored in real time through a monitoring tool, alarms are automatically triggered according to set thresholds, and container scheduling is adjusted.
[0056] Specifically, in this embodiment, monitoring tools such as Prometheus and Grafana are deployed to monitor the health status of each node and container in the Kubernetes cluster in real time, especially the performance indicators of the MaxScale container and the MySQL main library (such as latency, throughput, number of connections, etc.).
[0057] At the same time, set a threshold alarm. When the node resource utilization is close to 100% or the network delay is abnormal, the alarm is automatically triggered and the container scheduling is triggered.
[0058] The above method can significantly improve database write performance, reduce network latency, and enhance the scalability and fault tolerance of the system.
[0059] After actual operation tests, this method can improve performance by 35%. Before optimization, it takes 91.5643 seconds to run 100 tables with 100,000 rows of data per table through sysbench, and the TPS is 109.19 / second; after optimization, it takes 59.3123 seconds to run the same data, and the TPS is 168.55 / second.
[0060] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0061] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the embodiment of the present application also provides a MaxScale container automatic deployment system based on k8s.
[0062] like Figure 2 As shown, the k8s-based MaxScale container automatic deployment system includes:
[0063] The performance monitoring module is configured to dynamically adjust the position of the MaxScale container by monitoring the load and performance of the cluster nodes in real time based on the performance indicator data of each node;
[0064] The traffic prediction module is configured to embed the traffic prediction module inside the MaxScale container and predict the next write request traffic peak based on time series data analysis and machine learning;
[0065] The cross-node network optimization module is configured to optimize the transmission protocol and data encryption algorithm based on the container network to ensure the secure transmission of data packets transmitted across nodes;
[0066] The cluster monitoring and alarm module is configured to monitor cluster performance in real time and dynamically adjust resource and container scheduling according to load changes.
[0067] For the convenience of description, the above system is described by dividing the functions into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0068] The system of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0069] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.
[0070] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0071] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0072] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0073] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0074] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0075] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).
[0076] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0077] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0078] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.
[0079] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0080] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0081] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0082] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.
[0083] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0084] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.
Claims
1. A k8s-based MaxScale container automatic deployment method, characterized in that: include: Based on the performance indicator data of each node, the location of the MaxScale container is dynamically adjusted by monitoring the load and performance of the cluster nodes in real time; A traffic prediction module is embedded in the MaxScale container to predict the next write request traffic peak based on time series data analysis and machine learning. Optimized transmission protocols and data encryption algorithms based on container networks to ensure secure transmission of data packets across nodes; Monitor cluster performance in real time and dynamically adjust resources and container scheduling based on load changes.
2. The method according to claim 1, characterized in that According to the performance indicator data of each node monitored in real time, the position of the MaxScale container is dynamically adjusted by monitoring the load and performance of the cluster nodes in real time, including: By deploying real-time performance monitoring components in the Kubernetes cluster, performance indicator data of each node can be collected regularly; The load condition of the current node is determined according to the performance indicator data, and the traffic forwarding path of the MaxScale container is dynamically adjusted based on the load balancer pre-configured in k8s to dynamically adjust the position of the MaxScale container.
3. The method according to claim 2, characterized in that: in, The performance indicator data at least includes CPU usage, memory usage, network latency, and number of database connections.
4. The method according to claim 1, characterized in that: The traffic prediction module is embedded in the MaxScale container to collect historical request data of the database and predict the fluctuation of write request traffic in the next stage, including: The traffic prediction module is embedded in the MaxScale container. The traffic prediction module analyzes the fluctuation trend of traffic based on the historical write request volume of the database and the time series, and predicts the load situation in the next period of time. The MaxScale container adopts a write request buffering strategy during peak hours to buffer write requests; Read requests and write requests are processed separately. Write requests are forwarded to the MySQL master database through the MaxScale container, and read requests are diverted to the MySQL backup database based on the load situation.
5. The method according to claim 1, characterized in that The optimized transmission protocol and data encryption algorithm based on the container network to ensure the secure transmission of data packets transmitted across nodes include: Integrate a network plug-in that supports the QUIC protocol into the container network, and use a lightweight encryption algorithm to encrypt data when communicating between containers.
6. The method according to claim 1, characterized in that The real-time monitoring of cluster performance and dynamic adjustment of resources and container scheduling based on load changes include: The monitoring tool can be used to monitor the performance data of all nodes, Pods, and containers in the Kubernetes cluster in real time, automatically trigger alarms based on set thresholds, and adjust container scheduling.
7. A MaxScale container automatic deployment system based on k8s, characterized in that: include The performance monitoring module is configured to dynamically adjust the position of the MaxScale container by monitoring the load and performance of the cluster nodes in real time based on the performance indicator data of each node; The traffic prediction module is configured to embed the traffic prediction module inside the MaxScale container and predict the next write request traffic peak based on time series data analysis and machine learning; The cross-node network optimization module is configured to optimize the transmission protocol and data encryption algorithm based on the container network to ensure the secure transmission of data packets transmitted across nodes; The cluster monitoring and alarm module is configured to monitor cluster performance in real time and dynamically adjust resource and container scheduling according to load changes.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium, characterized in that: in, The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.