Request allocation method, system and device and storage medium

Abstract

The invention discloses a request allocation method, system and device and a storage medium. The method comprises the following steps: acquiring performance data of a node; predicting a load trend according to historical load data and the performance data, and pre-distributing a request according to the load trend; and according to the load trend and the performance data, distributing the request to a ClickHouse node based on a routing algorithm of weight. According to the embodiment of the invention, real-time dynamic allocation of the system can be realized based on the historical load data, the performance data and the routing algorithm based on the weight, and the problem of frequent faults of the system can be relieved. The method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a request allocation method, system, device and storage medium. Background Art

[0002] With the continuous development of big data technology, data storage and processing technology has gradually transitioned from traditional relational databases to large-scale distributed database systems. Among them, ClickHouse, as a high-performance columnar database system, is widely used in scenarios that require efficient data analysis and processing, such as network traffic monitoring of operators, real-time user behavior analysis, etc. However, in large-scale data processing scenarios, the concurrency and real-time requirements of data processing are extremely high, and the system load will vary greatly with traffic fluctuations; if there is no appropriate load balancing and intelligent routing mechanism, the system will face problems such as response delays and frequent failures.

[0003] Existing load balancing systems usually adopt static strategies, which cannot perceive changes in server load in real time and are difficult to cope with the sudden and continuous growth of large-scale data traffic. For example, some systems rely on traditional load balancing algorithms such as polling and minimum number of connections, which cannot make intelligent adjustments based on the real-time health status and load conditions of the nodes. They cannot solve the problem of frequent system failures. Summary of the invention

[0004] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0005] To this end, an object of the present invention is to provide an efficient request allocation method, system, device and storage medium.

[0006] In order to achieve the above technical purpose, one aspect of an embodiment of the present invention provides a request allocation method, comprising the following steps: obtaining performance data of a node; predicting a load trend based on historical load data and the performance data, and pre-allocating requests based on the load trend; and allocating the requests to ClickHouse nodes based on a weighted routing algorithm based on the load trend and the performance data. The embodiment of the present application can realize real-time dynamic allocation of the system based on historical load data and performance data and a weighted routing algorithm, which is conducive to alleviating the problem of frequent failures of the system.

[0007] In some embodiments, the request allocation method of an embodiment of the present invention, the obtaining of performance data of a node includes:

[0008] Determine the performance and corresponding query execution information through the ClickHouse system table;

[0009] Pull data through the Prometheus monitoring tool.

[0010] In some embodiments, in one embodiment of the present invention, predicting the load trend according to the historical load data and the performance data includes:

[0011] The historical load data and the performance data are input into a load trend prediction model to obtain a predicted load trend; the load trend prediction model is an LSTM model or an ARIMA model.

[0012] In some embodiments, in one embodiment of the present invention, the weight-based routing algorithm according to the load trend and the performance data allocates the request to the ClickHouse node, including:

[0013] Determine the weight corresponding to each parameter in the performance data;

[0014] Determining a dynamic weight of each node according to the performance data and the weight;

[0015] Assign the request to the ClickHouse node with the lowest dynamic weight.

[0016] In some embodiments, in one embodiment of the present invention, the method further comprises:

[0017] Monitor the health status of each node through a heartbeat detection mechanism, and determine whether the node is a faulty node based on the query response time and health check results of the node;

[0018] If the master node goes down, the request will be routed to the replica node through ClickHouse's replication mechanism to continue execution.

[0019] In some embodiments, in one embodiment of the present invention, the method further comprises:

[0020] The request enters the ClickHouse distributed table engine and waits for routing distribution.

[0021] In some embodiments, in one embodiment of the present invention, the method further comprises:

[0022] Obtaining a first dynamic weight of the first node at a first moment;

[0023] Obtain a second dynamic weight of the first node at a second moment; the second moment is later than the first moment;

[0024] If the difference between the second dynamic weight and the first dynamic weight is greater than or equal to a preset threshold, the request is allocated to a second node other than the first node.

[0025] On the other hand, an embodiment of the present invention provides a request allocation system, including:

[0026] The first module is used to obtain the performance data of the node;

[0027] A second module is used to predict the load trend according to the historical load data and the performance data, and pre-allocate the requests according to the load trend;

[0028] The third module is used to distribute the requests to ClickHouse nodes based on the load trend and the performance data and a weight-based routing algorithm.

[0029] On the other hand, an embodiment of the present invention provides a request allocation device, including:

[0030] at least one processor;

[0031] at least one memory for storing at least one program;

[0032] When the at least one program is executed by the at least one processor, the at least one processor is enabled to implement the above-mentioned request allocation method.

[0033] On the other hand, an embodiment of the present invention provides a storage medium storing a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned request allocation method.

[0034] The method provided by the embodiment of the present application includes: obtaining performance data of a node; predicting a load trend based on historical load data and the performance data, and pre-allocating requests based on the load trend; and allocating the requests to ClickHouse nodes based on a weighted routing algorithm based on the load trend and the performance data. The embodiment of the present application can realize real-time dynamic allocation of the system based on historical load data and performance data and a weighted routing algorithm, which is conducive to alleviating the problem of frequent system failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A schematic diagram of a flow chart of an embodiment of a request allocation method provided by the present invention;

[0037] Figure 2 A flowchart of another embodiment of the request allocation method provided by the present invention;

[0038] Figure 3 A schematic diagram of the structure of an embodiment of the request allocation system provided by the present invention;

[0039] Figure 4 A schematic diagram of the structure of an embodiment of the request allocation device provided by the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0041] First, the terms provided in the embodiments of the present application are explained:

[0042] ClickHouse: A column-based database management system for online analytical processing (OLAP).

[0043] Dynamic load prediction: Based on real-time and historical data, machine learning algorithms are used to predict server load and allocate resources in advance.

[0044] Intelligent routing: intelligently distributes query requests to the most appropriate node based on the node's load and health status.

[0045] Fault detection and switching mechanism: Automatically monitor node status and quickly switch requests to healthy nodes when a node fails.

[0046] With the continuous development of big data technology, data storage and processing technology has gradually transitioned from traditional relational databases to large-scale distributed database systems. Among them, ClickHouse, as a high-performance column-based database system, is widely used in scenarios that require efficient data analysis and processing, such as network traffic monitoring of operators and real-time user behavior analysis.

[0047] However, in large-scale data processing scenarios, the concurrency and real-time requirements of data processing are extremely high, and the system load will vary greatly with traffic fluctuations; without appropriate load balancing and intelligent routing mechanisms, the system will face problems such as response delays and frequent failures.

[0048] Existing load balancing systems usually adopt static strategies, which cannot perceive the changes in server load in real time and are difficult to cope with the sudden and continuous growth of large-scale data traffic. For example, some systems rely on traditional load balancing algorithms such as Round Robin and minimum number of connections, which cannot make intelligent adjustments based on the real-time health status and load conditions of the nodes.

[0049] In addition, although some systems have fault detection mechanisms, most of them are based on timing detection or heartbeat mechanisms and cannot respond quickly to sudden failures.

[0050] For example, Haproxy is one of the representatives of load balancing systems. Although it supports health checks and fault switching, its load balancing mechanism is relatively simple. It cannot predict future loads based on historical data, nor can it optimize request distribution through machine learning.

[0051] 1) Static load balancing: Most existing technologies rely on static load balancing strategies and cannot dynamically adjust request allocation according to real-time load conditions.

[0052] 2) Unable to predict future load: The lack of machine learning models based on historical data makes it difficult to predict traffic peaks in advance.

[0053] 3) Fault switching delay: The existing fault detection mechanism has a slow response speed and the switching process takes a long time.

[0054] 4) Low resource utilization: Load distribution is unbalanced, some nodes may be overloaded while other nodes are idle, resulting in low overall resource utilization.

[0055] The present invention aims to overcome the above-mentioned defects in the prior art by designing a ClickHouse request distribution system based on dynamic load prediction and intelligent routing, and achieve:

[0056] 1) Monitor node load in real time and predict future load conditions through machine learning.

[0057] 2) Intelligently adjust request allocation based on the health status and load conditions of the nodes to improve system resource utilization.

[0058] 3) Realize automatic fault detection and rapid switching to ensure high availability and stability of the system.

[0059] 4) Provide a more accurate and stable request allocation strategy to meet the needs of large-scale traffic monitoring.

[0060] The request allocation method and system proposed in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. First, the request allocation method proposed in the embodiments of the present invention will be described with reference to the accompanying drawings.

[0061] Reference Figure 1, a request allocation method is provided in an embodiment of the present invention. The request allocation method in the embodiment of the present invention can be applied to a terminal, can be applied to a server, can also be software running in a terminal or a server, etc. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The request allocation method in the embodiment of the present invention mainly includes the following steps:

[0062] S100: Obtain performance data of the node;

[0063] S200: predicting a load trend according to historical load data and the performance data, and pre-allocating requests according to the load trend;

[0064] S300: According to the load trend and the performance data, the request is allocated to the ClickHouse node based on a weighted routing algorithm.

[0065] In some possible implementations, the present invention achieves efficient and intelligent allocation of ClickHouse requests through dynamic load balancing algorithms, load prediction models based on machine learning, intelligent routing, and automatic fault detection and switching mechanisms, which is particularly suitable for real-time monitoring and analysis of large-scale network traffic in operator scenarios.

[0066] The system provided in the embodiment of the present application is composed of the following four modules, which respectively undertake the functions of real-time monitoring, load prediction, intelligent request allocation and fault handling:

[0067] Monitoring module: Real-time monitoring of key performance indicators of nodes, including CPU usage, memory usage, network traffic, disk I / O, etc., to provide accurate input data for load prediction.

[0068] Load forecasting module: Use machine learning algorithms such as LSTM (Long Short-Term Memory Network) or ARIMA (Autoregressive Integrated Moving Average Model) to predict future load peak time periods, especially for high concurrency of operator services, such as peak traffic periods and large-scale user Internet behavior analysis.

[0069] Routing module: Dynamically adjust the request allocation strategy based on load prediction and real-time monitoring results, use a weight-based intelligent routing algorithm to send requests to healthy nodes with low loads, and give priority to nodes with sufficient resources to handle critical tasks.

[0070] Fault detection and switching module: Automatically monitor the health status of each node and detect node failures in a timely manner. Combined with the **Service Level Agreement (SLA)** requirements of the operator's business, the system can switch requests to backup nodes within milliseconds to ensure uninterrupted services.

[0071] The present invention realizes intelligent allocation of ClickHouse query requests through ClickHouse's efficient distributed architecture and query processing capabilities, combined with dynamic load balancing, machine learning load prediction, intelligent routing, and automatic fault detection switching. The following detailed description of each module emphasizes the role of ClickHouse in each link.

[0072] The core role of ClickHouse in this application embodiment:

[0073] Under the technical framework of the present invention, ClickHouse is used as the core database of the system through its distributed table engine, parallel query execution, data replication mechanism and other features. In load balancing, query distribution, data query high availability and intelligent routing, ClickHouse directly participates in the following aspects:

[0074] 1) Distributed query execution: ClickHouse’s distributed table engine can split large queries into multiple subqueries and process them in parallel on multiple nodes. Intelligent routing determines how query requests are distributed to each node.

[0075] 2) Data replication: In a cluster environment, ClickHouse ensures high availability through data replication. The fault detection mechanism ensures that when a node goes down, it can automatically switch to other healthy replica nodes.

[0076] 3) Monitoring and log analysis: ClickHouse records query execution logs through system tables such as system.query_log, and provides historical load data for use by the load prediction module.

[0077] The architecture mainly includes monitoring module, load prediction module, intelligent routing module and fault detection and switching module. These four modules are closely integrated with ClickHouse's query processing mechanism. The working process of these modules is described in detail below.

[0078] Optionally, in one embodiment of the present invention, the acquiring the performance data of the node includes:

[0079] Determine the performance and corresponding query execution information through the ClickHouse system table;

[0080] Pull data through the Prometheus monitoring tool.

[0081] In some possible implementations, the role of the monitoring module is to collect the performance indicators and query load of each ClickHouse node in real time as the basic data for load prediction and intelligent routing.

[0082] Specific implementation:

[0083] 1. ClickHouse system tables such as system.query_log and system.metrics provide detailed performance and query execution information.

[0084] 2. Use the Prometheus monitoring tool to regularly pull data, covering core indicators such as CPU usage, memory usage, query response time, and network traffic.

[0085] 3. Monitoring points: including the average execution time of queries on each node, the amount of data transferred between nodes, the number of I / O operations, etc.

[0086] 4. Implement SQL: Query the ClickHouse system table to obtain load information:

[0087] SELECT

[0088] query_duration_ms,

[0089] memory_usage,

[0090] read_rows,

[0091] written_rows

[0092] FROM system.query_log

[0093] WHERE event_time>=now()-INTERVAL 1MINUTE.

[0094] Through the system table, the monitoring module can provide the load prediction module with real-time and high-precision query load data to ensure the accuracy of the prediction.

[0095] Optionally, in one embodiment of the present invention, predicting the load trend according to the historical load data and the performance data includes:

[0096] The historical load data and the performance data are input into a load trend prediction model to obtain a predicted load trend; the load trend prediction model is an LSTM model or an ARIMA model.

[0097] In some possible implementations, the role of the load prediction module is to predict future load trends by analyzing historical load data and current system monitoring data, thereby helping the intelligent routing module to pre-allocate resources.

[0098] Specific implementation:

[0099] 1. Load prediction data source: mainly relies on ClickHouse query log (system.query_log) and monitoring data (system.metrics).

[0100] 2. Prediction algorithm: Use LSTM (Long Short-Term Memory Network) or ARIMA model to predict future load, especially for query execution time and data read and write load trend prediction, to help perceive traffic peaks in advance.

[0101] 3. Specific implementation of Python code:

[0102] Use the ARIMA model to predict query execution load:

[0103] from statsmodels.tsa.arima.model import ARIMA

[0104] import pandas as pd

[0105] #Extract historical query execution time data from the ClickHouse system

[0106] data=pd.read_sql("SELECT query_duration_ms,event_time FROMsystem.query_log WHERE event_time>=now()-INTERVAL 1DAY",connection)

[0107] #Use ARIMA model for prediction

[0108] model=ARIMA(data['query_duration_ms'],order=(5,1,0))

[0109] model_fit = model.fit()

[0110] predictions = model_fit.forecast(steps = 10) # Predict the execution time of the next 10 queries

[0111] This module can help the system schedule resources in advance, especially optimizing query allocation strategies during peak hours (such as the operator's traffic peak period).

[0112] Optionally, in one embodiment of the present invention, allocating the request to the ClickHouse node based on a weighted routing algorithm according to the load trend and the performance data includes:

[0113] Determine the weight corresponding to each parameter in the performance data;

[0114] Determining a dynamic weight of each node according to the performance data and the weight;

[0115] Assign the request to the ClickHouse node with the lowest dynamic weight.

[0116] In some possible implementations, the role of the intelligent routing module is to dynamically allocate query requests to the most appropriate ClickHouse node by combining the real-time data of the monitoring module with the load prediction results.

[0117] Specific implementation:

[0118] 1. Weight-based routing algorithm:

[0119] The weight is calculated by weighting parameters such as CPU, memory, network I / O, and query execution time. The node with the lowest weight is given priority to process queries.

[0120] 2. Routing strategy of ClickHouse distributed table engine:

[0121] The Distributed engine routes queries based on node load. Unlike the default random or round-robin mechanism, the routing module dynamically selects the node with the lightest load.

[0122] 3. Implement sql and python code:

[0123] SQL:

[0124] Query routing using ClickHouse's least_loaded_shard strategy

[0125] CREATE TABLE distributed_table AS local_table

[0126] ENGINE=Distributed(cluster_name,database_name,table_name,'least_loaded_shard','random');

[0127] Python code:

[0128] Specific routing logic: The weight of each node is calculated by the following formula:

[0129] def calculate_weight(cpu_usage,memory_usage,network_io,query_duration):

[0130] #Weighted calculation of node load

[0131] return 0.4*cpu_usage+0.3*memory_usage+0.2*network_io+0.1*query_duration

[0132] The routing algorithm is adjusted in real time to distribute query requests to the optimal node based on the node's load status and prediction results, thereby maximizing the overall query performance of ClickHouse.

[0133] Optionally, in one embodiment of the present invention, the method further includes:

[0134] Monitor the health status of each node through a heartbeat detection mechanism, and determine whether the node is a faulty node based on the query response time and health check results of the node;

[0135] If the master node goes down, the request will be routed to the replica node through ClickHouse's replication mechanism to continue execution.

[0136] In some possible implementations, the role of the fault detection and switching module is: when a ClickHouse node fails, the system can detect and switch to other healthy nodes in time to ensure the high availability of the query service.

[0137] Specific implementation:

[0138] 1. Fault detection mechanism:

[0139] Use the heartbeat detection mechanism to monitor the health status of each node, and determine whether the node is operating normally by regularly checking the node's query response time, health check results, etc.

[0140] 2. Copy switching:

[0141] ClickHouse's replication mechanism allows query requests to be routed to replica nodes for continued execution when the primary node goes down. Replica management is implemented through the ReplicatedMergeTree engine.

[0142] 3. Implement SQL:

[0143] Use the ReplicatedMergeTree engine to ensure data replica availability.

[0144] CREATE TABLE replicated_table(

[0145] event_time DateTime,

[0146] user_id UInt32,

[0147] action String

[0148] )ENGINE=ReplicatedMergeTree(' / clickhouse / tables / {shard} / replicated_table','{replica}')

[0149] PARTITION BY toYYYYMM(event_time)

[0150] ORDER BY(user_id,event_time);

[0151] Failure switching logic: When a node fails, the system automatically switches to the replica node to ensure uninterrupted service and meet high availability requirements.

[0152] Optionally, in one embodiment of the present invention, the method further includes:

[0153] The request enters the ClickHouse distributed table engine and waits for routing distribution.

[0154] Optionally, in one embodiment of the present invention, the method further includes:

[0155] Obtaining a first dynamic weight of the first node at a first moment;

[0156] Obtain a second dynamic weight of the first node at a second moment; the second moment is later than the first moment;

[0157] If the difference between the second dynamic weight and the first dynamic weight is greater than or equal to a preset threshold, the request is allocated to a second node other than the first node.

[0158] In some possible implementations, the preset threshold in the embodiments of the present application is an acceptable dynamic weight mutation amount of the requesting node.

[0159] The query processing flow in the ClickHouse cluster:

[0160] 1) The user initiates a query request;

[0161] The query request first enters the ClickHouse distributed table engine and waits for routing distribution.

[0162] 2) Monitoring and prediction;

[0163] The monitoring module collects the load status of each node in real time, while the load prediction module predicts future query load trends.

[0164] 3) Intelligent routing;

[0165] Based on the load forecast and current node status, the intelligent routing module selects the most appropriate ClickHouse node to process the query.

[0166] 4) Fault detection and switching;

[0167] When a node fails, the system quickly detects and switches to other replica nodes.

[0168] The business relationship of each module is as follows: The monitoring module provides real-time data for load forecasting to ensure the accuracy of the forecast. The load forecasting module predicts future loads and assists intelligent routing in allocating query requests in advance. The intelligent routing module allocates query requests to the optimal node based on the results of load forecasting and real-time monitoring. The fault detection and switching module ensures that services are not interrupted when a node fails, maintaining high system availability.

[0169] Through the optimization of the above technical solutions, ClickHouse not only serves as the core database in the entire process, but also plays a key role in load balancing, query scheduling and high availability assurance.

[0170] Refer to the following Figure 2 , the request allocation method provided in the embodiment of the present application is described in detail:

[0171] 1) User request entry;

[0172] Users initiate query requests through HTTP, SQL, etc., and the system forwards them to the query scheduling center.

[0173] 2) Monitoring module;

[0174] This module monitors the load of ClickHouse nodes in real time, including CPU usage, memory usage, query response time, etc.

[0175] Use tools such as Prometheus to obtain monitoring data and extract query logs from the system.query_log table.

[0176] 3) Data analysis and cleaning module;

[0177] Process the data obtained from the monitoring module, perform operations such as standardization, missing value processing, and outlier detection, and provide clean data input for the load prediction module.

[0178] 4) Load prediction module;

[0179] Use machine learning algorithms such as LSTM or ARIMA, combined with historical query logs and real-time monitoring data, to predict future load.

[0180] The prediction results will be fed back to the intelligent routing module to help adjust the routing strategy in advance.

[0181] 5) Intelligent routing module;

[0182] Based on load forecasts and real-time monitoring data, the intelligent routing module dynamically adjusts the distribution of query requests and uses ClickHouse's least_loaded_shard strategy to assign query requests to the nodes with the lightest load.

[0183] The routing module dynamically selects the node with the lightest load. The calculation logic / principle of each node:

[0184] In the routing module of the ClickHouse cluster, in order to achieve efficient allocation of query requests, the routing module adopts a dynamic multi-dimensional load evaluation model to calculate the load of each node in real time, and allocate query requests to the node with the lightest load based on these calculation results. The model combines multiple performance indicators, such as CPU usage, memory usage, network I / O, and query execution time, and determines the load score of each node by weighted calculation of these indicators, thereby achieving intelligent scheduling.

[0185] 1. Load indicator selection and interpretation;

[0186] In order to comprehensively evaluate the load of each node, the routing module selects the following core performance indicators for real-time monitoring and calculation:

[0187] CPU usage (cpu_usage): reflects the processing capacity usage of the node. Nodes with high CPU usage are usually executing more or more complex tasks, so their responsiveness is relatively low.

[0188] Memory usage (memory_usage): Memory usage directly affects the data access efficiency during query execution. Nodes with high memory usage may face insufficient memory, resulting in performance degradation.

[0189] Network I / O load (network_io): This metric is used to measure the efficiency of data transmission between nodes and between nodes and clients. If the network I / O load is high, the bottleneck of data transmission may affect the response speed of queries.

[0190] Query execution time (query_duration): Query execution time reflects the historical performance of each node in processing query requests. The longer the execution time, the greater the current load of the node and the weaker the response capability.

[0191] 2. Dynamic load weight calculation model;

[0192] The routing module integrates the above multiple load indicators through a weighted calculation formula and calculates a comprehensive load score for each node. The formula assigns weights according to the importance of different indicators, and each weight can be adjusted according to actual conditions.

[0193] Node Load Weight = 0.4*cpu u sage+0.3*memory u sage+0.2*network i o+0.1*query d uration

[0194] CPU usage (40%) and memory usage (30%) account for a large proportion because they have the greatest impact on the node's computing and data processing capabilities.

[0195] Network I / O (20%) reflects whether the network bandwidth becomes a bottleneck, especially in distributed systems, which affects the transmission efficiency of query requests.

[0196] The query execution time (10%) is used as an auxiliary indicator to reflect the overall performance of the node when processing queries.

[0197] Through this weighted method, the routing module can accurately calculate the load score of each node, ensuring that the real-time load conditions of each node are given priority during selection.

[0198] 3. Dynamic selection and scheduling;

[0199] In actual operation, the routing module makes dynamic choices based on the load weight of each node:

[0200] I. Real-time load data acquisition: Get the latest performance data of each node from ClickHouse's system tables (such as system.metrics), including CPU, memory, network I / O, and query execution time.

[0201] II. Load weight calculation: Based on the acquired performance data, the routing module calculates the load score for each node using the above weighted formula.

[0202] III. Select the node with the lightest load: The routing module compares the load weights of all nodes and selects the node with the smallest weight to process the newly arrived query request. This ensures that the resources in the system are optimally utilized and avoids single point overload or resource waste.

[0203] IV. Continuous monitoring and adjustment: The system will continuously monitor the load of each node and dynamically adjust the query request allocation strategy. If the load weight of a node suddenly increases, the system will allocate subsequent requests to other nodes with lighter loads, thereby achieving global optimization.

[0204] 4. Advantages of real-time load scheduling:

[0205] Compared with traditional polling or static load balancing, the dynamic selection mechanism of the routing module based on load weight has the following advantages:

[0206] Strong adaptability: The load calculation model is updated in real time, which can quickly respond to load fluctuations and adapt to large-scale concurrent query scenarios.

[0207] Avoid single-point bottlenecks: By continuously monitoring the performance indicators of each node, the system can automatically schedule when the node load is too high to prevent a certain node from becoming a bottleneck of the system.

[0208] Maximized resource utilization: Query requests are always assigned to the node with the lightest load, which ensures that system resources are maximized and improves overall query processing efficiency.

[0209] By calculating the load weight and dynamically selecting the route, the routing module of the present invention can ensure the efficient operation of the system in a complex distributed environment, avoid the single point overload problem, and improve the overall resource utilization.

[0210] 6) Query the dispatch center;

[0211] The scheduling center receives query requests and divides them into pieces, distributing them to multiple ClickHouse nodes for execution to ensure load balancing and efficient query execution.

[0212] 7) Fault detection and switching module;

[0213] Continuously monitor the health status of each node. If a node failure is detected, it will automatically switch to a healthy replica node to ensure high availability of the system.

[0214] 8) Request execution feedback module;

[0215] After the query is executed, the system will organize the results and return them to the user, closing the entire query process.

[0216] Through detailed module design and clear logical flow, this architecture implements intelligent load prediction, request routing, and fault switching mechanisms under the ClickHouse system, ensuring efficient, stable, and intelligent query allocation management.

[0217] The design of the dynamic load balancing algorithm in the embodiment of the present application can dynamically adjust the request allocation strategy based on historical and real-time data. The load prediction model based on machine learning can effectively predict future traffic peaks. Automatic fault detection and fast switching mechanism ensure high system availability. The intelligent routing module can perform real-time optimization based on load and health status.

[0218] Dynamic load prediction and intelligent routing: Compared with traditional static load balancing methods, this invention predicts future loads through machine learning to achieve more intelligent request distribution. Fast fault detection and switching: Through real-time health monitoring and automatic fault switching, the system's fault tolerance is improved and the service interruption time is reduced.

[0219] In summary, the above method includes: obtaining performance data of the node; predicting the load trend according to the historical load data and the performance data, and pre-allocating the request according to the load trend; and allocating the request to the ClickHouse node based on the weighted routing algorithm according to the load trend and the performance data. The embodiment of the present application can realize real-time dynamic allocation of the system based on historical load data and performance data and a weighted routing algorithm, which is conducive to alleviating the frequent failure problem of the system.

[0220] Secondly, refer to the attached Figure 3 A request allocation system proposed according to an embodiment of the present invention is described.

[0221] Figure 3 1 is a schematic diagram of the structure of a request allocation system according to an embodiment of the present invention, wherein the system specifically comprises:

[0222] The first module 310 is used to obtain performance data of the node;

[0223] A second module 320, configured to predict a load trend based on historical load data and the performance data, and pre-allocate requests based on the load trend;

[0224] The third module 330 is used to distribute the request to the ClickHouse node based on the weight-based routing algorithm according to the load trend and the performance data.

[0225] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0226] Reference Figure 4 , an embodiment of the present invention provides a request allocation device, comprising:

[0227] at least one processor 410;

[0228] At least one memory 420, used to store at least one program;

[0229] When the at least one program is executed by the at least one processor 410, the at least one processor 410 implements the request allocation method.

[0230] Similarly, the contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0231] An embodiment of the present invention further provides a computer-readable storage medium, in which a program executable by a processor is stored. The program executable by the processor is used to execute the above-mentioned request allocation method when executed by the processor.

[0232] Similarly, the contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0233] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0234] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0235] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0236] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0237] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0238] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0239] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0240] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0241] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A request allocation method, characterized in that: The following steps are involved: Get the performance data of the node; Predicting a load trend based on historical load data and the performance data, and pre-allocating requests based on the load trend; According to the load trend and the performance data, the requests are distributed to ClickHouse nodes based on a weighted routing algorithm.

2. The request allocation method according to claim 1, characterized in that: The obtaining of the performance data of the node includes: Determine the performance and corresponding query execution information through the ClickHouse system table; Pull data through the Prometheus monitoring tool.

3. The request allocation method according to claim 2, characterized in that: The predicting of load trend according to the historical load data and the performance data includes: The historical load data and the performance data are input into a load trend prediction model to obtain a predicted load trend; the load trend prediction model is an LSTM model or an ARIMA model.

4. The request allocation method according to claim 1, characterized in that: The weight-based routing algorithm according to the load trend and the performance data distributes the request to the ClickHouse node, including: Determine the weight corresponding to each parameter in the performance data; Determining a dynamic weight of each node according to the performance data and the weight; Assign the request to the ClickHouse node with the lowest dynamic weight.

5. The request allocation method according to claim 1, characterized in that: The method further comprises: Monitor the health status of each node through a heartbeat detection mechanism, and determine whether the node is a faulty node based on the query response time and health check results of the node; If the master node goes down, the request will be routed to the replica node through ClickHouse's replication mechanism to continue execution.

6. The request allocation method according to claim 1, characterized in that: The method further comprises: The request enters the ClickHouse distributed table engine and waits for routing distribution.

7. The request allocation method according to claim 1, characterized in that: The method further comprises: Obtain a first dynamic weight of a first node at a first moment; Obtain a second dynamic weight of the first node at a second moment; the second moment is later than the first moment; If the difference between the second dynamic weight and the first dynamic weight is greater than or equal to a preset threshold, the request is allocated to a second node other than the first node.

8. A request allocation system, characterized in that: include: The first module is used to obtain the performance data of the node; A second module is used to predict the load trend according to the historical load data and the performance data, and pre-allocate the requests according to the load trend; The third module is used to distribute the requests to ClickHouse nodes based on the load trend and the performance data and a weight-based routing algorithm.

9. A request allocation device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the request allocation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to implement the request allocation method according to any one of claims 1 to 7 when executed by the processor.

Images