Intelligent clustering method, system and equipment for time sequence database and medium

Abstract

The invention relates to the technical field of database data processing, and discloses a time sequence database intelligent clustering method, system and device and a medium, and the method comprises the steps: firstly monitoring a time sequence database process in real time, and collecting a real-time monitoring data packet of a node in a time sequence database; processing the real-time monitoring data packet in the extended data layer through an extended Berkley packet filtering technology, and generating tracking index data and performance statistical index data; deploying a framework of the cluster in the time sequence database, generating different deployment algorithms according to the deployment mode of the cluster, and calculating the state of each node in the time sequence database, the node layout and the mapping relation of the nodes based on the different deployment algorithms in combination with the tracking index data and the performance statistical index data; and adjusting a write-in protocol adapted to the time sequence database, obtaining a plurality of nodes matched with the real-time monitoring data according to the mapping relationship, and writing the real-time monitoring data into the plurality of nodes.

Description

Technical Field

[0001] The present invention relates to the technical field of database data processing, and in particular to a time series database intelligent clustering method, system, device and medium. Background Art

[0002] Time series databases play a key role in the field of system monitoring. They can collect rich and diverse indicator data of various components and applications in the Kubernetes cluster in real time and efficiently. They are also very suitable for monitoring traditional businesses. They enable us to accurately understand the operating status of the system. With the help of indicator monitoring, potential problems can be discovered in a timely manner, thereby effectively ensuring the stable operation of the system.

[0003] However, the single-node mode of some time series databases has many shortcomings. For example, its data storage capacity is relatively limited, which easily leads to insufficient capacity when conducting large-scale monitoring. In addition, the processing capacity of a single node also has an upper limit, which will encounter performance bottlenecks during high-concurrency collection and query. Summary of the invention

[0004] The embodiments of the present invention provide a time series database intelligent clustering method, system, computer device and medium to solve the technical problems that due to the relatively limited node data storage capacity, insufficient capacity is prone to occur during large-scale monitoring, and the processing capacity of a single node also has an upper limit, which may lead to performance bottlenecks during high-concurrency collection and query.

[0005] In a first aspect, a time series database intelligent clustering method is provided, comprising:

[0006] Monitor the time series database process in real time, collect real-time monitoring data packets of nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracking indicator data and performance statistical indicator data;

[0007] The framework for deploying clusters in a time series database generates different deployment algorithms based on the cluster deployment mode. Based on different deployment algorithms, the state, node layout, and node mapping relationship of each node in the time series database are calculated in combination with Trace indicator data and performance statistical indicator data.

[0008] Adjust the write protocol adapted by the time series database, obtain multiple nodes matching the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data to multiple nodes.

[0009] In a second aspect, a time series database intelligent cluster system is provided, including:

[0010] The collection module is used to monitor the time series database process in real time, collect the real-time monitoring data packets of the nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracking indicator data and performance statistical indicator data;

[0011] The computing module is used to deploy the cluster framework in the time series database. It generates different deployment algorithms according to the cluster deployment mode. Based on different deployment algorithms, it combines Trace tracing indicator data and performance statistical indicator data to calculate the status, node layout and node mapping relationship of each node in the time series database.

[0012] The writing module is used to adjust the writing protocol adapted by the time series database, obtain multiple nodes matching the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data into multiple nodes;

[0013] The training module is used to calculate the optimal data query node of the time series database based on the machine learning algorithm;

[0014] The query module is used to query the optimal data query node and obtain real-time monitoring data.

[0015] In a third aspect, a computer device is provided, the device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0016] Memory, used to store computer programs;

[0017] The processor is used to implement the steps of the above-mentioned time series database intelligent clustering method when executing the program stored in the memory.

[0018] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned time series database intelligent clustering method are implemented.

[0019] In the scheme implemented by the above-mentioned time series database intelligent clustering method, system, computer device and storage medium, the time series database process can be monitored in real time to collect real-time monitoring data packets of nodes in the time series database, and the real-time monitoring data packets can be processed through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer to generate Trace tracing indicator data and performance statistical indicator data; the cluster framework is deployed in the time series database, different deployment algorithms are generated according to the deployment mode of the cluster, and the status, node layout and node mapping relationship of each node in the time series database are calculated based on different deployment algorithms combined with Trace tracing indicator data and performance statistical indicator data; the write protocol adapted by the time series database is adjusted, and multiple nodes matching the real-time monitoring data are obtained according to the mapping relationship, and the real-time monitoring data is written to multiple nodes. By real-time monitoring of data and machine learning to optimize query paths, the response speed of time series data queries and the throughput of write operations can be significantly improved while ensuring accuracy. Based on the cluster deployment mode and node status calculation, the system can dynamically adjust resource allocation, reduce system bottlenecks and ensure efficient use of hardware resources. By using eBPF extended Berkeley packet filtering technology and XDP extended data layer, the performance and process status of time series database nodes can be mastered in real time. The converted performance monitoring indicators can better represent the actual business situation. By intelligently writing data to multiple nodes and dynamically adapting the write protocol, the system can maintain stability and reliability in the face of large amounts of data and sudden loads, while improving the scalability of the cluster. Finally, the node performance is predicted by a machine learning algorithm, combined with reasonable task allocation, load balancing is achieved, further improving the overall performance of the system, and the performance improvement is significant compared to open source solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0021] Figure 1 This is a schematic diagram of an application environment of a time series database intelligent clustering method in one embodiment of the present invention;

[0022] Figure 2 It is a flow chart of a time series database intelligent clustering method in one embodiment of the present invention;

[0023] Figure 3 is a flow chart of a time series database intelligent clustering method in another embodiment of the present invention;

[0024] Figure 4 yes Figure 2A schematic flow chart of a specific implementation of step S10;

[0025] Figure 5 yes Figure 3 A schematic flow chart of a specific implementation of step S30;

[0026] Figure 6 It is a structural diagram of a time series database intelligent cluster system in one embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0028] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The time series database intelligent clustering method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment, the client first monitors the time series database process in real time, collects the real-time monitoring data packets of the nodes in the time series database, processes the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generates Trace tracing indicator data and performance statistical indicator data; deploys the cluster framework in the time series database, generates different deployment algorithms according to the deployment mode of the cluster, and calculates the status, node layout and node mapping relationship of each node in the time series database based on different deployment algorithms combined with Trace tracing indicator data and performance statistical indicator data; adjusts the write protocol adapted by the time series database, obtains multiple nodes matching the real-time monitoring data according to the mapping relationship, and writes the real-time monitoring data into multiple nodes. By real-time monitoring of data and machine learning to optimize query paths, the response speed of time series data queries and the throughput of write operations can be significantly improved while ensuring accuracy. Based on the cluster deployment mode and node status calculation, the system can dynamically adjust resource allocation, reduce system bottlenecks and ensure efficient use of hardware resources. By using eBPF extended Berkeley packet filtering technology and XDP extended data layer, the performance and process status of time series database nodes can be mastered in real time. The converted performance monitoring indicators can better represent the actual business situation. By intelligently writing data to multiple nodes and dynamically adapting the write protocol, the system can maintain stability and reliability in the face of large amounts of data and sudden loads, while improving the scalability of the cluster. Finally, the node performance is predicted by a machine learning algorithm, combined with reasonable task allocation, load balancing is achieved, further improving the overall performance of the system, and the performance improvement is significant compared to open source solutions.

[0031] The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 As shown, Figure 2 A flow chart of a time series database intelligent clustering method provided by an embodiment of the present invention includes the following steps:

[0033] S10: monitor the time series database process in real time, collect real-time monitoring data packets of nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracing indicator data and performance statistical indicator data;

[0034] The time series database intelligent clustering method provided by the present invention can be applied to the intelligent clustering of time series databases in various application scenarios. The intelligent clustering of time series databases is usually realized through a platform, which can receive real-time monitoring data collected by the client in real time. The present invention has the ability to efficiently capture and process data packets and can efficiently collect and process monitoring data packets of each node in the time series database, effectively improving the query performance, writing efficiency, system reliability and resource utilization of the time series database system. In particular, when processing large-scale, high-concurrency real-time data, it can ensure the efficient operation of the system and ensure the accuracy, reliability and real-time performance of the data.

[0035] Specifically, a time series database is a database used to store and query time series data. Time series data usually has a timestamp, and the data items are arranged in chronological order. Common application scenarios include: monitoring data, sensor data, and log data. Monitoring data includes CPU usage, memory usage, network traffic, etc. Sensor data includes temperature, humidity, and other data sent by IoT devices. Log data includes system logs, event logs, etc.

[0036] In the monitoring system, it is usually necessary to track the performance and indicator status of the time series database in real time, including: query performance, such as query response time, executed SQL query type, query frequency, etc.; resource utilization, CPU, memory, disk usage; data writing and reading rate, monitoring data insertion, storage and reading speed. These monitoring data can help operation and maintenance personnel understand the operating status of the time series database and promptly discover and solve bottlenecks or faults.

[0037] In a distributed time series database system, multiple nodes (or multiple database instances) usually share the task of data storage and processing. Therefore, monitoring a time series database should not only focus on the performance of each individual node, but also on the data flow between them. Real-time monitoring data packets come from network data packets of each node, which include information such as requests, responses, and data synchronization of the time series database. The content of the monitoring data packet may include: query requests, such as SQL query statement execution requests; data transmission, such as synchronization or query result data between nodes; status updates, such as the health status or load information of the node. Monitoring data packets can be obtained through network packet capture tools or kernel-level packet capture mechanisms (such as XDP).

[0038] XDP (eXpress Data Path) is an efficient network packet processing mechanism provided by the Linux kernel. It can process packets before they enter the kernel's network protocol stack, significantly improving processing performance. XDP accelerates the processing of packets by directly operating network packets at the driver level.

[0039] Features of the XDP extended data layer: XDP can greatly reduce the latency of data packets from the receiving end to the processing end; XDP is processed at the NIC driver layer, skipping the traditional kernel protocol stack and reducing the context switching overhead; custom processing programs can be written in conjunction with eBPF to filter, modify, discard, redirect, and perform other operations on network data packets.

[0040] eBPF (extended Berkeley Packet Filter) is a kernel technology that allows developers to write user-defined programs (usually written in C) and load these programs into the kernel for execution. eBPF has very high performance and can run dynamically in the kernel without modifying the kernel source code. In the context of network packet processing, eBPF allows developers to create packet filter programs that can be directly mounted on network interfaces, XDP layers, or other kernel functions to process packets. Common operations include: packet filtering, packet modification, and packet redirection. Packet filtering: Determine which packets should be discarded and which should be forwarded. Packet modification: The content of the packet can be modified, such as modifying the packet header, adding information, etc. Packet redirection: Redirecting packets from one network interface to another. Combined with the XDP extended data layer, eBPF can efficiently process packets before they enter the kernel protocol stack, reducing latency and computational overhead.

[0041] By combining the XDP extended data layer with the eBPF extended Berkeley packet filtering technology, the real-time monitoring data packets of the time series database can be efficiently processed. Specific operations include: using XDP to capture real-time monitoring data packets, which may contain query requests, response data, inter-node synchronization data, etc. of the time series database; using eBPF to write custom programs to filter and analyze these data packets, for example, query packets can be filtered to focus only on specific types of queries (such as high-latency queries), or performance analysis can be performed on response packets; through the processing results of the eBPF program, monitoring indicators can be generated, such as query latency, throughput, data transmission rate, etc.

[0042] Trace tracing refers to recording and tracking the execution path, event sequence, and other information of data packets or processes. Combined with the eBPF extended Berkeley packet filtering technology, users can generate detailed trace tracing data to help analyze the flow of real-time monitoring data packets, performance bottlenecks, etc. Trace tracing indicator data can track various events of the time series database, such as the processing of query requests, the transmission path of data packets, etc. For example, through the eBPF extended Berkeley packet filtering technology, the query processing process can be captured, the time from initiation to execution of each query can be analyzed, and the resource consumption of each query (CPU, memory, disk, etc.) can be tracked. By processing the real-time monitoring data packets, some performance statistics can be obtained, such as: query response time, the processing time of a specific query; node load, the usage of CPU, memory, and network resources of each time series database node; throughput, the data transmission rate, the number of data packets processed, etc.

[0043] This data can be used to: optimize the configuration or query strategy of the time series database by identifying bottlenecks and high-latency operations; trace data can help locate the time and specific location of the failure.

[0044] This step provides efficient data packet capture and processing capabilities for the real-time monitoring time series database system through the XDP extended data layer and eBPF extended Berkeley packet filtering technology. By combining the XDP extended data layer and eBPF extended Berkeley packet filtering technology, the monitoring data packets of each node in the time series database can be efficiently collected and processed, and detailed trace tracking indicator data and performance statistics can be generated to help the client monitor system performance in real time, analyze query bottlenecks, and optimize database operations.

[0045] Among them, Figure 4 As shown, step S10 includes:

[0046] S11, real-time monitoring of the time series database process, and collection of real-time monitoring data of nodes in the time series database;

[0047] S12, processes real-time monitoring data through eBPF extended Berkeley packet filtering technology, and obtains real-time monitoring data packets of time series database process nodes in real time;

[0048] S13, in the XDP extended data layer, the eBPF extended Berkeley packet filtering technology is used to process real-time monitoring data packets and generate Trace tracking indicator data and performance statistical indicator data.

[0049] For steps S11-S13, specifically, it is responsible for real-time monitoring of the time series database process, collecting node equipment monitoring indicators (CPU central processing unit, memory, I / O input and output, network card traffic, load and other monitoring indicators), processing query data packets through eBPF extended Berkeley packet filtering technology in the XDP extended data layer and generating Trace tracking indicator data and performance statistical indicator data, including: collecting node equipment monitoring indicators, including CPU central processing unit, memory, I / O input and output, network card traffic, load and other key monitoring indicators. Through ebpf extended Berkeley packet filtering technology, hook Trace tracking point / sched / sched_process_fork: process creation event, Trace tracking point / sched / sched_process_exit: process exit event, Trace tracking point / sched / sched_process_free: process release event and other related Trace tracking point nodes, so as to obtain relevant information of the time series database instance process in real time and report it to the cluster management module and acceleration module. In the XDP extended data layer, the query data packet is processed through the eBPF extended Berkeley packet filtering technology to generate Trace tracking indicator data and performance statistical indicator data.

[0050] Of course, the eBPF extended Berkeley packet filtering technology program is loaded in the XDP extended data layer. When receiving the network data packet sent by the client, the data link layer, network layer, transport layer protocol and header information of the network data packet are parsed layer by layer to determine whether the network data packet text conforms to the time series database serialization protocol specification. If the network data packet text conforms to the protocol specification, the network data packet is determined to be a time series database protocol data packet; otherwise, the network data packet is determined to be a non-time series database protocol data packet. If the network data packet is determined to be a non-time series database protocol data packet, the XDP extended data program returns XDP_PASS (XDP passes, which is an operation constant in XDP (eXpress Data Path) used to indicate that the network data packet continues to be passed to the kernel protocol stack), so that the network data packet continues to be sent to the kernel's TCP / IP network protocol stack for processing, which is consistent with the traditional processing method. If the network data packet is determined to be a time series database protocol data packet, it is parsed according to the protocol. After the parsing is completed, the network data packet is modified (the span information (switch mirror information) of the proxy server Trace link tracking is added), and the XDP extended data program returns XDP_PASS, so that the parsed network data packet is sent to the kernel's TCP / IP network protocol stack for further processing, which is consistent with the traditional processing method. In addition, performance monitoring indicators are also counted and uploaded to the acceleration module.

[0051] S20: Deploy a cluster framework in the time series database, generate different deployment algorithms according to the cluster deployment mode, and calculate the status, node layout, and node mapping relationship of each node in the time series database based on different deployment algorithms combined with Trace indicator data and performance statistical indicator data;

[0052] Responsible for generating different deployment algorithms based on the cluster deployment mode. The cluster configuration algorithm refers to the specific operation mode of the deployment cluster (including the number of nodes, the number of replicas, and cross-computer room redundant disaster recovery). The status and role information of each node in this time series database is generated according to the deployment mode to facilitate the subsequent reading of modules, writing modules, and accelerating the use of module-related functions.

[0053] In the actual application of time series databases, clusters and databases are closely integrated in the following ways: Distributed storage and computing: The cluster architecture supports distributed storage, which shards and distributes time series data to different nodes by time range or label (such as device ID, region, etc.). Distributed computing frameworks (such as Spark or other internal optimization engines) can process query tasks in parallel to improve query speed. Load balancing: The cluster distributes write requests and query requests to different nodes through load balancing to avoid overloading a single node. The load balancing mechanism in the monitoring system can also dynamically adjust the allocation strategy to optimize resource usage. Efficient query and write: The cluster can process a large number of write operations in parallel and write real-time data to multiple nodes. For query operations, the cluster can quickly locate the node where the data is located through the index mechanism, and summarize the results to improve query performance. Fault recovery: The cluster architecture supports node fault detection and recovery, and ensures data persistence and availability through automatic migration and replication mechanisms.

[0054] A cluster is a system consisting of multiple computing nodes, usually used to improve computing power, storage capacity or high availability.

[0055] Cluster deployment modes include peer-to-peer deployment mode, master-slave deployment mode, distributed deployment mode, and high-availability deployment mode. In peer-to-peer deployment mode, all nodes are equal and each node has the same function. It is suitable for load balancing and high-availability scenarios. In master-slave deployment mode, one master node is responsible for management and coordination, and multiple slave nodes execute tasks. It is suitable for scenarios where data storage and reading loads are separated. In distributed deployment mode, tasks, data, or services are divided into different nodes for processing. It is suitable for large-scale data processing (such as MapReduce) or microservice architecture. In high-availability deployment mode, redundancy design is used to ensure that services can continue to be provided even if some nodes fail.

[0056] Different cluster deployment modes will affect the deployment algorithm of cluster nodes. The purpose of the deployment algorithm is to determine how to reasonably allocate and schedule resources, tasks, and data in the cluster. According to the goals and load characteristics of the cluster, common deployment algorithms include: load balancing algorithm, resource scheduling algorithm, fault tolerance and high availability algorithm, and data distribution algorithm. The load balancing algorithm will evenly distribute the workload among multiple nodes to ensure that each node is not overloaded. Common methods include: polling, weighted polling, least connection, etc. The resource scheduling algorithm dynamically allocates tasks or services according to the resource status of each node (such as CPU, memory, disk, network bandwidth, etc.). Common methods include: priority-based scheduling, shortest job first, etc. The fault tolerance and high availability algorithm mainly automatically reallocates tasks to other healthy nodes when a failure occurs. Common methods include: heartbeat detection, master-slave switching, failover, etc. The data distribution algorithm is used to determine the storage location of data between nodes and optimize data access and storage efficiency. Common methods include: hash distribution, range distribution, location-based distribution, etc.

[0057] Trace tracing indicator data is detailed tracking information of events during the execution of an application or system, which can record the program's call stack, event sequence, function execution time, network requests, database operations, etc. The execution time is the start time and end time of each operation (such as query, calculation, data transmission). The event flow is the flow of events between different nodes, such as a request from node A to node B. The system call is the system call of each node or process, such as file reading and writing, network requests, database queries, etc. Errors and exceptions are records of errors and exceptions that occur in the cluster to help analyze bottlenecks and failures. Resource consumption is the amount of resources used by the node during execution, such as CPU time, memory consumption, I / O operations, etc.

[0058] Analysis of trace tracking indicator data By collecting trace data of each node in the cluster, the working status of each node in the cluster can be analyzed in detail. For example, if the response time of a node is too long, or system calls frequently fail, it means that there may be a problem with the node; or through trace data, it can be found that the execution time of some nodes or tasks is long, revealing the source of performance bottlenecks.

[0059] Performance statistics are operating indicators of various resources in the cluster, which help monitor the consumption and performance of system resources. Common performance statistics include: CPU usage, memory usage, I / O input and output performance, network latency and bandwidth. CPU usage is the CPU utilization of each node, reflecting the processing power and load status of the node. Memory usage is the memory usage of each node, reflecting whether the node has memory overflow or bottleneck. I / O input and output performance is the I / O performance of the disk and network, which helps analyze whether there is a bottleneck in the data access process of the node. Network latency and bandwidth are the network communication latency and bandwidth between nodes, which affect the data transmission and synchronization speed. Performance statistics can help evaluate the health of each node during the deployment process and dynamically adjust resource allocation and task scheduling.

[0060] In order to generate deployment algorithms according to different deployment modes and calculate the status, layout, and mapping relationship of each node in the cluster, you can combine Trace tracing indicator data with performance statistics for analysis.

[0061] First, the node status is calculated through trace data and performance statistics, and the CPU, memory, disk I / O, network usage, etc. of each node are calculated to evaluate its load. Then, by analyzing the trace data, the request response time, latency, and execution efficiency of each node are calculated. Finally, the health status of the node is judged based on the abnormality of the node (such as extremely high load, frequent errors or abnormalities). Abnormal nodes need to be isolated or tasks need to be rescheduled.

[0062] The computing node layout is based on performance data, evaluates the load status of each node, and dynamically adjusts the node layout to evenly distribute the load. It can determine whether adjustments are needed based on the execution time and system calls of each node in the trace data. Analyze the health status of the nodes to ensure that key nodes are redundantly deployed and can be quickly switched when a failure occurs, increasing fault tolerance and high availability.

[0063] The mapping relationship of nodes distributes tasks, data or services among cluster nodes. Based on the trace indicator data and performance statistics, computing-intensive tasks can be assigned to nodes with lower loads and data-intensive tasks can be assigned to nodes with stronger I / O input and output performance according to the resource utilization of each node. The storage location of data can be reasonably arranged according to the data access mode (such as hot data, cold data) and the storage mapping capability of the node. According to the network delay and bandwidth, the data transmission path between nodes can be adjusted to ensure the efficiency of data transmission.

[0064] By combining different cluster deployment modes and generating appropriate deployment algorithms, the status, layout, and mapping relationship of each node in the cluster are optimized based on Trace data and performance statistics. This can improve cluster performance and resource utilization, dynamically adjust task allocation and data storage between nodes, improve system reliability and fault recovery capabilities, and achieve efficient monitoring, optimization, and fault handling of the cluster, ensuring stable operation of the system under high load and complex scenarios.

[0065] S30: Adjust the writing protocol adapted by the time series database, obtain multiple nodes matching the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data into the multiple nodes.

[0066] Time series databases are used to efficiently store and query time series data, which are usually continuously generated monitoring data, sensor data, etc. The write protocol refers to the rules for how data is submitted to the database and stored. In order to improve throughput and performance, time series databases usually adopt the following write protocols: batch write, partition write, and asynchronous write. Batch write: Write multiple data at once to reduce network and disk I / O. Partition write: Partition data by time range or other dimensions and store them on different physical nodes. Asynchronous write: The data write operation is non-blocking. The data is first cached in memory and then asynchronously written to disk to increase the write speed.

[0067] The mapping relationship is how to distribute data, tasks or requests to different nodes of the system. In a time series database, the mapping relationship is usually determined based on the timestamp, label (such as device ID, region, etc.) or hash value of the data. Through this mapping, the system can determine which nodes are responsible for storing certain time ranges or specific types of data.

[0068] According to different system loads, query patterns, and data volumes, the time series database can dynamically adjust the write protocol. Adjustments may include: load balancing, adjusting the write strategy according to the resource usage of the node (such as CPU, memory, storage, etc.) to ensure load balancing and avoid overloading a certain node. Data routing, writing real-time monitoring data to the appropriate node based on the mapping relationship. The mapping relationship determines which node is responsible for storing specific data, thereby optimizing write efficiency and storage distribution. Fault-tolerant processing, through redundancy mechanisms, ensures that when some nodes fail, data can be written to other healthy nodes to ensure high system availability.

[0069] Based on the mapping relationship, the system can distribute real-time monitoring data to multiple nodes. The specific steps are as follows: Shard the data according to time, label or other dimensions, and assign each data fragment to different nodes for storage. Multi-node writing, write data to multiple nodes in parallel according to the predetermined writing strategy (such as load balancing, redundancy, etc.), for example, the written data can be transmitted to multiple nodes at the same time and stored according to the data type or time range. Data synchronization ensures data consistency and synchronization between nodes so that subsequent queries can access the complete data.

[0070] like Figure 3 As shown, S40: Calculate the optimal data query node of the time series database based on the machine learning algorithm;

[0071] In distributed systems and time series databases, the efficiency of query execution directly affects the overall performance of the system. In order to improve query performance, query execution is usually analyzed and optimized by collecting the following two types of data: query tracking indicator data and performance statistical indicator data. Machine learning technology is used to learn and optimize the execution path of database queries from a large amount of historical data. This process includes steps such as data collection and preprocessing, model selection and training, and real-time query optimization. Machine learning models can help predict and select the optimal query execution path, thereby improving query performance, reducing latency, and optimizing resource usage.

[0072] The query tracking indicator data records each step and each operation (such as SQL execution plan, data table scan, index search, etc.) during the query execution process, including the execution time, execution order, event flow, etc. of each operation. It also records the query delay time, execution path, processing time of each subquery, etc. By tracking this data, you can find possible bottlenecks in the query, such as a query step that is executed too slowly or performance degradation on some nodes.

[0073] Performance statistics record the usage of system resources, such as CPU usage, memory usage, disk I / O input and output, network bandwidth, etc. Performance statistics help analyze the allocation and consumption of system resources during query execution. For example, a query may consume too much CPU or memory, causing a system bottleneck.

[0074] The optimal data query node of a query is the optimal scheduling path of each step and each node when the query is executed in a time series database or distributed system. In the context of query optimization, the optimal data query node usually means that all operations are completed as quickly as possible during the query execution process. Resources (such as CPU, memory, I / O, etc.) are used as efficiently as possible. The query execution process avoids unnecessary operations (such as unnecessary full table scans, redundant data transmission, etc.).

[0075] Among them, query paths are optimized through machine learning. The goal of machine learning is to learn and optimize these execution paths based on query tracking data and performance statistics. Machine learning can help analyze and learn patterns in the query execution process to optimize query paths. By training the model, the system can predict and select the optimal query path.

[0076] Before machine learning, data collection and preprocessing are required, and a large amount of query tracking indicator data and performance statistical indicator data are collected. These data contain all the information during the query execution process: the execution plan of each query, the database operations, nodes, and operation time involved; the CPU, memory, I / O, etc. consumed by each query step; the overall execution time of the query, broken down into the time consumed by each stage.

[0077] Data preprocessing steps: clean the data to remove irrelevant or erroneous data; feature engineering to extract meaningful features. For example, the time taken for each operation, the amount of data involved, the type of operation, etc. can be extracted from the query tracking data; standardize the data so that data of different dimensions (such as CPU usage, memory usage, etc.) can be processed by the model. In order to calculate the optimal data query node for the query, a variety of machine learning algorithms can be used to learn and predict. Common algorithms include: supervised learning algorithms, reinforcement learning algorithms, and deep learning.

[0078] Supervised learning algorithms include regression algorithms and classification algorithms. Regression algorithms: If the goal is to predict the execution time or resource consumption of a query, regression models (such as linear regression, decision tree regression, random forest regression, etc.) can be used to learn the relationship between query execution time and various performance indicators. Classification algorithms: If the goal is to classify different execution paths of a query (such as selecting the optimal data query node, selecting the best node for data processing, etc.), classification algorithms (such as support vector machines, decision trees, K nearest neighbors, etc.) can be used.

[0079] Reinforcement learning algorithms are mainly used in the query execution process. The system can use reinforcement learning to optimize decisions. The reinforcement learning model interacts with the environment (i.e., the query execution process) and continuously adjusts the decision strategy based on rewards (such as query execution time or resource consumption) to optimize the query execution path. By exploring different query paths, reinforcement learning can gradually find the optimal query execution path. The reward signal can be designed based on factors such as query response time, resource consumption, and latency.

[0080] In deep learning, if the query execution process is complex and the amount of data is large, deep learning models (such as neural networks) can also be used for feature extraction and query path optimization. Deep learning models such as convolutional neural networks (CNN) or recurrent neural networks (RNN) can learn complex patterns in the query execution process and perform path prediction.

[0081] During deep learning training, the objective function can be designed based on query execution time, resource consumption, latency, etc. The model learns the optimal query path by optimizing this objective function. Choose a suitable loss function to measure the prediction error of the model. For example, in regression tasks, the mean square error (MSE) can be used as the loss function; in classification tasks, the cross entropy loss function can be used. Train the machine learning model through historical query data and optimize the model parameters so that the model can accurately predict the optimal execution path of the query.

[0082] For model evaluation and optimization, use cross-validation techniques to evaluate the generalization ability of the model and avoid overfitting. You can also use multiple algorithms for comparison to select the model that best suits the current query optimization problem. Alternatively, you can optimize the model's hyperparameters through grid search or random search to achieve the best performance in query path optimization. During the training process, adjust the model's hyperparameters, introduce regularization, add more features, or use more complex model structures based on the evaluation results.

[0083] The trained machine learning model can be deployed to the query optimization system to calculate and select the optimal data query node for the query in real time. The model makes predictions based on the query tracking data and performance statistics data collected in real time: when a query request arrives, the system will input the model based on the real-time tracking data (such as query type, data volume, node load, etc.) to predict the optimal query path. Based on the prediction results of the model, the system can dynamically adjust the query execution plan. For example, it may select different indexes, adjust the data partitioning scheme, select different nodes for calculation, etc. Based on the optimal data query node for the query, the system may dynamically adjust the allocation of cluster resources to ensure that the query can be executed in the best state.

[0084] The goal of optimizing query execution paths through machine learning models is to reduce query execution time, improve resource utilization efficiency, reduce latency, and improve the overall performance of the system. Therefore, the effectiveness of query optimization depends largely on the quality and diversity of training data, and a large amount of historical data is required to train effective models. The real-time performance of machine learning models may become a bottleneck, especially the use of complex models (such as deep learning models) in real-time query optimization may face latency issues. The execution plan and path of the query may vary under different loads, hardware configurations, and database environments, so it is necessary to ensure that the trained model can adapt to different deployment environments.

[0085] Among them, Figure 5 As shown, step S40 includes:

[0086] S41, obtaining all monitoring data in the time series database;

[0087] S42, training all monitoring data based on a machine learning algorithm and performing a weight evaluation of performance indicators, and converting the weight evaluation results of the performance indicators into a performance score;

[0088] S43. Obtain the optimal data query node based on the performance score.

[0089] For steps S11-S13, specifically, the service performance index of the time series database node server is calculated, which is equivalent to this part in the above expression

[0090]

[0091] Where: S j represents the performance score of the time series database, node j instance. The higher the value, the greater the probability of query selecting this node. j represents the performance score of node j in the time series database (dynamically generated by machine learning algorithm); NMi = [NM1, NM2, …, NMn], represents the set of server performance indicators; WNMi = [WNM1, WNM2, …, WNMn], represents the weight of the corresponding performance indicator.

[0092] Divide the trace tracking indicator data and performance statistical indicator data into training sets and test sets, and perform data preprocessing; use the training set to perform model training based on the machine learning model to learn and fit the mapping relationship between feature data and corresponding performance; convert the trained model into a performance score. By receiving performance indicators from instance detection nodes (such as E1request_total_num_rate represents the interface response rate per unit time, E2 request_total_time_tp99 represents the response time of the interface response TP99, Ei request_duration_ts_num_avg represents the average value of the time series returned within the request time, and other monitoring indicator data). First, divide the historical data into training sets and test sets, and perform data preprocessing, such as standardization or normalization. Then, select a suitable machine learning model and use the training set data for model training to learn and fit the relationship between feature data and corresponding performance to perform performance indicator weight evaluation, and convert the performance indicator weight evaluation results into performance scores. During the training process, optimize the model according to the evaluation results, and may adjust hyperparameters, introduce regularization, add more features, or adopt more complex model structures. These optimization measures are designed to improve the model's predictive performance on the test set to better adapt to the actual situation and improve model accuracy and generalization ability. Finally, we use the trained model for real-time performance scoring.

[0093] Receive event information from instance nodes (process start, service provision, stop, etc.) and convert it into event scores. Finally, we bring the corresponding data into the formula to get the selection of the optimal node for query.

[0094] like Figure 3 As shown, S50: query the optimal data query node and obtain real-time monitoring data;

[0095] Optimal data query node query refers to selecting the most efficient way to execute query operations based on the query execution requirements and system resource status. This includes: selecting the query execution path and real-time optimization. Selecting the query execution path: deciding how to execute the query in the database or cluster, selecting the optimal index, computing node, data storage location, etc. Real-time optimization: dynamically adjusting the query path based on the current system load, resource usage, and query characteristics. Real-time monitoring data refers to data collected by the system during operation about hardware resources (such as CPU, memory, I / O, network bandwidth, etc.) and software performance (such as response time, query latency, error rate, etc.). This data is usually transmitted in real time through monitoring tools or application logs.

[0096] The process of querying real-time monitoring data and processing data according to the optimal data query node is as follows: Data collection: Real-time collection of system and application performance data, including query execution time, resource consumption, etc. Query analysis: Through the analysis of real-time monitoring data, determine whether the current query path is efficient and whether there are resource bottlenecks or performance problems. Path optimization: If the monitoring data shows that the query execution efficiency is not high or the resource consumption is too large, the system can dynamically adjust the query path. Possible adjustment methods include changing the index, adjusting the load balancing, changing the computing node or data storage strategy, etc. Resource scheduling: According to the real-time demand of the query and the load of the system resources, the system resources are reasonably scheduled to ensure that the query can be executed in the optimal state. Execution feedback: The feedback loop of real-time monitoring data and query execution results ensures that the optimized path of the query continues to take effect in subsequent queries.

[0097] The optimization target method improves query efficiency, reduces query execution time and resource consumption; optimizes the query execution path in real time according to the current state of the system to avoid performance degradation due to resource bottlenecks; ensures that system resources are used efficiently and allocated reasonably to avoid resource waste or overload.

[0098] Add Trace tracking link tracking monitoring information for query operations; parse SQL structured query language, query nodes according to the optimal performance data dynamically calculated by the acceleration module, and aggregate the real-time monitoring data according to SQL structured query language and return it to the client; if the query process reports an error, record the error monitoring event and upload it to the cluster, and record the timeout time and retransmission times according to the retry mechanism. Through real-time monitoring data, the system can dynamically adjust the query path according to the actual execution of the query to achieve the optimal execution of the query. Real-time data processing and path optimization help improve query performance, improve the utilization of system resources, and ensure efficient query response.

[0099] The optimization target method is to improve the data writing speed and system throughput through parallel writing, load balancing and appropriate data sharding. It is also possible to easily add new nodes to process more data through a distributed architecture. It is also possible to ensure the reliability and fault tolerance of data writing through a redundancy mechanism.

[0100] Specifically, adjust the write protocol adapted by the time series database; use the consistency of the time series database based on the hash algorithm to calculate the nodes that need to be written, and write the real-time monitoring data to the corresponding multiple nodes according to the mapping relationship; if an error occurs during the writing process, record the monitoring event of the error and upload it to the cluster, and record the timeout time and number of retransmissions according to the retry mechanism.

[0101] Add Trace link tracking monitoring information for write operations. Use the consistent hash algorithm to calculate the nodes that need to be written to the time series data, and write it to the corresponding multiple time series database nodes according to the configured cluster write strategy. If an error occurs during the write process, the cluster module that records the monitoring event and uploads it will write it according to the retry mechanism (timeout T, number of retransmissions). Adjusting the write protocol of the time series database and writing real-time monitoring data to multiple nodes based on the mapping relationship is to optimize data storage and query performance. Through reasonable data sharding, load balancing and fault-tolerant strategies, it is possible to improve write efficiency, ensure high data availability, and meet the needs of large-scale data processing.

[0102] This embodiment can grasp the performance status and process status of the time series database node in real time by leveraging the powerful function of the eBPF extended Berkeley packet filtering technology. The performance monitoring indicators are converted through the XDP extended data layer to make them more realistically reflect the actual situation of the business. This provides us with deeper and more accurate performance insights, which helps to discover and solve potential problems in a timely manner. Business performance node score prediction model: By using advanced machine learning algorithms or other related technologies, the node performance score can be predicted in real time and accurately. The model makes full use of historical data and rich feature information to accurately estimate the score of each node. This intelligent prediction method provides a more reliable and intelligent basis for node allocation, significantly improving the efficiency and accuracy of queries. Server node performance evaluation indicators: When formulating query strategies, not only the prediction results of business performance are considered, but also a number of performance evaluation indicators of the server are integrated, such as CPU usage, memory usage, I / O read and write speed, and load conditions. By comprehensively evaluating the performance level of the server, query strategies can be formulated more accurately, thereby effectively improving the overall throughput of the cluster.

[0103] The cluster mode enhances the integration and analysis capabilities, and can integrate data from different regions and data centers for comprehensive and in-depth analysis, thereby providing more accurate monitoring results. Reliability has been greatly improved. When a single node fails, other nodes can seamlessly take over and continue to provide services, effectively avoiding the impact of single point failures on the performance degradation of the monitoring system. This feature ensures the stable operation of the monitoring system and provides a solid guarantee for the normal development of business.

[0104] It can be seen that in the above scheme, for the intelligent cluster management of the time series database, the time series database process is first monitored in real time, the real-time monitoring data packets of the nodes in the time series database are collected, and the real-time monitoring data packets are processed by the eBPF extended Berkeley packet filtering technology in the XDP extended data layer to generate Trace tracing indicator data and performance statistical indicator data; the cluster framework is deployed in the time series database, different deployment algorithms are generated according to the deployment mode of the cluster, and the status, node layout and node mapping relationship of each node in the time series database are calculated based on different deployment algorithms combined with Trace tracing indicator data and performance statistical indicator data; the write protocol adapted by the time series database is adjusted, and multiple nodes matching the real-time monitoring data are obtained according to the mapping relationship, and the real-time monitoring data is written to multiple nodes. By real-time monitoring of data and machine learning to optimize query paths, the response speed of time series data queries and the throughput of write operations can be significantly improved while ensuring accuracy. Based on the cluster deployment mode and node status calculation, the system can dynamically adjust resource allocation, reduce system bottlenecks and ensure efficient use of hardware resources. By using eBPF extended Berkeley packet filtering technology and XDP extended data layer, the performance and process status of time series database nodes can be mastered in real time. The converted performance monitoring indicators can better represent the actual business situation. By intelligently writing data to multiple nodes and dynamically adapting the write protocol, the system can maintain stability and reliability in the face of large amounts of data and sudden loads, while improving the scalability of the cluster. Finally, the node performance is predicted by a machine learning algorithm, combined with reasonable task allocation, load balancing is achieved, further improving the overall performance of the system, and the performance improvement is significant compared to open source solutions.

[0105] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0106] In one embodiment, a time series database intelligent clustering system is provided, and the time series database intelligent clustering system corresponds one-to-one to the time series database intelligent clustering method in the above embodiment. Figure 6 As shown, the time series database intelligent cluster system includes a collection module 101, a calculation module 102, a training module 104, a query module 105, and a writing module 103. The functional modules are described in detail as follows:

[0107] The collection module 101 is used to monitor the time series database process in real time, collect the real-time monitoring data packets of the nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracking indicator data and performance statistical indicator data;

[0108] The computing module 102 is used to deploy the cluster framework in the time series database, generate different deployment algorithms according to the cluster deployment mode, and calculate the state, node layout and node mapping relationship of each node in the time series database based on different deployment algorithms combined with Trace tracing indicator data and performance statistical indicator data;

[0109] The writing module 103 is used to adjust the writing protocol adapted by the time series database, obtain multiple nodes matched by the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data into the multiple nodes.

[0110] A training module 104 calculates the optimal data query node of the time series database based on a machine learning algorithm;

[0111] The query module 105 queries the optimal data query node and obtains real-time monitoring data.

[0112] In one embodiment, the training module 104 is specifically configured to:

[0113] Get all monitoring data in the time series database;

[0114] Based on machine learning algorithms, all monitoring data are trained and performance indicator weight evaluation is performed, and the performance indicator weight evaluation results are converted into performance scores;

[0115] Get the optimal data query node based on performance score.

[0116] In one embodiment, the training module 104 is further configured to:

[0117] Divide all monitoring data into training sets and test sets, and perform data preprocessing;

[0118] Train the training set based on the machine learning algorithm model and learn and fit the mapping relationship between feature data and corresponding performance indicators;

[0119] Evaluate the performance indicator weight according to the mapping relationship between the characteristic data and the corresponding performance indicator, and obtain the performance indicator weight evaluation result;

[0120] According to the evaluation results, the performance indicator weight evaluation results are converted into performance scores

[0121] In one embodiment, the writing module 103 is specifically used for:

[0122] Adjust the write protocol for time series database adaptation;

[0123] Based on the hash algorithm, the consistency calculation of the time series database is used to calculate the nodes that need to be written, and the real-time monitoring data is written to the corresponding multiple nodes according to the mapping relationship;

[0124] If an error occurs during the writing process, the monitoring event of the error is recorded and uploaded to the time series database, and the timeout time and number of retransmissions are recorded according to the retry mechanism.

[0125] In one embodiment, the query module 105 is specifically configured to:

[0126] Parse SQL structured query language, query the optimal data query node, obtain real-time monitoring data, aggregate the real-time monitoring data according to SQL structured query language, and return the aggregated results to the client;

[0127] If an error is reported during the query process, the monitoring event of the error is recorded and uploaded to the time series database, and the timeout time and number of retransmissions are recorded according to the retry mechanism.

[0128] In one embodiment, the acquisition module 101 is specifically used for:

[0129] Monitor the time series database process in real time and collect real-time monitoring data of nodes in the time series database;

[0130] Process the real-time monitoring data through eBPF extended Berkeley packet filtering technology, and obtain the real-time monitoring data packets of the time series database process nodes in real time;

[0131] In the XDP extended data layer, eBPF extended Berkeley packet filtering technology is used to process real-time monitoring data packets and generate Trace tracking indicator data and performance statistical indicator data.

[0132] The present invention provides a time series database intelligent cluster system, which first monitors the time series database process in real time, collects real-time monitoring data packets of nodes in the time series database, processes the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer and generates Trace tracing indicator data and performance statistical indicator data; deploys a cluster framework in the time series database, generates different deployment algorithms according to the deployment mode of the cluster, calculates the state of each node in the time series database, the node layout and the mapping relationship of the nodes based on the different deployment algorithms combined with the Trace tracing indicator data and the performance statistical indicator data; adjusts the write protocol adapted to the time series database, obtains multiple nodes matched by the real-time monitoring data according to the mapping relationship, and writes the real-time monitoring data into the multiple nodes. By real-time monitoring of data and machine learning to optimize query paths, the response speed of time series data queries and the throughput of write operations can be significantly improved while ensuring accuracy. Based on the cluster deployment mode and node status calculation, the system can dynamically adjust resource allocation, reduce system bottlenecks and ensure efficient use of hardware resources. By using eBPF extended Berkeley packet filtering technology and XDP extended data layer, the performance and process status of time series database nodes can be mastered in real time. The converted performance monitoring indicators can better represent the actual business situation. By intelligently writing data to multiple nodes and dynamically adapting the write protocol, the system can maintain stability and reliability in the face of large amounts of data and sudden loads, while improving the scalability of the cluster. Finally, the node performance is predicted by a machine learning algorithm, combined with reasonable task allocation, load balancing is achieved, further improving the overall performance of the system, and the performance improvement is significant compared to open source solutions.

[0133] For the specific definition of the time series database intelligent cluster system, please refer to the definition of the time series database intelligent cluster method above, which will not be repeated here. Each module in the above-mentioned time series database intelligent cluster system can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0134] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities and execute programs stored in the memory. The memory of the computer device includes non-volatile and / or volatile storage media, internal memory, and the memory is used to store computer programs. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a time series database intelligent clustering method.

[0135] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of a time series database intelligent clustering method

[0136] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0137] Monitor the time series database process in real time, collect real-time monitoring data packets of nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracking indicator data and performance statistical indicator data;

[0138] The framework for deploying clusters in a time series database generates different deployment algorithms based on the cluster deployment mode. Based on different deployment algorithms, the state, node layout, and node mapping relationship of each node in the time series database are calculated in combination with Trace indicator data and performance statistical indicator data.

[0139] Adjust the write protocol adapted by the time series database, obtain multiple nodes matching the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data to multiple nodes.

[0140] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0141] Monitor the time series database process in real time, collect real-time monitoring data packets of nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracking indicator data and performance statistical indicator data;

[0142] The framework for deploying clusters in a time series database generates different deployment algorithms based on the cluster deployment mode. Based on different deployment algorithms, the state, node layout, and node mapping relationship of each node in the time series database are calculated in combination with Trace indicator data and performance statistical indicator data.

[0143] Adjust the write protocol adapted by the time series database, obtain multiple nodes matching the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data to multiple nodes.

[0144] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0146] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0147] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A time series database intelligent clustering method, characterized in that: The method comprises: Monitor the time series database process in real time, collect real-time monitoring data packets of nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracking indicator data and performance statistical indicator data; The framework for deploying clusters in a time series database generates different deployment algorithms based on the cluster deployment mode. Based on different deployment algorithms, the state, node layout, and node mapping relationship of each node in the time series database are calculated in combination with Trace indicator data and performance statistical indicator data. Adjust the write protocol adapted by the time series database, obtain multiple nodes matching the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data to multiple nodes.

2. The time series database intelligent clustering method according to claim 1, characterized in that: After the steps of adjusting the writing protocol adapted by the time series database, acquiring multiple nodes matched by the real-time monitoring data according to the mapping relationship, and writing the real-time monitoring data into the multiple nodes, the method further includes: Calculate the optimal data query node of the time series database based on machine learning algorithms; Query the optimal data query node and obtain real-time monitoring data.

3. The time series database intelligent clustering method according to claim 2, characterized in that: The step of calculating the optimal data query node of the time series database based on the machine learning algorithm includes: Get all monitoring data in the time series database; Based on machine learning algorithms, all monitoring data are trained and performance indicator weight evaluation is performed, and the performance indicator weight evaluation results are converted into performance scores; Get the optimal data query node based on performance score.

4. The time series database intelligent clustering method according to claim 3, characterized in that: The steps of training all monitoring data based on the machine learning algorithm and performing performance indicator weight evaluation, and converting the performance indicator weight evaluation result into a performance score include: Divide all monitoring data into training sets and test sets, and perform data preprocessing; Train the training set based on the machine learning algorithm model and learn and fit the mapping relationship between feature data and corresponding performance indicators; Evaluate the performance indicator weight according to the mapping relationship between the characteristic data and the corresponding performance indicator, and obtain the performance indicator weight evaluation result; According to the evaluation results, the performance indicator weight evaluation results are converted into performance scores.

5. The time series database intelligent clustering method according to claim 2, characterized in that: The steps of querying the optimal data query node and obtaining real-time monitoring data include: Parse SQL structured query language, query the optimal data query node, obtain real-time monitoring data, aggregate the real-time monitoring data according to SQL structured query language, and return the aggregated results to the client; If an error is reported during the query process, the monitoring event of the error is recorded and uploaded to the time series database, and the timeout time and number of retransmissions are recorded according to the retry mechanism.

6. The time series database intelligent clustering method according to claim 1, characterized in that: The steps of adjusting the writing protocol adapted by the time series database, obtaining multiple nodes matched by the real-time monitoring data according to the mapping relationship, and writing the real-time monitoring data into the multiple nodes include: Adjust the write protocol for time series database adaptation; Based on the hash algorithm, the consistency calculation of the time series database is used to calculate the nodes that need to be written, and the real-time monitoring data is written to the corresponding multiple nodes according to the mapping relationship; If an error occurs during the writing process, the monitoring event of the error is recorded and uploaded to the time series database, and the timeout time and number of retransmissions are recorded according to the retry mechanism.

7. The time series database intelligent clustering method according to claim 1, characterized in that: The steps of real-time monitoring of the time series database process, collecting real-time monitoring data packets of nodes in the time series database, processing the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer and generating Trace tracking indicator data and performance statistical indicator data include: Monitor the time series database process in real time and collect real-time monitoring data of nodes in the time series database; Process the real-time monitoring data through eBPF extended Berkeley packet filtering technology, and obtain the real-time monitoring data packets of the time series database process nodes in real time; In the XDP extended data layer, eBPF extended Berkeley packet filtering technology is used to process real-time monitoring data packets and generate Trace tracking indicator data and performance statistical indicator data.

8. A time series database intelligent cluster system, characterized in that: The collection module is used to monitor the time series database process in real time, collect the real-time monitoring data packets of the nodes in the time series database, process the real-time monitoring data packets through the eBPF extended Berkeley packet filtering technology in the XDP extended data layer, and generate Trace tracking indicator data and performance statistical indicator data; The computing module is used to deploy the cluster framework in the time series database. It generates different deployment algorithms according to the cluster deployment mode. Based on different deployment algorithms, it combines Trace tracing indicator data and performance statistical indicator data to calculate the status, node layout and node mapping relationship of each node in the time series database. The writing module is used to adjust the writing protocol adapted by the time series database, obtain multiple nodes matching the real-time monitoring data according to the mapping relationship, and write the real-time monitoring data into multiple nodes; The training module is used to calculate the optimal data query node of the time series database based on the machine learning algorithm; The query module is used to query the optimal data query node and obtain real-time monitoring data.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the time series database intelligent clustering method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the time series database intelligent clustering method according to any one of claims 1 to 7 are implemented.

Images