A log processing method and device based on a ClickHouse cluster

Abstract

The application provides a log processing method and device based on a ClickHouse cluster, including: obtaining metadata information, the metadata information being determined in advance according to the format and fields of log data, the metadata information including a partition key; receiving log data, adding the metadata information to the log data to obtain parsed data, adding the parsed data to a buffer queue according to the partition key, and synthesizing a buffer queue with a queue length less than a preset length into one queue according to a preset merging time; selecting a node with node performance meeting a first preset condition from a node list as a first optimal node, and submitting the parsed data to the first optimal node. According to the characteristics that the fields of the log data are relatively fixed, the metadata information is predefined, and then the log data is cached to different queues according to the partition key, the log can be written into the ClickHouse node at a low frequency and a high volume, the number of write partitions and the number of partition mergings are reduced, and the writing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a log processing method and apparatus based on ClickHouse cluster. Background Technology

[0002] Clickhouse is a columnar database management system for OLAP (Online Analytical Processing). Data is stored in tables in a columnar manner within Clickhouse. High availability is achieved through the ReplicatedMergeTree engine and a Zookeeper cluster. Existing technologies can write data to Clickhouse using Kafka or Flink, but both methods consume significant resources and have low write efficiency. Summary of the Invention

[0003] This application provides a log processing method and apparatus based on a ClickHouse cluster. This method can reduce the number of partitions generated during writing and the number of partition merging operations, thereby improving writing efficiency.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] Firstly, a log processing method based on a ClickHouse cluster is provided. The method includes: acquiring metadata information of log data, wherein the metadata information is pre-determined according to the format and fields of the log data, and the metadata information includes a partition key; receiving the log data, adding the metadata information to the log data to obtain parsed data, adding the parsed data to a buffer queue according to the partition key, and merging buffer queues with queue lengths less than a preset length into one queue according to a preset merging time; selecting a node whose node performance meets a first preset condition from the node list of the ClickHouse cluster as a first optimal node, and submitting the parsed data in the buffer queue to the first optimal node.

[0006] As can be seen from the method described in the first aspect, by predefining metadata information based on the relatively fixed characteristics of log data fields, and then caching log data in different queues according to the partition key in the metadata information, logs can be written to ClickHouse nodes in a low-frequency but high-volume manner, reducing the number of partitions generated by writing and the number of partition merging times, thereby improving writing efficiency.

[0007] In conjunction with the first aspect, in one possible design scheme, the metadata information further includes one or more of the following: data source, cluster name, table name, distributed table name, table fields, and field mappings. Based on this possible design scheme, this embodiment can add parsed data to the corresponding buffer queue according to the database name, table name, and / or partition key of the log data.

[0008] In conjunction with the above design scheme, in some methods, merging buffer queues with queue lengths less than a preset length into one queue according to a preset merging time includes: scanning the queue length of each buffer queue according to the preset merging time, and merging buffer queues with queue lengths less than the preset length into one queue every timeout submission time based on the preset merging time.

[0009] In this embodiment, when merging buffer queues with queue lengths less than the preset length into one queue, the batch commit count can first be determined. This batch commit count limits the maximum number of parsed data items submitted to the ClickHouse cluster in each batch. Then, buffer queues with queue lengths less than the preset length are designated as short queues. Short queues whose sum of queue lengths is less than or equal to the batch commit count are merged into one queue. This avoids the problems of a large number of end queues accumulating memory and excessively small batch commit counts.

[0010] Based on this possible design scheme, this embodiment selects the node whose performance meets a first preset condition from the node list of the ClickHouse cluster as the first optimal node. This includes: acquiring the write speed and available space of each node in the node list every 5 seconds, evaluating the load status of each node, evaluating the load status of each node based on its write speed and available space, and sorting all nodes in the node list according to the load status; selecting the node with the best load status from the sorted node list as the first optimal node. By collecting ClickHouse node write performance and load information, a healthy node list is maintained, and the optimal node is selected for writing during commit, ensuring balanced node data.

[0011] In conjunction with the first aspect, in one possible design scheme, submitting the parsed data in the buffer queue to the first optimal node includes: detecting the queue length of each buffer queue in the ClickHouse cluster according to a preset detection time; when the queue length of any buffer queue reaches the single batch submission quantity, submitting the parsed data in the buffer queue whose queue length has reached the single batch submission quantity to the first optimal node.

[0012] In conjunction with the first aspect, in another possible design scheme, submitting the parsed data in the buffer queue to the first optimal node includes: detecting the JVM memory usage and data timeout of the ClickHouse cluster according to a preset detection time; when either the JVM memory usage reaches a preset memory threshold or the data timeout reaches the timeout submission time, the parsed data in the buffer queue is submitted to the first optimal node.

[0013] Based on the above possible design schemes, the method further includes: when the submission fails, selecting a node whose performance meets the second preset condition from the node list as the second optimal node, and submitting the parsed data in the buffer queue to the second optimal node.

[0014] Secondly, a log processing device based on a ClickHouse cluster is provided. This ClickHouse cluster-based log processing device includes: a metadata acquisition unit, a log data processing unit, and a data submission unit. The metadata acquisition unit is used to acquire metadata information of the log data, the metadata information being predetermined according to the format and fields of the log data; the log data processing unit is used to receive the log data, add the metadata information to the log data to obtain parsed data, add the parsed data to a buffer queue according to the partition key of the log data, and merge buffer queues with queue lengths less than a preset length into one queue according to a preset merging time; the data submission unit is used to select a node whose node performance meets a first preset condition from the node list of the ClickHouse cluster as a first optimal node, and submit the parsed data in the buffer queue to the first optimal node.

[0015] Optionally, in the above-mentioned log processing device based on ClickHouse cluster, the metadata information also includes one or more of the following: data source, cluster name, table name, distributed table name, table fields, and field mapping.

[0016] Optionally, the log data processing unit described in the second aspect may include: a partitioning unit, used to add the parsed data to the corresponding buffer queue according to the database name, table name and / or partition key of the log data.

[0017] Optionally, the log data processing unit described in the second aspect may further include: a queue merging unit, configured to scan the queue length of each of the buffer queues according to the preset merging time, and merge buffer queues with queue lengths less than the preset length into one queue every timeout commit time based on the preset merging time.

[0018] Optionally, the queue merging unit may include: a first processing unit, used to determine the number of submissions in a single batch, wherein the number of submissions in a single batch is used to limit the maximum number of parsed data submitted to the ClickHouse cluster in each batch; and a second processing unit, used to record buffer queues with queue lengths less than the preset length as short queues, and to merge short queues whose sum of queue lengths is less than or equal to the number of submissions in a single batch into one queue.

[0019] Optionally, the data submission unit described in the second aspect may include: a node evaluation unit, configured to acquire the write speed and available space of each node in the node list every 5 seconds, evaluate the load status of each node, evaluate the load status of each node based on the write speed and available space of each node, and sort all nodes in the node list according to the load status; and a node selection unit, configured to select the node with the best load status from the sorted node list as the first optimal node.

[0020] Optionally, the data submission unit described in the second aspect may further include: an optimal node first selection unit, used to detect the JVM memory usage and data timeout of the ClickHouse cluster according to a preset detection time, and when either the JVM memory usage reaches a preset memory threshold or the data timeout reaches the timeout submission time, the parsed data in the buffer queue is submitted to the first optimal node.

[0021] Optionally, the data submission unit described in the second aspect may further include: an optimal node second selection unit, used to detect the queue length of each buffer queue of the ClickHouse cluster according to a preset detection time, and when the queue length of any buffer queue reaches the single batch submission quantity, submit the parsed data in the buffer queue whose queue length has reached the single batch submission quantity to the first optimal node.

[0022] Optionally, the above-mentioned log processing device based on ClickHouse cluster may further include: an error retransmission unit, used to select a node whose node performance meets a second preset condition as a second optimal node from the node list when the submission fails, and submit the parsed data in the buffer queue to the second optimal node.

[0023] Furthermore, the technical effects of the log processing device based on the ClickHouse cluster described in the second aspect can be referred to the technical effects of the log processing method based on the ClickHouse cluster described in the first aspect, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic diagram illustrating the process of writing data to ClickHouse;

[0026] Figure 2 A flowchart illustrating the log processing method based on a ClickHouse cluster provided in this application embodiment;

[0027] Figure 3 A flowchart illustrating a log processing method based on a ClickHouse cluster, provided as another embodiment of this application;

[0028] Figure 4 A flowchart illustrating a log processing method based on a ClickHouse cluster, as provided in another embodiment of this application;

[0029] Figure 5 A schematic diagram illustrating the execution process of a log processing method based on a ClickHouse cluster provided in an embodiment of this application;

[0030] Figure 6 A schematic diagram illustrating the execution process of a log processing method based on a ClickHouse cluster, provided as another embodiment of this application;

[0031] Figure 7 A schematic diagram of the architecture of a log processing system based on a ClickHouse cluster, provided for another embodiment of this application;

[0032] Figure 8 A schematic diagram of the structure of a log processing device based on a ClickHouse cluster provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0034] Reference numerals: 800 - Log processing device based on ClickHouse cluster; 810 - Data acquisition unit; 820 - Data processing unit; 830 - Data submission unit; 2000 - Electronic device; 2001 - Processor; 2002 - Memory; 2003 - Transceiver; 2004 - Processor. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0036] In the embodiments of this application, words such as "exemplarily" and "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as an "example" in this application should not be construed as being better or more advantageous than other embodiments or design options. Specifically, the use of the word "example" is intended to present the concept in a concrete manner. Furthermore, in the embodiments of this application, the meaning expressed by "and / or" can be both, or it can be either one or the other.

[0037] Please see Figure 1 Currently, there are two main solutions for writing data to ClickHouse: Solution 1: Direct writing from Kafka to ClickHouse. ClickHouse creates an engine table mapping to Kafka topics, collects data into the engine table, creates materialized views to store index rules, and then parses the index rules to write the final data. Each topic corresponds to a table in ClickHouse, which is inflexible, and parsing data and creating materialized views consumes a lot of CPU, affecting the write speed. Solution 2: Writing to ClickHouse via Flink. After log data is processed by Kafka, Flink is used to parse, clean, and transform the log data before writing the processed log data to ClickHouse. Flink is typically used for large data write scenarios with many data partitions. The merge speed of MergeTree cannot keep up with the directory generation speed, which is prone to errors and cannot meet the real-time write requirements. Using Flink to output to ClickHouse results in a large number of partitions, and the generation and merging of partitions consume a lot of CPU and memory resources, resulting in a low write speed.

[0038] To address the aforementioned issues, the inventors of this application, after careful research, discovered that log data fields are relatively fixed. By predefining schema data and caching it in different partition queues based on partition keys, the low-frequency, high-volume writing method aligns with the characteristics of the ClickHouse database. It eliminates the need for intermediate clusters and temporary tables, reducing the number of partitions generated by writes and the number of partition merges, thereby reducing CPU and memory consumption.

[0039] The following will combine Figure 2 The log processing method based on ClickHouse cluster provided in the embodiments of this application will be described in detail. For example, Figure 2 This is a flowchart illustrating the log processing method based on a ClickHouse cluster provided in an embodiment of this application.

[0040] like Figure 2 As shown, this log processing method based on the ClickHouse cluster includes the following steps:

[0041] S210: Obtain metadata information of log data. The metadata information is predetermined according to the format and fields of the log data. The metadata information includes partition keys.

[0042] Metadata, also known as intermediary data or relay data, is data about data, primarily describing data properties to support functions such as indicating storage location, historical data, resource lookup, and file records. Specifically, log data has certain format specifications, and its fields are relatively fixed. Based on this characteristic of log data, the embodiments of this application can determine metadata information according to the format and fields of the log data.

[0043] In the embodiments of this application, the data content contained in the metadata information can be determined in advance based on the format and fields of the log data, such as determining that the metadata information must include a partition key. When log data is obtained, the value of the partition key in the metadata information is determined based on the currently obtained log data. Optionally, the metadata information may include information required for writing, such as the data source (ds), cluster name (cn), table name (tn), distribution table name (dtn), table fields (Table Fields), partition key (pk), and field mapping (fm), so that the parsed data with added metadata information can include all the information required to write to the ClickHouse database.

[0044] S220: Receive the log data, add the metadata information to the log data to obtain parsed data, add the parsed data to the buffer queue according to the partition key, and merge the buffer queues with a queue length less than the preset length into one queue according to the preset merging time.

[0045] Therefore, in some implementations, after receiving log data, metadata information can be added to the log data to obtain parsed data. Then, the parsed data can be added to a buffer queue according to the partition key. In the embodiments of this application, there can be one or more buffer queues. When there are multiple buffer queues, the parsed data (i.e., log data with added metadata information) can be partitioned using the partition key.

[0046] Parsed data can be partitioned into corresponding buffer queues based on different partitioning criteria using the partition key. Optionally, partitioning can be performed based on the data source of the log data, i.e., adding parsed data to the corresponding buffer queue according to the data source of the log data, such as grouping parsed data from the same data source into the same buffer queue; partitioning can also be performed based on the table name tn of the log data, i.e., adding parsed data to the corresponding buffer queue according to the table name tn of the log data, such as grouping parsed data from the same table name tn into the same buffer queue, etc. Optionally, partitioning can also be performed based on the database name dn of the log data, i.e., adding parsed data to the corresponding buffer queue according to the database name dn of the log data, such as grouping parsed data from the same database name dn into the same buffer queue, etc.

[0047] In the embodiments of this application, in order to prevent the buffer queue from being too short and resulting in too many partitions, buffer queues with a queue length less than a preset length can be merged into one queue according to a preset merging time. The preset merging time and preset length can be preset or adjusted in real time according to the current load, processing speed and other status of ClickHouse.

[0048] S230: Select a node whose performance meets the first preset condition from the node list of the ClickHouse cluster as the first optimal node, and submit the parsed data in the buffer queue to the first optimal node.

[0049] Optionally, the performance of each node in the ClickHouse cluster node list can be evaluated at regular intervals. Therefore, when submitting data, the node whose performance meets the first preset condition can be selected from the node list based on the performance evaluation results as the first optimal node. The first optimal node can be one or more nodes. By judging the first preset condition, the node with the best performance status in the node list can be filtered out, and then the parsed data in the buffer queue can be submitted to the optimal node, that is, the first optimal node in this embodiment of the application.

[0050] Optionally, the parsed data in the buffer queue will only be submitted to the first optimal node when certain conditions are met, such as when the number of writes, size, or timeout is satisfied, the parsed data in the buffer queue will be submitted as a batch to the first optimal node.

[0051] In summary, the embodiments of this application can predefine metadata information based on the relatively fixed characteristics of log data fields, and then cache log data in different queues according to the partition key in the metadata information. This enables low-frequency, high-volume writing of logs to ClickHouse nodes, reducing the number of partitions generated by writing and the number of partition merging operations, thereby improving writing efficiency.

[0052] Based on the above embodiments, in S220, when merging buffer queues with queue lengths less than the preset length into one queue according to the preset merging time, the queue length of each buffer queue can be scanned according to the preset merging time. Based on the preset merging time, buffer queues with queue lengths less than the preset length are merged into one queue every timeout submission time.

[0053] The following will address... Figure 2 For a detailed explanation of the implementation details of S220, please refer to [link / reference]. Figure 3 Optionally, in some implementations, the process of merging buffer queues with queue lengths less than a preset length into one queue according to a preset merging time may include the following steps:

[0054] S310: Scan the queue length of each buffer queue according to the preset merging time.

[0055] A scheduled task can be set to scan the queue length of each buffer queue according to a preset merging time. Optionally, the preset merging time can be a time frequency, that is, the queue length of each buffer queue can be scanned at each preset merging time interval; alternatively, the preset merging time can also be a time point, that is, the queue length of each buffer queue can be scanned at each preset merging time.

[0056] S320: Determine the batch submission quantity, which is used to limit the maximum number of parsed data submitted to the ClickHouse cluster in each batch.

[0057] S330: Buffer queues whose queue length is less than the preset length are recorded as short queues. Based on the preset merging time, every timeout submission time, short queues whose sum of queue length is less than or equal to the number of submissions in a single batch are merged into one queue.

[0058] The batch quantity (bq) can be preset, with a default value of 100,000, or it can be updated in real time. Limiting the batch quantity improves write efficiency, while updating it enhances flexibility. In some implementations, the preset length can also be called the short queue length (sql), which can be set to 1 / 4 of the batch quantity (bq / 4) by default. Queues with a length less than the preset length are called short queues. Based on the preset merge time, every timeout commit time (stt, the default value can be set to 10 seconds, but it can also be changed in real time), queues (Q1, ..., Qn) with a length less than the short queue length are merged into a long queue, provided that sum(Q1, ..., Qn) <= batch quantity bq. This avoids the problems of large queues accumulating memory and excessively small batch sizes.

[0059] The following will address... Figure 2 For a detailed explanation of the implementation details of S230, please refer to [link / reference]. Figure 4 Optionally, in some implementations, the process of selecting a node whose performance meets a first preset condition from the node list of the ClickHouse cluster as the first optimal node may include S410-S420, and the process of submitting the parsed data in the buffer queue to the first optimal node may include S430, that is, S230 may include the following steps:

[0060] S410: Every 5 seconds, obtain the write speed and available space of each node in the node list, evaluate the load status of each node, evaluate the load status of each node based on the write speed and available space of each node, and sort all nodes in the node list according to the load status.

[0061] S420: Select the node with the best load status from the sorted list of nodes as the first optimal node.

[0062] S430: Detect the JVM memory usage and data timeout of the ClickHouse cluster according to the preset detection time. When either the JVM memory usage reaches the preset memory threshold or the data timeout reaches the timeout submission time, submit the parsed data in the buffer queue to the first optimal node.

[0063] First, a scheduled task can be set up to probe each ClickHouse node every 5 seconds, obtaining write speed and available space, assessing node load status, sorting nodes according to their assessment status, and removing unreachable nodes. The scheduled task can also check the system's JVM memory usage, buffer queue data length, and data timeout every 5 seconds. If any one of these conditions is met, a commit task is triggered. The commit task selects the optimal node from the maintained node list for writing and returns the write result. That is, when either JVM memory usage reaches a preset memory threshold or the data timeout reaches the timeout commit time, the parsed data in the buffer queue can be committed to the first optimal node. Alternatively, the queue length of each buffer queue in the ClickHouse cluster can be checked at preset detection intervals. When the queue length of any buffer queue reaches the single batch commit limit, the parsed data in the buffer queue with the queue length reaching the single batch commit limit is committed to the first optimal node.

[0064] In addition, when a submission fails, the parsed data can be submitted to other nodes in the node list. For example, a node whose performance meets the second preset condition can be selected from the node list as the second optimal node, and the parsed data in the buffer queue can be submitted to the second optimal node.

[0065] In some implementation methods, please refer to Figure 5 and Figure 6 The method for writing log data to a ClickHouse cluster can specifically include the following steps:

[0066] S1: Determine the log format specifications and predefine metadata information such as the schema.

[0067] S1-1: Utilizing relatively fixed log formats and fields, predefine schema and metadata information, including the data source (ds), cluster name (cn), table name (tn), distributed table name (dtn), table fields (Table Fields), partition key (pk), field mapping (fm), and other information required for writing.

[0068] S1-2: To improve write efficiency while maintaining flexibility, the batch submission quantity (bq, default value is 100000), short queue length (sql, default value is bq / 4), submission timeout time (stt, default value is 10 seconds), and JVM usable memory (jum, default value is 256M) can be changed in real time.

[0069] S2: Receive data, append metadata information, and add it to the buffer queue according to the partition key. Periodically merge short queues.

[0070] S2-1: First, the access process receives the data, then obtains the predefined metadata information, and populates it into the data. The data contains all the information to be written into the data table.

[0071] S2-2: Add data to different buffer queues based on the database name dn, table name tn, and partition key pk.

[0072] S2-3: Scheduled task execution queue merging. The buffer queue length is scanned periodically. If the queue exceeds a timeout commit timestt, n queues (Q1, ..., Qn) with a queue length smaller than the short queue length are merged into a long queue. The condition is that sum(Q1, ..., Qn) <= single batch commit quantity bq, to ​​avoid the problems of a large number of end queues accumulating memory and the single batch commit quantity being too small.

[0073] S3: Once the commit conditions are met, select a healthy shard to execute the commit task. If an error occurs, retry until successful.

[0074] S3-1: The scheduled task probes each ClickHouse node every 5 seconds to obtain the write speed and available space, evaluates the node load status, sorts them according to the evaluation status, and removes unreachable nodes.

[0075] S3-2: The scheduled task detects the JVM memory usage, buffer queue data length, and data timeout time of the system. If any one of these conditions is met, the task is submitted.

[0076] S3-3: Submitting a task involves selecting the optimal node from the maintained list of nodes to write to and returning the write result.

[0077] S3-4: If the write operation fails, select another node to write to until the write operation is successful.

[0078] In summary, the architecture diagram of the log processing method and apparatus based on the ClickHouse cluster proposed in this application can be found in the following embodiments. Figure 7 It can flexibly adapt to various data sources and balance writes. It converts data into a standardized format, caches it in a partition queue, and dynamically selects nodes for balanced writes. It features fast write speeds and high efficiency. This embodiment utilizes partition buffering and batch commits to suit ClickHouse's low-frequency, high-volume write characteristics, reducing the number of partitions generated during writes, thereby reducing merge operations, improving write efficiency, and reducing CPU and memory consumption.

[0079] The specific execution process and effects of the log processing method based on the ClickHouse cluster have been described in the relevant parts of the log processing method based on the ClickHouse cluster in the above embodiments, and will not be repeated here.

[0080] based on Figures 1-7 The log processing method based on ClickHouse cluster provided in the embodiments of this application has been described in detail. The following will describe in detail the virtual device corresponding to the log processing method based on ClickHouse cluster provided in the embodiments of this application, namely the log processing device based on ClickHouse cluster.

[0081] For example, Figure 8 This is a schematic diagram of the structure of the log processing device 800 based on the ClickHouse cluster provided in this application embodiment. Figure 8 As shown, the log processing device 800 based on the ClickHouse cluster includes: a metadata acquisition unit 810, a log data processing unit 820, and a data submission unit 830.

[0082] For ease of explanation, Figure 8 Only the main components of the ClickHouse cluster-based log processing apparatus 800 are shown. In some embodiments, the ClickHouse cluster-based log processing apparatus 800 is not limited to... Figure 8 The components shown may also include Figure 8 Other components besides those shown.

[0083] Metadata acquisition unit 810 is used to acquire metadata information of log data, wherein the metadata information is predetermined according to the format and fields of the log data;

[0084] The log data processing unit 820 is used to receive the log data, add the metadata information to the log data to obtain parsed data, add the parsed data to a buffer queue according to the partition key of the log data, and merge buffer queues with a queue length less than a preset length into one queue according to a preset merging time.

[0085] The data submission unit 830 is used to select a node whose performance meets a first preset condition from the node list of the ClickHouse cluster as the first optimal node, and submit the parsed data in the buffer queue to the first optimal node.

[0086] Optionally, in the above-mentioned log processing device based on ClickHouse cluster, the metadata information also includes one or more of the following: data source, cluster name, table name, distributed table name, table fields, and field mapping.

[0087] Optionally, the log data processing unit 820 may include a partitioning unit, used to add the parsed data to the corresponding buffer queue according to the database name, table name and / or partition key of the log data.

[0088] Optionally, the log data processing unit 820 may further include: a queue merging unit, used to scan the queue length of each buffer queue according to the preset merging time, and merge buffer queues with queue lengths less than the preset length into one queue every timeout submission time based on the preset merging time.

[0089] Optionally, the queue merging unit may include: a first processing unit, used to determine the number of submissions in a single batch, wherein the number of submissions in a single batch is used to limit the maximum number of parsed data submitted to the ClickHouse cluster in each batch; and a second processing unit, used to record buffer queues with queue lengths less than the preset length as short queues, and to merge short queues whose sum of queue lengths is less than or equal to the number of submissions in a single batch into one queue.

[0090] Optionally, the data submission unit 830 may include: a node evaluation unit, configured to acquire the write speed and available space of each node in the node list every 5 seconds, evaluate the load status of each node, evaluate the load status of each node based on the write speed and available space of each node, and sort all nodes in the node list according to the load status; and a node selection unit, configured to select the node with the best load status from the sorted node list as the first optimal node.

[0091] Optionally, the data submission unit 830 may further include: an optimal node first selection unit, used to detect the JVM memory usage and data timeout of the ClickHouse cluster according to a preset detection time, and when either the JVM memory usage reaches a preset memory threshold or the data timeout reaches the timeout submission time, the parsed data in the buffer queue is submitted to the first optimal node.

[0092] Optionally, the data submission unit 830 may further include: an optimal node second selection unit, used to detect the queue length of each buffer queue of the ClickHouse cluster according to a preset detection time, and when the queue length of any buffer queue reaches the single batch submission quantity, submit the parsed data in the buffer queue whose queue length reaches the single batch submission quantity to the first optimal node.

[0093] Optionally, the above-mentioned log processing device 800 based on ClickHouse cluster may further include: an error retransmission unit, used to select a node whose node performance meets a second preset condition as a second optimal node from the node list when the submission fails, and submit the parsed data in the buffer queue to the second optimal node.

[0094] Furthermore, the technical effects of the aforementioned ClickHouse cluster-based log processing device 800 can be referenced from the technical effects of the ClickHouse cluster-based log processing method described in any of the above embodiments, and will not be repeated here.

[0095] Optionally, embodiments of this application also provide a computer-readable storage medium, which includes a computer program or instructions that, when executed on a computer, cause the log processing method or training method based on the ClickHouse cluster provided in any embodiment of this application to be executed.

[0096] Optionally, embodiments of this application also provide an electronic device for executing the log processing method and apparatus based on ClickHouse cluster provided in any embodiment of this application.

[0097] like Figure 9 As shown, electronic device 2000 may include processor 2001.

[0098] Optionally, the electronic device 2000 may also include a memory 2002 and / or a transceiver 2003.

[0099] The processor 2001 is coupled to the memory 2002 and the transceiver 2003, which can be connected via a communication bus.

[0100] The following is combined Figure 9 A detailed introduction to each component of the electronic device 2000:

[0101] The processor 2001 is the control center of the electronic device 2000. It can be a single processor or a collective term for multiple processing elements. For example, the processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0102] Optionally, the processor 2001 can perform various functions of the electronic device 2000 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0103] In a specific implementation, as one example, the processor 2001 may include one or more CPUs, for example... Figure 9 CPU0 and CPU1 are shown in the diagram.

[0104] In a specific implementation, as one example, the electronic device 2000 may also include multiple processors, for example... Figure 9 The processors 2001 and 2004 are shown. Each of these processors can be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0105] The memory 2002 is used to store the software program that executes the solution of this application, and is controlled by the processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0106] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the processor 2001 or may exist independently and be accessible through the interface circuit of the electronic device 2000. Figure 9 (Not shown in the image) is coupled to processor 2001, and this embodiment does not specifically limit this.

[0107] Transceiver 2003 is used for communication with other electronic devices. For example, if electronic device 2000 is a mobile terminal, transceiver 2003 can be used to communicate with a network device or with another terminal device. As another example, if electronic device 2000 is a network device, transceiver 2003 can be used to communicate with a terminal device or with another network device.

[0108] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 9 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0109] Optionally, the transceiver 2003 can be integrated with the processor 2001, or it can exist independently and be connected via the interface circuit of the electronic device 2000. Figure 9 (Not shown in the image) is coupled to processor 2001, and this embodiment does not specifically limit this.

[0110] It should be noted that, Figure 9 The structure of the electronic device 2000 shown does not constitute a limitation on the electronic device. Actual electronic devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0111] Furthermore, the technical effects of the electronic device 2000 can be referenced from the technical effects of the log processing method based on the ClickHouse cluster described in the above method embodiments, and will not be repeated here.

[0112] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0113] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0114] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0115] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0116] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0117] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0123] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A log processing method based on a ClickHouse cluster, characterized in that, The method includes: Obtain metadata information of log data, wherein the metadata information is predetermined based on the format and fields of the log data, and the metadata information includes partition keys; The system receives the log data, adds the metadata information to the log data to obtain parsed data, adds the parsed data to the buffer queue according to the partition key, and merges the buffer queues with a queue length less than the preset length into one queue according to the preset merging time. Select the node whose performance meets the first preset condition from the node list of the ClickHouse cluster as the first optimal node, and submit the parsed data in the buffer queue to the first optimal node. Every 5 seconds, obtain the write speed and available space of each node in the node list, and evaluate the load status of each node. Evaluate the load status of each node based on the write speed and available space of each node, and sort all nodes in the node list according to the load status. Select the node with the best load status from the sorted list of nodes as the first optimal node.

2. The method according to claim 1, characterized in that, The metadata information also includes one or more of the following: data source, cluster name, table name, distributed table name, table fields, and field mapping.

3. The method according to claim 2, characterized in that, The step of adding the parsed data to the buffer queue according to the partition key of the log data includes: The parsed data is added to the corresponding buffer queue based on the database name, table name, and / or partition key of the log data.

4. The method according to claim 1, characterized in that, The step of merging buffer queues with a length less than a preset length into one queue according to a preset merging time includes: According to the preset merging time, the queue length of each buffer queue is scanned. Based on the preset merging time, buffer queues with a queue length less than the preset length are merged into one queue every timeout submission time.

5. The method according to claim 4, characterized in that, Based on the preset merging time, every timeout submission period, buffer queues with a queue length less than the preset length are merged into one queue, including: Determine the batch submission quantity, which is used to limit the maximum number of parsed data items submitted to the ClickHouse cluster in each batch; Buffer queues with a queue length less than the preset length are designated as short queues. Based on the preset merging time, every timeout submission time, short queues whose sum of queue length is less than or equal to the number of submissions in a single batch are merged into one queue.

6. The method according to claim 1, characterized in that, Submitting the parsed data in the buffer queue to the first optimal node includes: The JVM memory usage and data timeout of the ClickHouse cluster are detected according to a preset detection time. When either the JVM memory usage reaches a preset memory threshold or the data timeout reaches the timeout submission time, the parsed data in the buffer queue is submitted to the first optimal node.

7. The method according to claim 1, characterized in that, Submitting the parsed data in the buffer queue to the first optimal node includes: The queue length of each buffer queue in the ClickHouse cluster is detected according to a preset detection time. When the queue length of any buffer queue reaches the single batch submission limit, the parsed data in the buffer queue whose queue length has reached the single batch submission limit is submitted to the first optimal node.

8. The method according to claim 1, characterized in that, The method further includes: When a submission fails, a node whose performance meets the second preset condition is selected from the node list as the second optimal node, and the parsed data in the buffer queue is submitted to the second optimal node.

9. A log processing device based on a ClickHouse cluster, characterized in that, The device includes: Metadata acquisition unit, used to acquire metadata information of log data, wherein the metadata information is predetermined according to the format and fields of the log data; The log data processing unit is used to receive the log data, add the metadata information to the log data to obtain parsed data, add the parsed data to the buffer queue according to the partition key of the log data, and merge the buffer queues with a queue length less than a preset length into one queue according to a preset merging time. The data submission unit is used to select a node whose performance meets a first preset condition from the node list of the ClickHouse cluster as the first optimal node, and submit the parsed data in the buffer queue to the first optimal node. The unit obtains the write speed and available space of each node in the node list every 5 seconds, evaluates the load status of each node, evaluates the load status of each node based on the write speed and available space, and sorts all nodes in the node list according to the load status. Select the node with the best load status from the sorted list of nodes as the first optimal node.

Images