High-radix memory optimization method based on InfluxDB and improved InfluxDB

By optimizing the Compaction process and data block reading strategy, dynamically adjusting the data block size and delaying non-essential data reading, the memory consumption problem of InfluxDB in high-cardinality scenarios is solved, and the system robustness and performance are improved.

CN119003176APending Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411123282.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

InfluxDB consumes too much memory in high cardinality scenarios, resulting in performance degradation and potential crashes, and is unable to effectively deal with high cardinality data load.

Method used

By optimizing the Compaction process, data block positioning and iterator reading process, dynamically adjust the data block size and delay the reading and decompression of non-essential data, reduce memory usage, and only low-level merging is retained in high-level Compaction to avoid excessive data blocks. .

Benefits of technology

It effectively reduces the memory consumption of InfluxDB in high-cardinality scenarios, improves system robustness and write performance, and maintains read and write performance in low-cardinality scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003176A_ABST
    Figure CN119003176A_ABST
Patent Text Reader

Abstract

The invention discloses an InfluxDB-based high-radix memory optimization method and an improved InfluxDB, relates to the technical field of databases, and solves the technical problem of how to reasonably modify storage and query processes of the InfluxDB to reduce memory consumption of the InfluxDB in a high-radix scene so as to enable the InfluxDB to bear a higher-radix data load. The method comprises the following steps: reducing time sequence data volume which must reside in a memory in a high-radix query scene by optimizing a Compaction process of InfluxDB, and optimizing a data block positioning process and an iterator data reading process based on the skewness of data distribution between timelines to delay reading and decompression of unnecessary data blocks; according to the method, on the premise of keeping the read-write performance of the database in the low-radix scene, the memory resources required by the InfluxDB in the high-radix scene can be greatly reduced, and the robustness of the InfluxDB is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a high-cardinality memory optimization method based on InfluxDB and an improved InfluxDB. Background Art

[0002] Time series data refers to data points indexed in time order. These data are usually composed of continuous measurements from the same source within a fixed time interval and are used to track changes over time. Time series data consists of measurement indicators (metric), tag sets (tag set), measurement field sets (field set) and timestamps (timestamp). As an important form of data, time series data is widely used in various monitoring systems, the Internet of Things, and financial data analysis.

[0003] A time series database is a type of database used to manage time series data. It should have at least write and query operations. Some time series databases also provide update and delete operations. Write writes one or a batch of time series data points to a time series database; query queries data from a time series database and needs to support filter queries based on time ranges and tag combinations; update updates time series data with the same tag value and timestamp; delete deletes data from the database based on time ranges, tag values, and fields. Time series databases need to handle the writing of millions of data points per second. The increasing amount of time series data leads to a rapid increase in storage costs. At the same time, a large number of queries will cause the time series database to consume a large amount of memory. The memory consumption problem under high load has greatly reduced the read and write performance of InfluxDB. For some special loads, when InfluxDB executes a single query task, its memory consumption can reach tens of GB or even hundreds of GB, which will cause InfluxDB to crash in severe cases.

[0004] The characteristics of time series data, such as continuous high-throughput reading and writing and multi-dimensional metadata management, cause high cardinality problems in time series databases. Since a time series can contain multiple index columns, and each index column can contain multiple unique values, every time the index column increases, or the number of unique values ​​in an index column increases, the cardinality of the time series will increase exponentially, leading to high cardinality problems in time series databases. The high cardinality problem, on the one hand, increases the time required for the time series database to locate unique values, which not only reduces query performance, but also causes a decrease in write throughput; on the other hand, high cardinality causes the number of indexes to increase exponentially, which in turn causes the problem of excessive memory consumption.

[0005] To solve the above problems, different time series databases have made improvements in time series data models, index structures, and metadata storage architectures. These improvements usually involve compromises, such as between query efficiency and resource usage, to maximize the cardinality of time series that the time series database can manage. For example, to solve the problem of high cardinality of time series data, InfluxDB redesigned the storage engine IOx to store each tag and field as a column, reducing the total number of columns and improving performance. However, in high-cardinality scenarios, the memory consumption problem of InfluxDB affects the robustness of the runtime. Summary of the invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a high-cardinality memory optimization method based on InfluxDB and an improved InfluxDB, which solves the technical problem that the prior art cannot reasonably transform the business process of InfluxDB, reduce the memory consumption of InfluxDB in high-cardinality scenarios, and enable InfluxDB to withstand higher cardinality data loads.

[0007] A high-cardinality memory optimization method based on InfluxDB includes: reducing the amount of time series data that must reside in memory in a high-cardinality query scenario by optimizing the Compaction process, and optimizing the data block positioning process and the iterator data reading process based on the skewness of data distribution between timelines to delay the reading of unnecessary data blocks.

[0008] Furthermore, the compaction process optimization includes: dynamically controlling the data block size corresponding to each timeline when InfluxDB is written to the disk according to the timeline cardinality written to the cache at a certain moment, thereby dynamically adjusting the number of data entries read into the buffer by the iterator according to the timeline cardinality at a certain moment during query.

[0009] Furthermore, a threshold m of the occupied memory size is set for the cache flushing timing in InfluxDB. Once the sum of the data volumes of each timeline in the cache reaches this threshold, a forced flushing is triggered. In this way, in scenarios where the cardinality of parallel write timelines is high, the data volume in a single timeline will be flushed before reaching the upper limit of the original size of the data block. In scenarios where the cardinality of parallel write timelines is low, the size of the data block will not be affected. If the cardinality of concurrent write timelines is n, to reach the threshold, an average of about m / n data must be written to each timeline before early forced flushing is triggered.

[0010] Furthermore, the compaction process optimization also includes: in a high cardinality scenario, when the database performs compaction on the TSM file, canceling the compaction at the high level of the TSM tree and retaining only the compaction at the low level, so as to avoid merging data blocks at the high level and control the size of the data blocks.

[0011] Furthermore, the optimization of the data block positioning process includes: in the query initialization phase, the timeline iterator no longer obtains all data block information overlapping with the time range to be queried, but only obtains data block information containing the query start time stamp.

[0012] Furthermore, the data blocks are read in stages to synchronize the reading progress of data blocks with overlapping time ranges to prevent disorder. By utilizing the ordered characteristics of data blocks in the TSM file, after reading the data block information containing the time starting point curTs of the current time range, the information of the next data block is read, and the starting timestamp of the next data block is used as the time ending point nextTs of the current time range. After the time series data in the current time range is read, a new round of data block positioning process needs to be started. At this time, the iterator uses the value of the time ending point nextTs of the original time range as the time starting point curTs of the current new time range.

[0013] Furthermore, the optimization of the iterator data reading process includes: after the upper-level iterator is built, a pre-read is initiated. The pre-read does not actually trigger the reading of the data block, but compares the timestamps that have been read or are about to be read by the timeline iterators at the bottom of the iterator tree, and selects the earliest timestamp among the iterators. During the pre-read process, the selection result is returned to the bottom-level timeline iterator to guide the timing of the bottom-level timeline iterator to trigger the I / O operation to pre-fetch data.

[0014] Furthermore, the pre-read operation is initiated by the Merger iterator, which is used to sniff the earliest timestamp value in the timeline iterator to be sorted during the query preparation phase; the Merger iterator uses each timeline iterator as the underlying sub-iterator input, first compares the timestamp size returned by each sub-iterator, retains the earliest timestamp, and the data entry corresponding to the timestamp is used as the current return value of the Merger iterator. Subsequently, when the next method of the Merger iterator is called, the next method of the sub-iterator, i.e., the timeline iterator, is cascaded to obtain the next piece of data. When the next method of the child iterator is called, the current timestamp information of the Merger iterator is included; when there is no data in the buffer of the underlying iterator, if it is found that the difference between the current timestamp passed in by the Merger iterator and the starting timestamp curTs of the current time range of the underlying iterator is greater than the time width of a data block, then no I / O operation will be initiated to read the data block; if the difference between the timestamp passed in by the Merger iterator and the starting timestamp curTs of the current time range held by the underlying iterator is less than the time width of a data block, then the underlying iterator will initiate an I / O operation to prefetch data, and the prefetched data blocks will not be decompressed temporarily. The data blocks will be decompressed when and only when they need to be accessed.

[0015] An improved InfluxDB is applied with a high cardinality memory optimization method based on InfluxDB. In terms of data block size, a method of dynamically adjusting the data block size according to the cardinality of the timeline at each moment is used for optimization. In terms of data block pre-fetching strategy, the reading and decompression of unnecessary data in the current query phase is delayed for optimization, thereby reducing the memory consumption of InfluxDB in high cardinality scenarios.

[0016] The beneficial effects of the present invention include:

[0017] To address the problem of excessive memory usage of InfluxDB in high-cardinality scenarios, the compaction process, data block positioning process, and iterator data reading process of InfluxDB were optimized. The optimized InfluxDB system was tested to compare the read and write performance and memory usage under the same load. The test data show that the present invention can effectively reduce the memory overhead required for InfluxDB to execute query operations in high-cardinality scenarios, thereby improving the robustness of InfluxDB. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an example of skewed distribution of time series data involved in the embodiment.

[0019] Figure 2 The figure is a schematic diagram of the optimized data block positioning process involved in the embodiment.

[0020] Figure 3 The following is a diagram showing how the peak values ​​of memory usage of the existing InfluxDB and the improved InfluxDB vary with the cardinality in the embodiments.

[0021] Figure 4 The embodiment involves a query performance change of the existing InfluxDB and the improved InfluxDB with the cardinality size.

[0022] Figure 5 The write performance of the existing InfluxDB and the improved InfluxDB involved in the embodiments varies with the cardinality size. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0024] A high-cardinality memory optimization method based on InfluxDB includes: reducing the amount of time series data that must reside in memory in a high-cardinality query scenario by optimizing the Compaction process, and delaying the reading of unnecessary data blocks by optimizing the data block positioning process and the iterator data reading process based on the skewness of data distribution between timelines.

[0025] The formula for calculating memory size in high cardinality problems is: Total memory occupied = number of timelines (cardinality size) × number of data items in a single timeline × size of a single data item. Among the three variables in the formula, the size of a single data item is determined by the business needs of InfluxDB itself and cannot be changed by optimizing the algorithm or modifying the data structure. The number of timelines is determined by business needs and query conditions and cannot be changed. Since the query result return process has temporal locality, it can be optimized from the perspective of the number of data items in a single timeline read each time. Since the read operation uses data blocks as the smallest unit, there are two entry points:

[0026] 1. In terms of data block size, reduce the amount of time series data that must reside in memory in high-cardinality query scenarios. By reducing the amount of time series data that InfluxDB reads from disk each time, the memory overhead required for database buffering data can be reduced. However, this approach means that the database needs more I / O operations to complete the same query, which will lead to a decrease in query performance. In order not to affect the database read and write performance in low-cardinality scenarios, adjustments are made only in high-cardinality scenarios, thereby reducing the database's demand for memory resources while taking into account performance.

[0027] 2. In terms of data block pre-fetching strategy, optimization is performed by delaying the reading and decompression of unnecessary data in the current query phase. In some scenarios, the distribution of time series data between different timelines is skewed, such as Figure 1 As shown in the figure, the bold part indicates that the corresponding timeline has time series data within the corresponding time range. Assuming that the timelines shown in the figure are divided into the same group and returned to the upper-level iterator during the query process, InfluxDB will read the data block with the earliest timestamp in each timeline into memory for comparison by default. However, in the case of skewed time series data, theoretically only the data blocks in TimeSeries1 and TimeSeries2 need to be read at time t0.

[0028] In another embodiment, the compaction process optimization includes: dynamically controlling the data block size corresponding to each timeline when InfluxDB is written to the disk according to the timeline cardinality written to the cache at a certain moment, thereby dynamically adjusting the number of data entries read into the buffer by the iterator according to the timeline cardinality size at a certain moment during query.

[0029] In a disk I / O initiated by the timeline iterator, data is read in units of data blocks. This is because InfluxDB uses a compression algorithm with a higher compression ratio but does not support streaming decompression for disk data compression, and data compression is in units of data blocks. There are two key moments when InfluxDB writes data blocks on disk: one is to write data in the cache to the disk as a TSM file, which is called Minor Compaction; the other is that InfluxDB will merge data blocks of the same timeline in multiple TSM files and rewrite all data into a new TSM file, which is called Major Compaction. Under the default configuration of InfluxDB, a data block contains a maximum of 1,000 time series data. If there are more than 1,000 time series data to be written to the disk, these time series data will be organized into multiple data blocks and then written.

[0030] A memory usage threshold m is set for the InfluxDB Cache flushing timing. Once the sum of the data volumes of each timeline in the cache reaches this threshold, a forced flush is triggered. In this way, in scenarios where the cardinality of parallel write timelines is high, the data volume in a single timeline will be flushed before reaching the upper limit of the original size of the data block. In scenarios where the cardinality of parallel write timelines is low, the size of the data block will not be affected. If the number of concurrent cardinality is n, to reach the threshold, an average of about m / n data must be written to each timeline before early forced flushing is triggered.

[0031] Assume that a threshold of the occupied memory size (such as 50MB) is set for the InfluxDB Cache to be flushed to disk. Once the amount of data in the cache reaches this threshold, a forced flush is triggered. In the scenario where the number of concurrent write bases is high, the amount of data in each timeline will be flushed to disk before reaching the upper limit of the data block size. In the scenario where the number of concurrent write bases is low, the size of the data block will not be affected. Taking the lower number of concurrent write timeline bases of 10,000 as an example, to reach the threshold of 50MB, an average of about 5,000 data items need to be written to each timeline before triggering early flushing, which is greater than the 1,000 in the default configuration. Therefore, the data block size is still the default value and is not affected by the new policy. When the number of concurrent write timeline bases is high and greater than 50,000, the new policy will cause each timeline to be flushed to disk before writing the default configuration of 1,000 data items, resulting in a smaller data block size.

[0032] In this way, the data block size of the time series data on the disk will change dynamically with the size of the timeline cardinality contained in the database at the same time, which will affect the amount of data decompressed into the buffer at a single time during the query process, and ultimately reduce memory usage.

[0033] In another embodiment, the compaction process optimization further includes: in a high cardinality scenario, when the database performs compaction on the TSM file, canceling the compaction at the high level of the TSM tree and retaining only the compaction at the low level, avoiding merging of data blocks at the high level, so that the size of the data block is controlled.

[0034] Since InfluxDB rarely updates or deletes data at a single point, the advantages of InfluxDB in executing the compaction process are not obvious compared with relational databases or KV databases that use the LSM tree structure. The role of InfluxDB in compaction is to reduce the number of data files and merge incomplete data blocks (less than 1,000 time series data) to improve the compression rate.

[0035] Because the size of the database needs to be controlled, in order to prevent the high-level compaction process from generating large data blocks, we chose to modify the compaction plan of InfluxDB, cancel high-level compactions (such as compactions at the L3 and L4 layers), and only retain low-level compactions (such as L1 and L2 layers) to prevent data blocks from being merged at the high levels of L3 and L4 and to prevent the data block size from getting out of control.

[0036] In another embodiment, the optimization of the data block positioning process includes: in the query initialization phase, the timeline iterator no longer obtains all data block information overlapping with the time range to be queried, but only obtains data block information containing the query start time stamp.

[0037] The data block positioning process is redesigned as follows Figure 2 Although the time ranges of the time series data contained in data blocks b1, b2, and b3 overlap with the time range to be checked, only the location information of data blocks b1 and b2 is obtained in the initialization stage.

[0038] In another embodiment, the data blocks are read in stages to synchronize the reading progress of data blocks with overlapping time ranges to prevent disorder; utilizing the ordered characteristics of data blocks in the TSM file, after reading the data block information containing the time starting point curTs of the current time range, the information of the next data block is read, and the starting timestamp of the next data block is used as the time ending point nextTs of the current time range; after the time series data within the current time range is read, a new round of data block positioning process needs to be started, and at this time, the iterator uses the value of the time ending point nextTs of the original time range as the time starting point curTs of the current new time range.

[0039] Since the data to be checked may exist in the TSM file, although the time intervals of the data blocks in a single TSM file will not overlap, due to factors such as potential network latency, the data blocks may still have overlapping time ranges when the same timeline is written to different TSM files on the disk. In order to ensure that the data blocks with overlapping time ranges can be correctly merged and the correctly sorted data is returned to the upper-level iterator, the readMax and readMin member variables in the location structure are used to read the data blocks in stages. InfluxDB reads data through the information indicated by the location structure, and readMin and readMax represent the minimum and maximum timestamps that have been read, respectively. According to the values ​​set by readMin and readMax, the system continues to read the next timestamp and its corresponding data. Thanks to this, the reading range of the data block can be easily controlled.

[0040] By taking advantage of the ordered nature of data blocks in TSM files, in addition to reading the data block information containing curTs, we can read the information of another data block backwards. The starting timestamp of this data block is recorded as nextTs. In this round, the timeline iterator will return data with timestamps in the interval [curTs, nextTs) to the upper layer.

[0041] After the time series data within the current time range is read, a new round of data block positioning process needs to be started. At this time, the iterator assigns nextTs to curTs and performs a data block positioning process similar to the above, such as Figure 2 Shown in the right half.

[0042] There may be a situation where the query start timestamp is not included in any data block. For example, if the query does not specify a start timestamp, 0 is used as the start time by default. In this scenario, after the first round of data block location process is completed, it can be found that no data block has been obtained, and the nextTs timestamp indicates the earliest timestamp to be queried. At this time, you only need to execute the above process again. Since all information is provided by metadata in memory, it will not cause unnecessary I / O operations and cause query performance to deteriorate.

[0043] In another embodiment, the process of iterator reading data is optimized including: after the upper-level iterator is constructed, a pre-read is initiated, which does not actually trigger the reading of the data block, but compares the timestamps that have been read or are about to be read by each timeline iterator at the bottom of the iterator tree, and selects the earliest timestamp among the iterators. During the pre-read process, the selection result is returned to the bottom-level timeline iterator to guide the timing of the bottom-level timeline iterator to trigger the I / O operation to pre-fetch data.

[0044] In another embodiment, the pre-reading operation is initiated by the Merger iterator, which is used to sniff the earliest timestamp value in the timeline iterator to be sorted during the query preparation phase; the Merger iterator uses each timeline iterator as the underlying sub-iterator input, first compares the timestamp size returned by each sub-iterator, retains the earliest timestamp, and uses the data entry corresponding to the timestamp as the current return value of the Merger iterator. Subsequently, when the next method of the Merger iterator is called, the next method of the sub-iterator, i.e., the timeline iterator, is cascaded to obtain the next piece of data. When the next method of the child iterator is called, the current timestamp information of the Merger iterator is attached; when there is no data in the buffer of the underlying iterator, if it is found that the difference between the current timestamp passed in by the Merger iterator and the starting timestamp curTs of the current time range of the underlying iterator is greater than the time width of a data block, then no I / O operation will be initiated to read the data block; if the difference between the timestamp passed in by the Merger iterator and the starting timestamp curTs of the current time range held by the underlying iterator is less than the time width of a data block, then the underlying iterator will initiate an I / O operation to prefetch data, and the prefetched data blocks will not be decompressed temporarily, and the data blocks will be decompressed when and only when they need to be accessed.

[0045] In another embodiment, an improved InfluxDB is provided, and a high-cardinality memory optimization method based on InfluxDB is applied. In terms of data block size, optimization is performed by dynamically adjusting the data block size according to the cardinality of the timeline at each moment; in terms of data block prefetching strategy, optimization is performed by delaying the reading and decompression of unnecessary data in the current query phase, thereby reducing the memory consumption of InfluxDB in high-cardinality scenarios.

[0046] The improved InfluxDB involved in the present invention is experimentally compared with the existing InfluxDB, and the specific contents are as follows:

[0047] 1. Experimental environment

[0048] The test software and hardware environment is shown in Table 1. The test data is generated using the TDengine open source timing data generator.

[0049] Table 1 Experimental environment configuration

[0050]

[0051]

[0052] 2. High cardinality optimization performance comparison test

[0053] In high-cardinality scenarios, the following comparisons are made from the perspectives of read / write performance and memory usage:

[0054] 1. Read and write performance: In low cardinality scenarios, the read and write performance of the improved InfluxDB (M-InfluxDB) should be similar to that of the existing InfluxDB, and there should be no obvious performance degradation;

[0055] 2. Memory usage: Compared with the existing InfluxDB, M-InfluxDB can handle queries with higher cardinality under the same memory.

[0056] First, compare the changes in the peak memory usage of InfluxDB and M-InfluxDB with the cardinality. Figure 3 shown.

[0057] From the test results, we can see that M-InfluxDB occupies less memory in high cardinality scenarios. 5 When the memory usage of M-InfluxDB is only 23.9% of that of InfluxDB, the memory overhead required for operation is greatly reduced. This is because: first, the buffer size used to store decompressed data in the timeline iterator, and second, the location data structure used to locate the location of the target data block. The advantage of M-InfluxDB is that the same memory can accommodate data loads with higher cardinality. For verification, when the cardinality size is 1×10 6 In the test data set, InfluxDB's memory usage quickly increased to 60.32GB within a period of time after the query started, and then it crashed due to an OOM (OutOfMemory) error. The peak memory usage of M-InfluxDB during the entire query process was 12.66GB, and the query operation was successfully completed.

[0058] The test results of InfluxDB and M-InfluxDB are as follows: Figure 4 and 5 shown.

[0059] The query performance of M-InfluxDB is slightly lower than that of InfluxDB, and the decline increases with the increase of cardinality. This is because, in high-cardinality scenarios, the block size of time series data blocks under each time line is reduced. From the perspective of high-cardinality performance tuning, M-InfluxDB significantly reduces the memory usage during the query process at the cost of a certain loss of query performance.

[0060] Compared with query performance, the write performance of M-InfluxDB is improved compared with InfluxDB. This is because the high-level query plan of InfluxDB is cancelled, which frees up more CPU computing power and I / O bandwidth.

[0061] In summary, through the high-cardinality memory optimization method, the memory resources required by the database in a high-cardinality environment can be greatly reduced while maintaining the read and write performance of the database in low-cardinality scenarios, thereby improving the robustness of the time series database.

[0062] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A high-cardinality memory optimization method based on InfluxDB, characterized in that: include: By optimizing the compaction process, the amount of time series data that must reside in memory can be reduced in high-cardinality query scenarios. Based on the skewness of data distribution between timelines, the data block positioning process and the iterator data reading process are optimized to delay the reading and decompression of unnecessary data blocks.

2. According to claim 1, a high-cardinality memory optimization method based on InfluxDB is characterized in that: The compaction process optimization includes: dynamically controlling the data block size corresponding to each timeline when InfluxDB is written to the disk according to the timeline cardinality written to the cache at a certain moment.

3. According to a high cardinality memory optimization method based on InfluxDB according to claim 2, it is characterized in that: A threshold m of the memory size is set for the cache flushing time in InfluxDB. Once the sum of the data volume of each timeline in the cache reaches this threshold, a forced flushing is triggered.

4. According to a high cardinality memory optimization method based on InfluxDB according to claim 1, it is characterized in that: The compaction process optimization also includes: in a high cardinality scenario, when the database compacts the TSM file, canceling the compaction at the high level of the TSM tree and retaining only the compaction at the low level.

5. According to a high cardinality memory optimization method based on InfluxDB according to claim 1, it is characterized in that: The optimization of the data block positioning process includes: in the query initialization phase, the timeline iterator no longer obtains all data block information overlapping with the time range to be queried, but only obtains the data block information containing the query start time stamp.

6. According to claim 5, a high-cardinality memory optimization method based on InfluxDB is characterized in that: The data blocks are read in stages to synchronize the reading progress of data blocks with overlapping time ranges. The ordered nature of data blocks in TSM files is used to read the data block information containing the time start point curTs of the current time range, and then read the information of the next data block, and use the start timestamp of the next data block as the time end point nextTs of the current time range. After the time series data in the current time range is read, a new round of data block positioning process needs to be started. At this time, the iterator uses the value of the time end point nextTs of the original time range as the time start point curTs of the current new time range.

7. The high-cardinality memory optimization method based on InfluxDB according to claim 1 is characterized in that: The process optimization of iterator reading data includes: after the upper-level iterator is built, a pre-read is initiated, and the pre-read is to compare the timestamps that have been recently read or are about to be read by the timeline iterators at the bottom of the iterator tree, and select the earliest timestamp among the iterators as the selection result. During the pre-reading process, the selection result is sent to the bottom-level timeline iterator to guide the timing of the bottom-level timeline iterator to trigger the I / O operation to pre-fetch data.

8. The high-cardinality memory optimization method based on InfluxDB according to claim 7 is characterized in that: The pre-read operation is initiated by the Merger iterator, which is used to sniff the earliest timestamp value in the timeline iterator to be sorted during the query preparation phase. The Merger iterator uses each timeline iterator as the underlying sub-iterator input, first compares the timestamp size returned by each sub-iterator, retains the earliest timestamp, and uses the data entry corresponding to the timestamp as the current return value of the Merger iterator. Subsequently, when the next method of the Merger iterator is called, the next method of the sub-iterator, i.e., the timeline iterator, is cascaded to obtain the next piece of data. When the next method of the sub-iterator is called, the Merger iterator is attached. Current timestamp information; when there is no data in the buffer of the underlying iterator, if it is found that the difference between the current timestamp passed in by the Merger iterator and the starting timestamp curTs of the current time range of the underlying iterator is greater than the time width of a data block, then no I / O operation will be initiated to read the data block; if the difference between the timestamp passed in by the Merger iterator and the starting timestamp curTs of the current time range held by the underlying iterator is less than the time width of a data block, then the underlying iterator will initiate an I / O operation to prefetch data, and the prefetched data blocks will not be decompressed temporarily. The data blocks will be decompressed when and only when they need to be accessed.

9. An improved InfluxDB, characterized in that: A high-cardinality memory optimization method based on InfluxDB is applied as described in any one of claims 1 to 8. In terms of data block size, a method of dynamically adjusting the data block size according to the cardinality of the timeline at each moment is used for optimization. In terms of data block prefetching strategy, the reading and decompression of non-essential data in the current query phase is delayed for optimization, thereby reducing the memory consumption of InfluxDB in high-cardinality scenarios.