Data access method and device for time sequence database

Abstract

The invention provides a data access method and device for a time sequence database. The data access method includes: determining, by a host device, whether a host memory usage amount exceeds a first threshold; and based on determining that the host memory usage amount exceeds a first threshold, deleting, by the host device, one or more in-memory indices having an earliest time range in the host memory to determine that the host memory usage amount exceeds the first threshold, one or more in-memory indices in the host memory having an earliest time range are deleted by the host device.

Description

Technical Field

[0001] The present application relates to the technical field of storage databases, and more specifically, to a data access method and device for a time series database. Background Art

[0002] In order to solve the memory overflow problem of InfluxDB, the occurrence of memory overflow can be delayed by designing a reasonable data structure (for example, designing and optimizing data patterns such as tags and fields). However, this method relies on the designer's skills and cannot fundamentally eliminate the problem of memory overflow. As time goes by, the amount of data becomes larger, so memory overflow will still occur.

[0003] In addition, you can prevent memory overflow by persisting the index on disk and loading it into the host memory when you use it. Although this method can solve the memory overflow problem caused by too much index data, it will reduce the speed of data reading. In addition, when the sequence file (SeriesFile) reaches a certain size, traversing the index in the Series file and loading it into the host memory to merge it (compaction) will also cause memory overflow if the cardinality of the index data is large.

[0004] Therefore, there is an urgent need for a method and device that can improve the reading speed of InfluxDB data while preventing memory overflow. Summary of the invention

[0005] The object of the present invention is to provide a data access method and device for a time series database, so as to at least solve the problems in the above-mentioned related technologies, or not solve any of the above-mentioned problems.

[0006] According to one aspect of an embodiment of the present disclosure, a data access method for a time series database is provided, comprising: determining, by a host device, whether host memory usage exceeds a first threshold; and based on determining that the host memory usage exceeds the first threshold, deleting, by the host device, one or more in-memory indexes with the earliest time range in the host memory.

[0007] According to an embodiment of the present disclosure, by deleting an in-memory index based on whether the host memory usage exceeds a threshold, host memory overflow can be avoided.

[0008] Optionally, the method also includes: based on the host device determining that the host memory usage exceeds a first threshold, the host device determining whether there are one or more sequence indexes and one or more time series indexes TSI corresponding to the one or more in-memory indexes in the disk; and based on determining that the one or more sequence indexes and the one or more TSIs corresponding to the in-memory indexes do not exist in the disk, the computing high-speed link near memory processing CXL-PNM circuit generates the one or more sequence indexes and the one or more TSIs corresponding to the one or more in-memory indexes, and stores the generated one or more sequence indexes and the one or more TSIs in the disk.

[0009] According to an embodiment of the present disclosure, the CXL-PNM unit performs the generation of sequence index and TSI file, which can reduce the load of the host CPU.

[0010] Optionally, the method also includes: determining, by the host device, whether the host memory usage is less than a second threshold; and based on determining that the host memory usage is less than the second threshold, reloading, by the host device, the one or more in-memory indexes that were recently deleted from the host memory into the host memory, wherein the second threshold is greater than the first threshold.

[0011] According to the embodiments of the present disclosure, it is possible to ensure that hotspot data is stored in the host memory, thereby improving the reading speed of the hotspot data.

[0012] Optionally, the step of reloading the one or more in-memory indexes that were recently deleted from the host memory into the host memory includes: determining, by the host device, a difference between a deletion time of the one or more in-memory indexes that were recently deleted from the host memory and a current time; and based on determining that the difference is greater than a third threshold, reloading, by the host device, the one or more in-memory indexes that were recently deleted from the host memory into the host memory.

[0013] According to an embodiment of the present disclosure, frequent loading and deletion of indexes of the same shard can be avoided.

[0014] Optionally, the method also includes: based on determining that the timing database is restarted, the host device rebuilds one or more indexes of the current shard into the host memory, and rebuilds one or more indexes of the non-current shard into the computing high-speed link CXL memory of the CXL-PNM circuit.

[0015] According to an embodiment of the present disclosure, rebuilding the index of the non-current shard into the CXL memory of the CXL-PNM unit can reduce the load of the host CPU and reduce the host memory overflow.

[0016] Optionally, the method further includes: based on determining that the size of one or more time structure merge tree TSM files in the time series database exceeds a threshold, the CXL-PNM circuit performs compression on the one or more TSM files.

[0017] Optionally, the step of compressing the one or more TSM files by the CXL-PNM circuit includes: sending the one or more TSM files by the host device to the computing high-speed link CXL memory of the CXL-PNM circuit, compressing the one or more TSM files in the CXL memory through the near memory processing PNM engine of the CXL-PNM circuit; and storing the compressed one or more TSM files in the disk by the CXL-PNM circuit.

[0018] According to an embodiment of the present disclosure, the CXL-PNM unit performs compression of TSM files, which can reduce the host resource usage.

[0019] Optionally, each shard in the time series database has a corresponding sequence file.

[0020] Optionally, the method also includes: based on determining that the size of the sequence file corresponding to each shard exceeds a size threshold, the host device sends the sequence file corresponding to each shard to a computing high-speed link near memory processing CXL-PNM circuit; and compresses the sequence file through the CXL-PNM circuit.

[0021] According to an embodiment of the present disclosure, by performing compression of sequence files through the CXL-PNM unit, the resource occupation of the host CPU can be reduced.

[0022] Optionally, the method also includes: in response to receiving a read request for the timing database, the host device searches for an index corresponding to the read request in the host memory; based on determining that the index corresponding to the read request is not found in the host memory, the host device queries the computing high-speed link CXL memory of the CXL-PNM circuit for an index corresponding to the read request; and based on determining that the index corresponding to the read request is not found in the CXL memory, the host device queries the disk for data corresponding to the read request.

[0023] Optionally, the method further comprises: based on determining that data corresponding to the read request is found in the disk, storing, by the host device, the one or more sequence indexes and the one or more TSIs corresponding to the data in the CXL memory.

[0024] According to an embodiment of the present disclosure, by storing the sequence index and TSI corresponding to the data in the CXL memory, the reading speed of the data can be improved when it is read next time.

[0025] According to another aspect of an embodiment of the present disclosure, a data access device for a time series database is provided, comprising: a memory configured to store one or more instructions; a processor operably connected to the memory and configured to execute the one or more instructions stored in the memory, wherein the one or more instructions, when executed by the processor, cause the data access device to: determine whether host memory usage exceeds a first threshold, and based on determining that the host memory usage exceeds the first threshold, delete one or more in-memory indexes with the earliest time range in the host memory.

[0026] Optionally, when the one or more instructions are executed by the processor, the data access device causes: based on determining that the host memory usage exceeds a first threshold, to determine whether there are one or more sequence indexes and one or more time series indexes TSI corresponding to the one or more in-memory indexes on the disk; and based on determining that the one or more sequence indexes and the one or more TSIs corresponding to the one or more in-memory indexes do not exist on the disk, to generate the one or more sequence indexes and the one or more TSIs corresponding to the one or more in-memory indexes, and to store the generated one or more sequence indexes and the one or more TSIs on the disk.

[0027] Optionally, when the one or more instructions are executed by the processor, the data access device causes the data access device to: determine whether the host memory usage is less than a second threshold; and based on determining that the host memory usage is less than the second threshold, reload the one or more in-memory indexes that were recently deleted from the host memory into the host memory.

[0028] Optionally, when the one or more instructions are executed by the processor, the data access device causes the data access device to: determine the difference between a deletion time of the one or more in-memory indexes that were most recently deleted from the host memory and a current time; and based on determining that the difference is greater than a third threshold, reload the one or more in-memory indexes that were most recently deleted from the host memory into the host memory.

[0029] Optionally, when the one or more instructions are executed by the processor, the data access device causes: based on determining that the timing database is restarted, to rebuild one or more indexes of the current shard into the host memory, and to rebuild one or more indexes of the non-current shard into the computing high-speed link CXL memory of the computing high-speed link near memory processing CXL-PNM circuit.

[0030] Optionally, when the one or more instructions are executed by the processor, the data access device causes the data access device to: based on determining that the size of one or more time structure merge tree TSM files in the time series database exceeds a threshold, perform compression on the one or more TSM files.

[0031] Optionally, when the one or more instructions are executed by the processor, the data access device causes the data access device to: send the one or more TSM files to the CXL memory of the CXL-PNM circuit, wherein the CXL-PNM circuit is configured to compress the one or more TSM files in the CXL memory through a near memory processing PNM engine of the CXL-PNM circuit, and store the compressed one or more TSM files in the disk.

[0032] Optionally, each shard in the time series database has a corresponding sequence file.

[0033] Optionally, when the one or more instructions are executed by the processor, the data access device causes the data access device to: based on determining that the size of the sequence file corresponding to each shard exceeds a size threshold, send the sequence file corresponding to each shard to a computing high-speed link near memory processing CXL-PNM circuit, wherein the CXL-PNM circuit is configured to perform compression on the sequence file.

[0034] Optionally, when the one or more instructions are executed by the processor, the data access device causes: in response to receiving a read request for the timing database, to search the host memory for an index corresponding to the read request; based on determining that the index corresponding to the read request is not found in the host memory, to query the computing high-speed link CXL memory of the CXL-PNM circuit for an index corresponding to the read request; and based on determining that the index corresponding to the read request is not found in the CXL memory, to query the disk for data corresponding to the read request.

[0035] Optionally, when the one or more instructions are executed by the processor, cause the data access device to: based on determining that data corresponding to the read request is found in the disk, store the one or more sequence indexes and the one or more TSIs corresponding to the data in the CXL memory.

[0036] According to another aspect of an embodiment of the present disclosure, a non-temporary computer-readable medium storing instructions is provided, wherein, when the instructions are executed by a processor, the processor executes a method comprising the following steps: determining, by a host device, whether host memory usage exceeds a first threshold; and based on determining that the host memory usage exceeds the first threshold, deleting, by the host device, one or more in-memory indexes with the earliest time range in the host memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other objects and features of the present invention will become more apparent from the following description in conjunction with the accompanying drawings which exemplarily illustrate embodiments of the present invention, in which:

[0038] Figure 1 A flow chart of a data access method for InfluxDB according to an embodiment of the present disclosure is shown;

[0039] Figure 2 An overall schematic diagram of a data access method for InfluxDB according to an embodiment of the present disclosure is shown;

[0040] Figure 3 A schematic diagram showing compression of a Time-StructuredMerge tree (TSM) file performed by a Compute ExpressLink Processing-Near-Memory (CXL-PNM) unit according to an embodiment of the present disclosure is shown;

[0041] Figure 4 A flowchart showing a process of reading data according to an embodiment of the present disclosure is shown; Figure 5 A block diagram showing the structure of a data access device for InfluxDB according to an embodiment of the present disclosure; and

[0042] Figure 6 is a block diagram of an example computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same or similar elements, features and structures. However, it is not intended that the present disclosure be limited to specific embodiments by the various embodiments described herein, and it is intended that: the present disclosure covers all modifications, equivalents and / or substitutes of the present disclosure, as long as they are within the scope of the attached claims and their equivalents. The terms and words used in the following specification and claims are not limited to their dictionary meanings, but are only used to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only, and not for the purpose of limiting the present disclosure defined by the attached claims and their equivalents.

[0044] It should be understood that the singular includes the plural unless the context clearly indicates otherwise.The terms "include", "comprising" and "having" used herein indicate the presence of disclosed functions, operations or elements, but do not exclude other functions, operations or elements.

[0045] For example, the expression "A or B" or "at least one of A and / or B" may indicate A and B, A or B. For example, the expression "A or B" or "at least one of A and / or B" may indicate (1) A, (2) B, or (3) both A and B.

[0046] In various embodiments of the present disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to another component (e.g., a second component) or being "coupled" or "connected" to another component (e.g., the second component), the component may be directly connected to the other component or may be connected through another component (e.g., a third component). In contrast, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to another component (e.g., the second component) or being directly coupled to or directly connected to another component (e.g., the second component), there is no other component (e.g., the third component) between the component and the other component.

[0047] The expression "configured to" used in describing various embodiments of the present disclosure may be used interchangeably with expressions such as "suitable for", "having the ability to", "designed to", "suitable for", "manufactured to", and "capable of", for example, depending on the circumstances. The term "configured to" may not necessarily indicate that it is "specially designed to" in terms of hardware. On the contrary, the expression "a device configured to..." in some cases may indicate that the device and another device or part "can..." For example, the expression "a processor configured to perform A, B, and C" may indicate a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit CPU or an application processor (AP)) for performing the corresponding operations by executing at least one software program stored in a memory device.

[0048] The terms used herein are to describe certain embodiments of the present disclosure, but are not intended to limit the scope of other embodiments. Unless otherwise noted herein, all terms used herein (including technical or scientific terms) may have the same meaning as those generally understood by those skilled in the art. Typically, the terms defined in the dictionary should be considered to have the same meaning as the contextual meaning in the relevant field, and, unless clearly defined herein, should not be understood differently or understood to have an overly formal meaning. In any case, the terms defined in the present disclosure are not intended to be interpreted as excluding embodiments of the present disclosure.

[0049] The embodiments of the present disclosure refer to InfluxDB. As understood by those of ordinary skill in the art, InfluxDB is an open source time series database. InfluxDB can be used to store and retrieve time series data in various fields, such as, for example, operational monitoring, application indicators, IoT sensor data, and real-time analysis. As understood by those of ordinary skill in the art, the embodiments of the present disclosure are not limited to InfluxDB, but may also include any suitable time series database known to those of ordinary skill in the art. InfluxDB can improve query performance by building indexes. There are two types of indexes in InfluxDB: in-memory indexes and time series indexes (TSI). In-memory indexes are stored in memory, which can support tens of millions of series data, but host memory resources are limited. In order to support hundreds of millions and billions of series data, indexes are mapped to disk files to generate TSIs.

[0050] In one or more examples, for ease of description, the memory index hereinafter may be used interchangeably with the in-memory index.

[0051] Figure 1A flowchart of a data access method for InfluxDB according to an embodiment of the present disclosure is shown.

[0052] Reference Figure 1 In operation S101, the host determines whether the host memory usage exceeds a first threshold.

[0053] In operation S102, based on determining that the host memory usage exceeds a first threshold, the host deletes an in-memory index with the earliest time range in the host memory. In one or more examples, the memory usage refers to the amount of memory occupied by data. For example, if half of the memory is filled with data, the memory usage is 50%.

[0054] As understood by those skilled in the art, the memory index stored in the host memory corresponds to the shards in the disk. For example, the first in-memory index, the second in-memory index, and the third in-memory index in the host memory may correspond to the first shard, the second shard, and the third shard in the disk, respectively, and if the time range corresponding to the third shard is earlier than the time range corresponding to the first shard and the second shard, then when the host memory usage exceeds the first threshold, the third in-memory index may be deleted first.

[0055] As those skilled in the art will appreciate, as the Series cardinality increases, the remaining available host memory capacity is insufficient, which may lead to a memory overflow problem. By deleting some memory indexes in the host memory, the remaining available host memory can be increased, thereby avoiding host memory overflow.

[0056] In one or more examples, after deleting the memory index with the earliest time range, if the host memory usage still exceeds the first threshold or does not exceed the first threshold but exceeds the fourth threshold, the remaining memory indexes in the host memory can continue to be deleted. In other words, the memory indexes in the host memory can be deleted to ensure that the host memory usage is between the fourth threshold and the first threshold.

[0057] Figure 2 An overall schematic diagram of a data access method for InfluxDB according to an embodiment of the present disclosure is shown.

[0058] Reference Figure 2 , the host memory usage can be monitored by the dynamic adjuster to determine whether the host memory usage exceeds a first threshold.

[0059] In one or more examples, the dynamic adjustment machine can run on a host computer.

[0060] In one or more examples, Figure 1The method shown may also include: based on determining that the host memory usage exceeds a first threshold, determining by the host whether there is a series index (series index) and TSI corresponding to the memory index in the disk; based on determining that the sequence index and TSI corresponding to the memory index do not exist in the disk, generating a series index and TSI corresponding to the memory index by the CXL-PNM unit, and storing the generated sequence index and TSI in the disk. The CXL-PNM unit may be referred to as a CXL-PNM circuit. As understood by a person of ordinary skill in the art, the CXL-PNM unit is a high-speed interconnect, industry-standard interface for communication between processors, accelerators, memory, storage, and other IO devices. CXL improves efficiency by allowing composability, scalability, and flexibility of heterogeneous and distributed computing architectures. CXL can maintain memory consistency between the CPU memory space and the attached device memory. In one or more examples, CXL-PNM provides processing power outside the processor and near the memory module.

[0061] As will be appreciated by those skilled in the art, a sequence index indicates an index in a sequence file in a shard stored on disk, and a TSI indicates an index in a TSM file in a shard stored on disk. In one or more examples, a TSM file may be a collection of memory-mapped read-only files. A TSM file may consist of (i) a header, (ii) a block, (iii) an index, and (iv) a footer. A header may be a number identifying the file type and a version number. A block may be a sequence of pairs of CRC32 checksums and data. The block data is opaque to the file. CRC32 may be used for block-level error detection. The length of the block is stored in the index. In one or more examples, the block is followed by an index of the block in the file. The index may consist of a sequence of index entries ordered lexicographically by key and then by time. The key may include a measurement name, a tag set, and a field. In one or more examples, there is an index block entry for each block in the TSM file containing the key. The footer may store an offset from the start of the index.

[0062] As understood by those skilled in the art, the CXL-PNM unit may be a CXL-PNM card or a device or unit based on CXL-PNM technology.

[0063] Reference Figure 2 The CXL-PNM unit may include Compute Express Link (CXL) memory and a Processing-near-Memory (PNM) engine. CXL memory may be used as an extension of host memory.

[0064] Reference Figure 2, an index persisting module in the CXL-PNM unit may generate a sequence index and TSI corresponding to the memory index, and store the generated sequence index and TSI in the disk.

[0065] In one or more examples, the dynamic scheduler may further monitor the state of the PNM engine. When it is determined that the PNM engine is in an idle state, the PNM engine may generate sequence files and TSI files of non-current slices.

[0066] According to an embodiment of the present disclosure, the generation of the sequence index and the TSI is performed by the CXL-PNM unit, which can reduce the host CPU usage.

[0067] In one or more examples, Figure 1 The method shown may also include: determining by the host whether the host memory usage is less than a second threshold; based on determining that the host memory usage is less than the second threshold, reloading by the host a memory index recently deleted from the host memory into the host memory.

[0068] According to an embodiment of the present disclosure, when the remaining available space in the host memory is large (for example, the remaining available space exceeds a threshold), reloading the recently deleted memory index into the host memory can keep the frequently accessed data in the host memory as much as possible, thereby ensuring high-speed reading of hot data. In one or more examples, the hot data can be frequently accessed data (for example, data accessed more than a threshold number of times within a time interval).

[0069] In one or more examples, the step of reloading, by the host, a memory index that was recently deleted from the host memory into the host memory includes: determining, by the host, a difference between a deletion time of the memory index that was recently deleted from the host and a current time, and based on determining that the difference is greater than a third threshold, reloading, by the host, the memory index that was recently deleted from the host into the host memory.

[0070] In one or more examples, a counter may be used to determine whether a predetermined time has passed since the deletion time of a most recently deleted memory index.

[0071] According to an embodiment of the present disclosure, determining whether to rebuild the memory index based on the difference between the deletion time of the memory index and the current time can ensure that the index that has just been unloaded cannot be loaded into the host memory in a short period of time, thereby preventing frequent deletion and loading of the memory index corresponding to the same shard.

[0072] In one or more examples, Figure 1The method shown may also include: when InfluxDB is restarted, the host rebuilds the index of the current shard into the host memory, and rebuilds the index of the non-current shard into the CXL memory of the CXL-PNM unit.

[0073] As those skilled in the art will appreciate, InfluxDB can write written data to multiple shards for storage based on the retention period. For example, if InfluxDB stores a week's worth of data, each shard can only store one day's worth of data, the currently written data can be put into the current shard, the next day's data can be written into the second shard, and so on.

[0074] If the InfluxDB service is restarted, it will determine whether the shard used for writing at the current time is a new shard or a previously existing shard. Specifically, when the service is restarted, if the current time has not exceeded the time range of the existing shard, the data can be written to the latest shard that already existed. If the time exceeds the time range of the latest shard, a new shard will be created for data writing. Shards that are not currently written are all non-current shards.

[0075] In other words, the current shard indicates the shard that the current write is targeting, and the non-current shard indicates other shards stored in InfluxDB.

[0076] As those skilled in the art understand, the merging of TSM files in the related art is performed by the host, which increases the CPU and IO loads for a short period of time, and the duration varies depending on the size of the data. During the merging (e.g., compression), the speed of database writing and reading is greatly affected.

[0077] In one or more examples, when TSM files in InfluxDB need to be merged, the CXL-PNM unit performs a merge on the TSM files that need to be merged. In one or more examples, if the size of the TSM file exceeds a size threshold, it is determined that the TSM file needs to be merged (e.g., compressed).

[0078] In one or more examples, the step of merging the TSM files that need to be merged by the CXL-PNM unit includes: sending the TSM files to the CXL memory of the CXL-PNM unit through the host, merging the TSM files in the CXL memory through the PNM engine of the CXL-PNM unit, and storing the merged TSM files to the disk by the CXL-PNM unit.

[0079] Figure 3FIG. 4 is a schematic diagram showing a TSM file merge performed by a CXL-PNM unit according to an embodiment of the present disclosure.

[0080] Reference Figure 3 When a TSM file in the Nth shard triggers a merge, the data in the TSM file can be loaded into the CXL memory, and the data in the TSM file can be merged by, for example, a compactor in the CXL-PNM unit.

[0081] According to an embodiment of the present disclosure, by performing merging (eg, compression) of TSM files through a CXL-PNM unit, CPU occupancy can be reduced and data transmission costs between the CPU and the memory can be reduced.

[0082] In one or more examples, each shard in InfluxDB has a corresponding series file (SeriesFile).

[0083] In one or more examples, in the process of storing data, a sequence file may be generated according to the slices.

[0084] As will be appreciated by those skilled in the art, reference Figure 2 , all shards in the prior art correspond to one SeriesFile. In contrast, according to an embodiment of the present disclosure, each shard has a corresponding SeriesFile. For example, in the prior art, N shards correspond to one SeriesFile, while according to an embodiment of the present disclosure, N shards correspond to N SeriesFiles. For example, each shard has its own sequence file.

[0085] As SeriesFile grows, SeriesFile may also need to be merged.

[0086] In one or more examples, Figure 1 The method shown may further include: based on determining that the sequence files corresponding to each slice need to be merged, the host sends the sequence files corresponding to each slice to the CXL-PNM unit, and the CXL unit merges the sequence files.

[0087] According to an embodiment of the present disclosure, the data volume of the SeriesFile at the shard level is smaller than the data volume of the SeriesFile at the database level. Therefore, merging the SeriesFile through the host or the CXL-PNM unit can avoid a large amount of resource occupation and / or memory overflow.

[0088] In one or more examples, Figure 1The method shown may also include: in response to receiving a read request for the InfLUXDB, the host searches for an index corresponding to the read request in the host memory; based on not finding the index query corresponding to the read request in the host memory, the host queries the CXL memory for an index corresponding to the request; based on not finding the index in the CXL memory, the host queries the disk for data corresponding to the read request.

[0089] In one or more examples, Figure 1 The method further includes: based on finding the data corresponding to the request in the disk, the host storing a sequence index and a TSI corresponding to the data in the CXL memory.

[0090] According to an embodiment of the present disclosure, since the sequence index and TSI corresponding to the data are stored in the CXL memory, when the data is read next time, the index corresponding to the data can be quickly found in the CXL memory, thereby improving the reading speed of the data.

[0091] In one or more examples, index data in the CXL memory can be evicted via a Least Recently Used (LRU) algorithm.

[0092] In one or more examples, index data with the oldest time range in the CXL memory may be deleted preferentially.

[0093] Figure 4 A flowchart of a process of reading data from InfluxDB according to one or more embodiments of the present disclosure is shown.

[0094] Reference Figure 4 In operation 401, it is determined whether the Series ID corresponding to the read request is in the host memory (eg, DRAM). If the Series ID is in the host memory, the process proceeds to operation S407, otherwise, the process proceeds to operation S402.

[0095] In operation S402, it is determined whether the SeriesID corresponding to the read request is in the CXL memory. If the SeriesID is in the CXL memory of the CXL-PNM device, the process proceeds to operation S407, otherwise, the process proceeds to operation S403.

[0096] In operation S403, based on the time information included in the read request, it is determined in which shard the to-be-read data corresponding to the read request is located.

[0097] In operation S404, a SeriesID and a Serieskey are obtained from the determined shard.

[0098] In operation S405 , the host sends the TSI and series index corresponding to the read data to the CXL memory.

[0099] In operation S406, the data is read from the TSM file of the determined shard.

[0100] In operation S407 , data aggregation is performed and the data is returned.

[0101] According to an embodiment of the present disclosure, in the process of searching for a key by SeriesID, if there is no hit in the memory, the entire disk data needs to be traversed. According to an embodiment of the present disclosure, since the sequence file is stored in fragments, the data query range can be narrowed, thereby improving the data reading speed.

[0102] Reference above Figures 1 to 4 The data access method for InfluxDB according to the embodiment of the present disclosure is described below. Figure 5 A data access device for InfluxDB according to an embodiment of the present disclosure is described.

[0103] Figure 5 A block diagram showing the structure of a data access device 500 for InfluxDB according to an embodiment of the present disclosure is shown.

[0104] Reference Figure 5 , the data access device 500 may include a first determining unit 501 and a deleting unit 502 .

[0105] Those skilled in the art should understand that the data access device 500 may further include other components, and at least one of the components included in the data access device 500 may be combined or split.

[0106] In one or more examples, the first determining unit 501 may be configured to determine whether the host memory usage exceeds a first threshold.

[0107] In one or more examples, the deleting unit 502 may be configured to: based on determining that the host memory usage exceeds a first threshold, delete an in-memory index having an earliest time range in the host memory.

[0108] In one or more examples, the data access device 500 also includes: a second determination unit, configured to determine whether there is a sequence index and TSI corresponding to the index in the memory in the disk based on determining that the host memory usage exceeds a first threshold; and a CXL-PNM unit, configured to generate a sequence index and TSI corresponding to the memory index based on determining that the sequence index and TSI corresponding to the memory index do not exist in the disk, and store the generated sequence index and TSI in the disk.

[0109] In one or more examples, the first determining unit 501 is further configured to determine whether the host memory usage is less than a second threshold.

[0110] In one or more examples, the deleting unit 502 is further configured to: based on determining that the host memory usage is less than a second threshold, reload the in-memory index that was recently deleted from the host memory into the host memory.

[0111] In one or more examples, the deletion unit 502 is configured to: determine the difference between a deletion time of a memory index that was most recently deleted from the host memory and a current time; and based on determining that the difference is greater than a third threshold, reload the memory index that was most recently deleted from the host memory into the host memory.

[0112] In one or more examples, the data access device 500 further includes: a reconstruction unit configured to rebuild the index of the current shard into the host memory and rebuild the index of the non-current shard into the CXL memory of the CXL-PNM unit when InfluxDB is restarted.

[0113] In one or more examples, the CXL-PNM unit is further configured to: when TSM files in InfluxDB need to be merged, merge the TSM files that need to be merged. For example, when the size of the TSM file exceeds a size threshold, it can be determined that the TSM file may need to be merged (e.g., compressed).

[0114] In one or more examples, the data access device 500 further includes: a sending unit configured to send the TSM file to a CXL memory of a CXL-PNM, and wherein the CXL-PNM unit is configured to: merge the TSM files in the CXL memory through a PNM engine of the CXL-PNM unit, and store the merged TSM files in the disk.

[0115] In one or more examples, each shard in InfluxDB has a corresponding sequence file.

[0116] In one or more examples, the sending unit is further configured to: based on determining that the sequence files corresponding to each slice need to be merged, send the sequence files corresponding to each slice to the CXL-PNM unit, wherein the CXL-PNM unit is further configured to perform merging on the sequence files.

[0117] In one or more examples, the data access device 500 also includes a search unit, which is configured to: in response to receiving a read request for the InfLUXDB, search for an index corresponding to the read request in the host memory; based on not finding the index query corresponding to the read request in the host memory, query the CXL memory of the CXL-PNM unit for the index corresponding to the read request; based on not finding the index corresponding to the read request in the CXL memory, query the disk for data corresponding to the read request.

[0118] In one or more examples, the data access device further includes a storage unit configured to: based on finding the data corresponding to the request in the disk, store a sequence index and a TSI corresponding to the data in the CXL memory.

[0119] According to one or more embodiments of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor executes the data access method for InfluxDB according to the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0120] Figure 6 is a block diagram of an example computer system according to an embodiment of the present disclosure.

[0121] Figure 6 yes Figure 1 The device 600 may correspond to the data access device 500 ( Figure 5 ).like Figure 6 As shown, the apparatus 600 may include a bus 610 , a processor 620 , a memory 630 , a storage component 640 , an input component 650 , an output component 660 , and a communication interface 670 .

[0122] The bus 610 includes components that allow communication between components of the device 600. The processor 620 is implemented in hardware, firmware, or a combination of hardware and software. The processor 620 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other types of processing components. In some embodiments, the processor 620 includes one or more processors that can be programmed to perform functions. The memory 630 includes a random access memory (RAM), a read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) that store information and / or instructions used by the processor 620.

[0123] The storage component 640 stores information and / or software related to the operation and use of the device 600. For example, the storage component 640 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or other types of non-transitory computer-readable media, and corresponding drives.

[0124] Input components 650 include components that permit device 600 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 650 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 660 include components that provide output information from device 600 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0125] The communication interface 670 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 600 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 670 may allow the device 600 to receive information from another device and / or provide information to another device. For example, the communication interface 670 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0126] Device 600 may perform one or more processes described herein. Device 600 may perform these processes in response to processor 620 executing software instructions stored by non-transitory computer-readable media (such as memory 630 and / or storage component 640). Computer-readable media is defined herein as non-transitory memory devices. Memory devices include memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0127] The software instructions may be read into the memory 630 and / or storage component 640 from another computer-readable medium or from another device via the communication interface 670. When executed, the software instructions stored in the memory 630 and / or storage component 640 may cause the processor 620 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0128] Figure 6 The number and arrangement of components shown in FIG. 1 are provided as examples. In practice, Figure 6 Compared to those components shown in , device 600 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components (e.g., one or more components) of device 600 may perform one or more functions described as being performed by another set of components of device 600.

[0129] According to one or more embodiments of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to implement the data access method for a time series database described herein.

[0130] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

Claims

1. A data access method for a time series database, comprising: Determining, by the host device, whether the host memory usage exceeds a first threshold; as well as Based on determining that the host memory usage exceeds a first threshold, one or more in-memory indexes having an earliest time range in the host memory are deleted by the host device.

2. The data access method according to claim 1, further comprising: Based on the host device determining that the host memory usage exceeds a first threshold, the host device determines whether there are one or more sequence indexes and one or more time series indexes TSI corresponding to the one or more in-memory indexes in the disk; as well as Based on determining that the one or more sequence indexes and the one or more TSIs corresponding to the in-memory indexes do not exist in the disk, the computing high-speed link near memory processing CXL-PNM circuit generates the one or more sequence indexes and the one or more TSIs corresponding to the one or more in-memory indexes, and stores the generated one or more sequence indexes and the one or more TSIs in the disk.

3. The data access method according to claim 1, further comprising: Determining, by the host device, whether the host memory usage is less than a second threshold; as well as Based on determining that the host memory usage is less than a second threshold, reloading, by the host device, the one or more in-memory indexes that were recently deleted from the host memory into the host memory, The second threshold is greater than the first threshold.

4. The data access method according to claim 3, wherein: The step of reloading the one or more in-memory indexes that were recently deleted from the host memory into the host memory comprises: Determining, by the host device, a difference between a deletion time of the one or more in-memory indexes most recently deleted from the host memory and a current time; and Based on determining that the difference is greater than a third threshold, the one or more in-memory indexes that were recently deleted from the host memory are reloaded into the host memory by the host device.

5. The data access method according to claim 2, further comprising: Based on determining that the timing database is restarted, the host device rebuilds one or more indexes of the current shard into the host memory, and rebuilds one or more indexes of the non-current shard into the computational high-speed link CXL memory of the CXL-PNM circuit.

6. The data access method according to claim 1, further comprising: Based on determining that the size of one or more temporal structure merge tree TSM files in the time series database exceeds a threshold, the CXL-PNM circuit performs compression on the one or more TSM files.

7. The data access method according to claim 6, wherein: The step of performing compression on the one or more TSM files by the CXL-PNM circuit comprises: Sending the one or more TSM files to a computational express link CXL memory of the CXL-PNM circuit by the host device, and performing compression on the one or more TSM files in the CXL memory by a near memory processing PNM engine of the CXL-PNM circuit; and The one or more TSM files are compressed and stored in the disk by the CXL-PNM circuit.

8. The data access method according to claim 1, wherein: Each shard in the time series database has a corresponding sequence file.

9. The data access method according to claim 8, further comprising: Based on determining that the size of the sequence file corresponding to each shard exceeds the size threshold, the host device sends the sequence file corresponding to each shard to the computing high-speed link near memory processing CXL-PNM circuit; and Compression is performed on the sequence file by the CXL-PNM circuit.

10. The data access method according to claim 2, further comprising: In response to receiving a read request for the time series database, the host device searches the host memory for an index corresponding to the read request; Based on determining that the index corresponding to the read request is not found in the host memory, the host device queries the computing high-speed link CXL memory of the CXL-PNM circuit for the index corresponding to the read request; as well as Based on determining that the index corresponding to the read request is not found in the CXL memory, the host device searches the disk for data corresponding to the read request.

11. The data access method according to claim 10, further comprising: Based on determining that the data corresponding to the read request is found in the disk, the host device stores the one or more sequence indexes and the one or more TSIs corresponding to the data in the CXL memory.

12. A data access device for a time series database, comprising: a memory configured to store one or more instructions; a processor operably connected to the memory and configured to execute the one or more instructions stored in the memory, Wherein, the one or more instructions, when executed by the processor, cause the data access device to: A determination is made as to whether host memory usage exceeds a first threshold, and based on determining that the host memory usage exceeds the first threshold, one or more in-memory indexes having an earliest time range in the host memory are deleted.

13. A non-transitory computer-readable medium storing instructions, wherein: When the instructions are executed by a processor, the processor is caused to execute a method comprising the following steps: Determining, by the host device, whether the host memory usage exceeds a first threshold; as well as Based on determining that the host memory usage exceeds a first threshold, one or more in-memory indexes having an earliest time range in the host memory are deleted by the host device.

Images