Time series data processing method and device, program product and electronic equipment

Abstract

The invention discloses a time sequence data processing method and device, a computer program product and electronic equipment, and relates to the technical field of data processing. The method comprises the steps that first time sequence data to be stored are obtained, and the first time sequence data comprise a key field, a value field and a timestamp field; the key field comprises a time period field; and storing the first time sequence data, wherein the first time sequence data and the stored time sequence data of the same time period field are merged and stored as a key value pair containing a timestamp field. According to the method and the device, the time series data with the same time period fields are merged and stored, so that the data storage cost can be reduced to a certain extent, and the data reading efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology. More specifically, the embodiments of the present disclosure relate to a time series data processing method, a time series data processing device, a computer program product, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Time series data is a series of data based on time, with characteristics such as data append writing, high traffic throughput, large data scale, and multi-dimensional label retrieval.

[0004] In the related technology, the B+ tree storage structure is used for time series data, which will lead to a large number of random writes, very low reading efficiency, and a large amount of lock overhead. In the scenario of large-volume writing, it will limit the bottleneck of system performance.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a time series data processing method, a time series data processing device, a computer program product and an electronic device, thereby at least to a certain extent solving the problem of high storage cost and low reading efficiency of time series data in the related art.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a time series data processing method is provided, which obtains first time series data to be stored, wherein the first time series data includes a key field, a value field, and a timestamp field; the key field includes a time period field; the first time series data is stored, and the first time series data is merged with the stored time series data of the same time period field and stored as a key-value pair including a timestamp field.

[0009] In an exemplary embodiment of the present disclosure, storing the first time series data includes: storing the first time series data in a memory buffer; if the memory buffer is greater than a first threshold, converting the memory buffer into an immutable memory buffer; if the immutable memory buffer is greater than a second threshold, refreshing the first time series data stored in the immutable memory buffer to a data file on a disk.

[0010] In an exemplary embodiment of the present disclosure, before writing the first time series data into a memory buffer, the method further includes: writing the first time series data into a write-ahead log, wherein the write-ahead log is used to recover the time series data when the time series data is lost.

[0011] In an exemplary embodiment of the present disclosure, the key field further includes a tag field, and the method further includes: establishing an inverted index for the tag field corresponding to the first time series data, and storing the inverted index in an index file on a disk.

[0012] In an exemplary embodiment of the present disclosure, the method further includes: searching the stored time series data according to the field information to be retrieved to obtain second time series data.

[0013] In an exemplary embodiment of the present disclosure, the field information to be retrieved includes a label field to be retrieved, and the stored time series data is retrieved according to the field information to be retrieved to obtain the second time series data, including: determining the key field corresponding to the label field to be retrieved by looking up the inverted index corresponding to the label field to be retrieved; and retrieving the stored time series data according to the key field corresponding to the label field to be retrieved to obtain the second time series data.

[0014] In an exemplary embodiment of the present disclosure, the stored time series data is retrieved according to the key field corresponding to the tag field to be retrieved to obtain the second time series data, including: determining a target key-value pair including a timestamp field from the stored time series data according to the key field corresponding to the tag field to be retrieved; and aggregating the value fields corresponding to each timestamp field in the target key-value pair to obtain the second time series data.

[0015] According to the second aspect of the present disclosure, a time series data processing device is disclosed, which includes: a time series data acquisition module, used to acquire first time series data to be stored, the first time series data including a key field, a value field, and a timestamp field; the key field includes a time period field; a time series data storage module, used to store the first time series data, the first time series data and the stored time series data of the same time period field are merged and stored as a key-value pair including a timestamp field.

[0016] According to a third aspect of an embodiment of the present disclosure, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the time series data processing method of the first aspect and its possible implementation methods.

[0017] According to the fourth aspect of an embodiment of the present disclosure, an electronic device is disclosed, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the timing data processing method of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0018] The technical solution disclosed in this disclosure has the following beneficial effects:

[0019] In the above-mentioned time series data processing process, the first time series data to be stored is obtained, and the first time series data includes a key field, a value field, and a timestamp field; the key field includes a time period field; the first time series data is stored, and the first time series data and the stored time series data of the same time period field are merged and stored as a key-value pair including a timestamp field. In the present disclosure, when storing time series data, the time series data of the same time period field are merged and stored, which can reduce the cost of data storage to a certain extent, and the data within a specified time period range can be queried without reading a large number of data records, which can improve the efficiency of data reading to a certain extent.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic flow chart showing a method for processing time series data in this exemplary embodiment;

[0023] Figure 2 A schematic diagram of a structure of writing sequential data in this exemplary embodiment is shown;

[0024] Figure 3 A flowchart of a time series data storage in this exemplary embodiment is shown;

[0025] Figure 4 A schematic diagram showing a structure of time series data retrieval in this exemplary embodiment is shown;

[0026] Figure 5 A schematic diagram showing a flow chart of time series data retrieval in this exemplary embodiment is shown;

[0027] Figure 6 A system architecture diagram of time series data processing in this exemplary embodiment is shown;

[0028] Figure 7 A structural block diagram of a time series data processing device in this exemplary embodiment is shown;

[0029] Figure 8 An electronic device for implementing the above-mentioned time series data processing method in this exemplary embodiment is shown.

[0030] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0031] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0033] According to an embodiment of the present disclosure, a time series data processing method, a time series data processing device, a computer program product and an electronic device are proposed.

[0034] Any number of elements in the drawings is for illustration and not limitation, and any naming is for distinction only and does not have any limiting meaning.

[0035] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0036] In the related technologies disclosed in the present invention, a B+ tree storage structure is used for time series data, which will result in a large amount of random writes, very low reading efficiency, and a large amount of lock overhead. In the scenario of large-volume writes, it will limit the bottleneck of system performance.

[0037] Based on the above problems, the present invention proposes a time series data processing method. When storing time series data, the time series data of the same time period field are merged and stored, which can reduce the cost of data storage to a certain extent. Moreover, there is no need to read a large number of data records to query the data in a specified time period range, which can improve the efficiency of data reading to a certain extent.

[0038] Application Scenario Overview

[0039] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario. For example, it can be applied to application scenarios related to time series data such as smart cities, the Internet of Things, the Internet of Vehicles, and the Industrial Internet, and the present disclosure does not specifically limit this.

[0040] Exemplary Methods

[0041] In combination with the above application scenarios, refer to Figure 1 A time series data processing method according to an exemplary embodiment of the present disclosure is described.

[0042] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for processing time series data according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the time series data processing method may include:

[0043] Step S110, obtaining first time series data to be stored, the first time series data including a key field, a value field, and a timestamp field; the key field includes a time period field;

[0044] Step S120: store the first time series data, and merge the first time series data with the stored time series data of the same time period field and store them as a key-value pair including a timestamp field.

[0045] Implementation Figure 1 The time series data processing method shown in the figure merges and stores the time series data of the same time period field when storing the time series data, which can reduce the cost of data storage to a certain extent, and can query the data in the specified time period range without reading a large number of data records, which can improve the efficiency of data reading to a certain extent.

[0046] These steps are described in detail below.

[0047] In step S110, first time series data to be stored is obtained, where the first time series data includes a key field, a value field, and a timestamp field; the key field includes a time period field.

[0048] Among them, time series data refers to data that needs to be recorded in chronological order, and each piece of time series data can correspond to a time point.

[0049] The first time series data refers to the time series data that needs to be stored currently, which may include but is not limited to multiple fields such as a key field, a value field, and a timestamp field. Exemplarily, the data format corresponding to the first time series data may be {key, ts, val}, where key is used to represent the key field; ts is used to represent the timestamp field; and val is used to represent the value field.

[0050] Optionally, the key field may include: a time period field, wherein the time period field may be used to record the time period for generating the time series data.

[0051] Optionally, when the time period field is used to record the time period for generating time series data, the time period for generating time series data can be represented by the baseline time of the time period for generating time series data, for example, the start time, end time, middle time, etc. of the time period, so as to reduce storage overhead, which is not specifically limited in this disclosure. For example, the time period of 10:00:00 to 11:00:00 can be recorded as 10:00:00.

[0052] Optionally, the key field may also include: a metric field, a tag field, and a time period field. Exemplarily, the data format corresponding to the key field may be:<metricxs,basetime,tagkvs> Among them, metr icxs is used to represent the metric field set, which may include one or more metric fields; basetime is used to represent the time period field; tagkvs is used to represent the tag field set, which may include one or more tag fields.

[0053] Among them, the metric field can be used to record the collection index items. The tag field can be used to record the attribute information corresponding to the time series data that does not change with the timestamp. Taking the CPU (Central Processing Unit) monitoring scenario as an example, the metric field can be, for example, the CPU usage rate, and the tag field can be, for example, the name of the computer room that generates the time series data, the host name that generates the time series data, etc., which is not specifically limited in this disclosure.

[0054] Optionally, the value field can be used to record the specific value collected. Taking the monitoring scenario as an example, the value field can be, for example, the collected host CPU usage value.

[0055] Optionally, the timestamp field can be used to record the time point when the time series data is generated, such as the date when the data is generated.

[0056] In step S120, the first time series data is stored, and the first time series data is merged with the stored time series data of the same time period field and stored as a key-value pair including a timestamp field.

[0057] Optionally, the stored time series data refers to time series data that has been stored. Exemplarily, these time series data can be pre-stored through a storage structure of LSM (Log-Structured-Merge-Tree). LSM is a common database storage model suitable for key-value storage.

[0058] Optionally, after obtaining the first time series data, the first time series data can be written in an appended form to the key-value pair corresponding to the stored time series data of the same time period field. This can not only ensure the writing performance and reduce the number of keys, but also the data points of the same time period field can be read once, which can improve the data reading efficiency to a certain extent.

[0059] In an optional implementation, the above-mentioned storage of the first time series data can be achieved through the following steps: storing the first time series data in a memory buffer; if the memory buffer is greater than a first threshold, converting the memory buffer into an immutable memory buffer; if the immutable memory buffer is greater than a second threshold, refreshing the first time series data stored in the immutable memory buffer to a data file on the disk.

[0060] For example, Figure 2 As shown, a structural diagram of timing data writing is provided. The first timing data {key, ts, val} can be stored in a memory buffer (memory buffer). Optionally, the memory buffer can use a jump table structure to implement caching. If the memory buffer is greater than a first threshold, the memory buffer is converted into an immutable memory buffer (immutable memory buffer). If the immutable memory buffer is greater than a second threshold, the first timing data stored in the immutable memory buffer is refreshed to the data file (Data File) on the disk. Among them, the first threshold is used to limit the critical value of the memory buffer being converted into an immutable memory buffer, and the second threshold is used to limit the critical value of the immutable memory buffer being refreshed to the data file on the disk, which can be specifically set by the developer, and the present disclosure does not specifically limit this.

[0061] By gradually storing time series data to disk through memory buffers, immutable memory buffers, and disk data files, time series data can be stored for a long time.

[0062] In an optional implementation, the first time series data is written into a write-ahead log, and the write-ahead log is used to recover the time series data when the time series data is damaged or lost.

[0063] For example, Figure 2As shown, before writing the first time series data {key, ts, val} into the memory buffer, the first time series data may be first written into a write-ahead log.

[0064] Write-ahead logs can be used to recover time series data when it is damaged or lost. Writing the time series data to be stored into the write-ahead log in advance can prevent data from being damaged or lost in abnormal scenarios, which may lead to data not being stored correctly. Abnormal scenarios can include extreme abnormal scenarios such as process crashes, machine crashes, or power outages.

[0065] In an optional implementation, when the key field includes a tag field, the following steps may also be performed: creating an inverted index for the tag field corresponding to the first time series data, and storing the inverted index in an index file on a disk.

[0066] Optionally, a key field may include one or more tag fields.

[0067] Among them, the inverted index is an indexing method that can quickly locate the key field containing the tag field by mapping each tag field in the key field to the key field containing the tag field. Optionally, the established inverted index can be stored in the index file in the form of key-value pairs to improve the reading efficiency of time series data.

[0068] For example, Figure 2 As shown, the key value representation of the inverted index can be<tagtv_1+key_1,v1> ,<tagtv_2+key_2,v2> ,..., etc. Wherein, tagtv_1 is a tag field contained in the key field key_1, and tagtv_2 is a tag field contained in the key field key_2. Wherein, tagtv_1+key_1, tagtv_2+key_2 can be regarded as the keys of the inverted index; v1, v2 can be regarded as the values ​​of the inverted index, which can record the value of the value field corresponding to the corresponding key field, the storage address corresponding to the corresponding key field, etc., and can also record other information, which is not specifically limited in this disclosure.

[0069] For example, the key value representation of the inverted index can also be<tagtv_0,key_1+key_2+...+key_m> , where tagtv_0 is a tag field common to key fields key_1, key fields key_2, ..., key fields key_m, and m is the number of key fields including tag field tagtv_0.

[0070] By performing a key value scan on the index file, you can query the inverted index and find the key field that contains a specific tag field.

[0071] For example, Figure 2 As shown, when storing time series data, the time series data with the same time period field can be merged in the background so that the time series data with the same time period field is stored in the same key-value record to reduce the number of key-value pairs and improve data reading efficiency.

[0072] Exemplarily, the first time series data is {key, ts, val}, where key=<metricxs,basetime,tagkvs> , the {ts, val} in the first time series data can be appended to the end of the {ts, val} of the stored time series data in the same time period field as the first time series data.

[0073] Exemplarily, the first time series data is {key0, ts0, val0}, where key0=<metricxs,base time,tagkvs0> , the stored time series data of the same period field as the first time series data is {key', ts', val'}, where key' =<metricxs,basetime,tagkvs'> When merging the first time series data with the stored time series data of the same time period field, tagkvs0 in the first time series data can be appended to tagkvs', and {ts0,val0} in the first time series data can be appended to {ts',val'}, to obtain<key”,[ts',val'][ts0,val0]> , where "key" =<metricxs,basetime,tagkvs'+tagkvs0> It should be noted that, in the above exemplary implementation, key0, ts0, and val0 may respectively represent the key field, timestamp field, and value field of the first time series data, and tagkvs0 represents the tag field set contained in the key field of the first time series data; key', ts', and val' respectively represent the key field, timestamp field, and value field of the stored time series data, and tagkvs' represents the tag field set contained in the key field of the stored time series data.

[0074] For example, Figure 3 As shown, a flowchart of time series data storage is provided, which may specifically include the following steps:

[0075] Step S301, obtaining first time series data to be stored, the first time series data including a key field, a value field, and a timestamp field; the key field includes a metric field, a time period field, and a tag field;

[0076] Step S302, writing the first time series data into a write-ahead log, where the write-ahead log is used to recover the time series data when the time series data is damaged or lost;

[0077] Step S303, storing the first time series data in a memory buffer; if the memory buffer is greater than a first threshold, converting the memory buffer into an immutable memory buffer;

[0078] Step S304, if the immutable memory buffer is greater than the second threshold, the first time series data is refreshed to the data file on the disk by merging the first time series data with the stored time series data of the same time period field, and an inverted index is established for the label field corresponding to the first time series data, and the inverted index is stored in the index file on the disk.

[0079] In an optional implementation, the following steps may also be performed: based on the field information to be retrieved, the stored time series data is retrieved to obtain the second time series data.

[0080] The second time series data refers to the time series data retrieved through data retrieval.

[0081] The field information to be searched may be the field information input by the user that needs to be searched.

[0082] Exemplarily, the field information to be retrieved may include a time period field to be retrieved. According to the time period field to be retrieved, time series data matching the time period field to be retrieved may be queried from a memory buffer, an immutable memory buffer, and / or a disk data file.

[0083] In the present disclosure, the time series data of the same time period field are merged and stored. When searching by the time period field, it can be completed in one reading, which can improve the reading efficiency of the time series data to a certain extent.

[0084] In an optional implementation, the field information to be retrieved may also include a tag field to be retrieved. The above-mentioned retrieval of the stored time series data according to the field information to be retrieved to obtain the second time series data may be achieved through the following steps: determining the key field corresponding to the tag field to be retrieved by searching the inverted index corresponding to the tag field to be retrieved; retrieving the stored time series data according to the key field corresponding to the tag field to be retrieved to obtain the second time series data.

[0085] For example, Figure 4 As shown, a schematic diagram of the structure of time series data retrieval is provided, in which the tag fields tagkv1, tagkv2, ... to be retrieved can be searched in the index file to obtain the key fields key1, key2, ..., keyn corresponding to the tag fields to be retrieved, where n is the number of key fields found. Among them, the index file can use an inverted index to store the mapping relationship between the tag field and the key field.

[0086] Furthermore, the searched key fields may be filtered by intersection to avoid subsequent repeated searches of the same key fields, thereby further improving search efficiency.

[0087] Exemplarily, according to the key field corresponding to the tag field to be retrieved, the time series data matching these key fields can be retrieved from the memory buffer, the immutable memory buffer, and the data file to obtain the second time series data. The data file stores the time series data in the form of key values.

[0088] By reading the value fields of all key fields corresponding to the tag field to be retrieved, the second time series data is obtained, which can reduce the amount of data retrieved by tag to a certain extent and improve the retrieval efficiency.

[0089] Exemplarily, the key fields corresponding to the label fields to be retrieved can also be merged according to the time period fields to obtain multiple synthetic key fields, and based on the synthetic key fields, the time series data matching the synthetic key fields can be retrieved from the data file to obtain the second time series data, so as to reduce the search objects and further improve the search efficiency.

[0090] In an optional embodiment, the above-mentioned retrieval of the stored time series data according to the key field corresponding to the tag field to be retrieved to obtain the second time series data can be achieved through the following steps: determining a target key-value pair including a timestamp field from the stored time series data according to the key field corresponding to the tag field to be retrieved; aggregating the value fields corresponding to each timestamp field in the target key-value pair to obtain the second time series data.

[0091] The target key-value pair refers to a key-value pair determined from the stored time series data and matching the tag field to be retrieved.

[0092] Exemplarily, the target key-value pair may be determined from a memory buffer, an immutable memory buffer, or a data file according to a key field corresponding to a tag field to be retrieved.

[0093] Exemplarily, the key fields corresponding to the tag fields to be retrieved may be merged according to the time period fields to obtain multiple synthetic key fields, and the target key-value pairs may be determined from the stored time series data based on the synthetic key fields.

[0094] As shown in the following Table 1, target key-value pairs determined from the stored time series data are listed.

[0095] Table 1

[0096] key value <metricxs,tagkvs_1,basetime> [ts1,val1]...[tsn,valn] <metricxs,tagkvs_2,basetime> [ts1,val1']...[tsn,valn']... ... ...

[0097] Optionally, the value fields corresponding to the timestamp fields in the target key-value pair are aggregated to obtain the second time series data, such as Figure 4As shown, the data corresponding to each timestamp can be accumulated to obtain the aggregated data value of each timestamp node within a certain time range.<metricxs,tagkvs_1,basetime> ;key2=<metricxs,tagkvs_2,basetime> , ts1-tsn represents the time range formed by the timestamp field ts1 and the timestamp field tsn.

[0098] For example, Figure 5 As shown, a schematic diagram of a process of time series data retrieval is provided, which may specifically include the following steps:

[0099] Step S501, determining the key field corresponding to the tag field to be searched by searching the inverted index corresponding to the tag field to be searched;

[0100] Step S502, determining a target key-value pair including a timestamp field from the stored time series data according to the key field corresponding to the tag field to be retrieved;

[0101] Step S503: Aggregate the value fields corresponding to the timestamp fields in the target key-value pair to obtain second time series data.

[0102] like Figure 6 As shown, a system architecture diagram for time series data processing is provided. When the first time series data with a timestamp field is written to the disk, it can be written into a write-ahead log in advance to ensure that the data already written into the storage engine will not be lost. After the first time series data is written into the write-ahead log, the first time series data can be directly written into the memory buffer. When the size of the memory buffer reaches a certain threshold, the memory buffer can be converted into an immutable memory buffer. When the number of immutable memory buffers exceeds the threshold, the immutable memory buffer can be refreshed to the data file on the disk, and an inverted index is established for each label field in the first time series data, and the inverted index is written into the index file. When the immutable memory buffer is refreshed to the data file on the disk, the time series data of the same time period field can be merged in the background.

[0103] When you need to obtain time series data for a period of time by label field, you can first obtain all key fields containing the specified label field according to the label index, and filter the intersection of these key fields to obtain all key fields that meet the specified label field within a period of time. Then, based on the key fields that meet the specified label field, obtain the target key-value pairs in the corresponding period from the memory buffer, the immutable memory buffer, and the data file in turn, and aggregate the value fields corresponding to each timestamp field in the target key-value pairs to finally obtain the second time series data.

[0104] The implementation of the embodiments of the present disclosure solves many challenges in the storage and retrieval of time series data, such as large write traffic, large data scale, high storage cost, low input / output efficiency, low label dimension retrieval efficiency, etc., through data point appending writing, key granularity data point merging, label dimension inverted indexing, etc., and can support the storage of trillion-level data points, millisecond-level retrieval and aggregation response of tens of millions of indicators. In addition, the present disclosure adopts a key-value pair-based recording method to realize the storage of underlying time series data, which does not affect the retrieval and aggregation logic related to the upper-level time series data, and has strong general and maintenance capabilities.

[0105] Exemplary Devices

[0106] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Figure 7 A time series data processing device according to an exemplary embodiment of the present disclosure will be described.

[0107] See also Figure 7 , Figure 7 FIG. 2 is a block diagram of a time series data processing device according to an exemplary embodiment of the present disclosure. Figure 7 As shown, a time series data processing device 700 according to an exemplary embodiment of the present disclosure includes: a time series data acquisition module 710 and a time series data storage module 720, wherein:

[0108] The time series data acquisition module 710 is used to acquire first time series data to be stored, where the first time series data includes a key field, a value field, and a timestamp field; the key field includes a time period field;

[0109] The time series data storage module 720 is used to store the first time series data. The first time series data is combined with the stored time series data of the same time period field and stored as a key-value pair including a timestamp field.

[0110] In an optional embodiment, based on the aforementioned scheme, the time series data storage module 720 can be configured to: store the first time series data in a memory buffer; if the memory buffer is greater than a first threshold, convert the memory buffer into an immutable memory buffer; if the immutable memory buffer is greater than a second threshold, refresh the first time series data stored in the immutable memory buffer to a data file on the disk.

[0111] In an optional implementation, based on the aforementioned solution, before writing the first time series data into the memory buffer, the time series data processing device 700 further includes: a write-ahead log writing module, used to write the first time series data into a write-ahead log, and the write-ahead log is used to recover the time series data when the time series data is damaged or lost.

[0112] In an optional implementation, based on the aforementioned scheme, the key field also includes a label field, and the time series data processing device 700 also includes: an inverted index establishment module, which is used to establish an inverted index for the label field corresponding to the first time series data, and store the inverted index in an index file on the disk.

[0113] In an optional implementation, based on the above solution, the time series data processing device 700 further includes: a time series data retrieval module, configured to retrieve the stored time series data according to the field information to be retrieved, and obtain the second time series data.

[0114] In an optional embodiment, based on the aforementioned scheme, the field information to be retrieved includes the label field to be retrieved, and the time series data retrieval module may include: a key field search module, used to determine the key field corresponding to the label field to be retrieved by searching the inverted index corresponding to the label field to be retrieved; a time series data acquisition module, used to retrieve the stored time series data according to the key field corresponding to the label field to be retrieved, and obtain the second time series data.

[0115] In an optional embodiment, based on the aforementioned scheme, the time series data acquisition module can be configured to: determine a target key-value pair containing a timestamp field from the stored time series data according to the key field corresponding to the tag field to be retrieved; aggregate the value fields corresponding to each timestamp field in the target key-value pair to obtain second time series data.

[0116] By implementing the embodiments of the present disclosure, when storing time series data, the time series data of the same time period field are merged and stored, which can reduce the cost of data storage to a certain extent, and the data within a specified time period range can be queried without reading a large number of data records, which can improve the efficiency of data reading to a certain extent.

[0117] It should be noted that although several modules or units of the timing data processing device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0118] Exemplary Program Products

[0119] After introducing the apparatus according to the exemplary embodiment of the present disclosure, next, a program product according to the exemplary embodiment of the present disclosure will be described.

[0120] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the time series data processing method is implemented.

[0121] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0122] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0123] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0124] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure. Exemplarily, the following steps may be included:

[0125] Acquire first time series data to be stored, the first time series data including a key field, a value field, and a timestamp field; the key field includes a time period field;

[0126] The first time series data is stored, and the first time series data is combined with the stored time series data of the same time period field and stored as a key-value pair including a timestamp field.

[0127] In an optional implementation, based on the aforementioned scheme, the above-mentioned storage of the first time series data can be achieved through the following steps: storing the first time series data in a memory buffer; if the memory buffer is greater than a first threshold, converting the memory buffer into an immutable memory buffer; if the immutable memory buffer is greater than a second threshold, refreshing the first time series data stored in the immutable memory buffer to a data file on the disk.

[0128] In an optional implementation, based on the above solution, before writing the first time series data into the memory buffer, the following steps may also be performed: writing the first time series data into a write-ahead log, where the write-ahead log is used to recover the time series data when the time series data is damaged or lost.

[0129] In an optional implementation, based on the above solution, the key field also includes a tag field, and the following steps can also be performed: establish an inverted index for the tag field corresponding to the first time series data, and store the inverted index in an index file on the disk.

[0130] In an optional implementation, based on the above solution, the following steps may also be performed: according to the field information to be retrieved, the stored time series data is retrieved to obtain the second time series data.

[0131] In an optional implementation, based on the aforementioned scheme, the field information to be retrieved includes a label field to be retrieved, and the above-mentioned retrieval of the stored time series data according to the field information to be retrieved to obtain the second time series data can be achieved through the following steps: determining the key field corresponding to the label field to be retrieved by searching the inverted index corresponding to the label field to be retrieved; and retrieving the stored time series data according to the key field corresponding to the label field to be retrieved to obtain the second time series data.

[0132] In an optional implementation, based on the aforementioned scheme, the stored time series data is retrieved according to the key field corresponding to the tag field to be retrieved to obtain the second time series data, which can be achieved through the following steps: according to the key field corresponding to the tag field to be retrieved, determine the target key-value pair containing the timestamp field from the stored time series data; aggregate the value fields corresponding to each timestamp field in the target key-value pair to obtain the second time series data.

[0133] By implementing the embodiments of the present disclosure, when storing time series data, the time series data of the same time period field are merged and stored, which can reduce the cost of data storage to a certain extent, and the data within a specified time period range can be queried without reading a large number of data records, which can improve the efficiency of data reading to a certain extent.

[0134] Exemplary Electronic Devices

[0135] After introducing the time series data processing method, apparatus, and program product according to the exemplary embodiment of the present disclosure, next, an electronic device according to another exemplary embodiment of the present disclosure is introduced.

[0136] The exemplary embodiment of the present disclosure also provides an electronic device capable of implementing the above-mentioned time series data processing method. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as program code. The processor executes the method in the exemplary embodiment by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0137] Reference below Figure 8 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Figure 8 The electronic device 800 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0138] like Figure 8 As shown, the electronic device 800 may include: a processor 810 , a memory 820 , a bus 830 , an I / O (input / output) interface 840 , a network adapter 850 , and a display 860 .

[0139] The memory 820 may include a volatile memory, such as a RAM 821, a cache unit 822, and may also include a non-volatile memory, such as a ROM 823. The memory 820 may also include one or more program modules 824, such program modules 824 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 824 may include each module in the above-mentioned device.

[0140] The processor 810 may include one or more processing units. For example, the processor 810 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0141] The processor 810 may be configured to execute executable instructions stored in the memory 820 , such as executing any one or more method steps in the exemplary embodiment.

[0142] Exemplarily, the processor 810 may perform the following steps:

[0143] Acquire first time series data to be stored, the first time series data including a key field, a value field, and a timestamp field; the key field includes a time period field;

[0144] The first time series data is stored, and the first time series data is combined with the stored time series data of the same time period field and stored as a key-value pair including a timestamp field.

[0145] In an optional implementation, based on the aforementioned scheme, the above-mentioned storage of the first time series data can be achieved through the following steps: storing the first time series data in a memory buffer; if the memory buffer is greater than a first threshold, converting the memory buffer into an immutable memory buffer; if the immutable memory buffer is greater than a second threshold, refreshing the first time series data stored in the immutable memory buffer to a data file on the disk.

[0146] In an optional implementation, based on the above solution, before writing the first time series data into the memory buffer, the following steps may also be performed: writing the first time series data into a write-ahead log, where the write-ahead log is used to recover the time series data when the time series data is damaged or lost.

[0147] In an optional implementation, based on the above solution, the key field also includes a tag field, and the following steps can also be performed: establish an inverted index for the tag field corresponding to the first time series data, and store the inverted index in an index file on the disk.

[0148] In an optional implementation, based on the above solution, the following steps may also be performed: according to the field information to be retrieved, the stored time series data is retrieved to obtain the second time series data.

[0149] In an optional implementation, based on the aforementioned scheme, the field information to be retrieved includes a label field to be retrieved, and the above-mentioned retrieval of the stored time series data according to the field information to be retrieved to obtain the second time series data can be achieved through the following steps: determining the key field corresponding to the label field to be retrieved by searching the inverted index corresponding to the label field to be retrieved; and retrieving the stored time series data according to the key field corresponding to the label field to be retrieved to obtain the second time series data.

[0150] In an optional implementation, based on the aforementioned scheme, the stored time series data is retrieved according to the key field corresponding to the tag field to be retrieved to obtain the second time series data, which can be achieved through the following steps: according to the key field corresponding to the tag field to be retrieved, determine the target key-value pair containing the timestamp field from the stored time series data; aggregate the value fields corresponding to each timestamp field in the target key-value pair to obtain the second time series data.

[0151] By implementing the embodiments of the present disclosure, when storing time series data, the time series data of the same time period field are merged and stored, which can reduce the cost of data storage to a certain extent, and the data within a specified time period range can be queried without reading a large number of data records, which can improve the efficiency of data reading to a certain extent.

[0152] The bus 830 is used to realize the connection between different components of the electronic device 800, and may include a data bus, an address bus, and a control bus.

[0153] The electronic device 800 can communicate with one or more external devices 900 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 840 .

[0154] The electronic device 800 can communicate with one or more networks through the network adapter 850. For example, the network adapter 850 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 850 can communicate with other modules of the electronic device 800 through the bus 830.

[0155] The electronic device 800 may display a graphical user interface and the like through the display 860 .

[0156] although Figure 8 Not shown, other hardware and / or software modules may also be provided in the electronic device 800, including but not limited to: a display, a microcode, a device driver, a redundant processor, an external disk drive array, a RAID (Redundant Arrays of Independent Disks) system, a tape drive, and a data backup storage system.

[0157] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0158] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0159] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A time series data processing method, characterized in that: The method comprises: Acquire first time series data to be stored, the first time series data comprising a key field, a value field, and a timestamp field; the key field comprises a time period field; The first time series data is stored, and the first time series data is merged with the stored time series data of the same time period field and stored as a key-value pair including a timestamp field.

2. The method according to claim 1, characterized in that The storing the first time series data includes: storing the first time series data into a memory buffer; If the memory buffer is larger than a first threshold, converting the memory buffer into an immutable memory buffer; If the immutable memory buffer is larger than a second threshold, the first time series data stored in the immutable memory buffer is refreshed to a data file on a disk.

3. The method according to claim 2, characterized in that Before writing the first time series data into the memory buffer, the method further includes: The first time series data is written into a write-ahead log, where the write-ahead log is used to recover the time series data when the time series data is damaged or lost.

4. The method according to claim 1, characterized in that: The key field also includes a tag field, and the method further includes: An inverted index is created for the tag field corresponding to the first time series data, and the inverted index is stored in an index file on a disk.

5. The method according to claim 1, characterized in that The method further comprises: The stored time series data is retrieved according to the field information to be retrieved to obtain the second time series data.

6. The method according to claim 5, characterized in that The field information to be retrieved includes a tag field to be retrieved, and the stored time series data is retrieved according to the field information to be retrieved to obtain the second time series data, including: Determine the key field corresponding to the tag field to be retrieved by searching the inverted index corresponding to the tag field to be retrieved; The stored time series data is retrieved according to the key field corresponding to the tag field to be retrieved to obtain the second time series data.

7. The method according to claim 6, characterized in that The step of retrieving the stored time series data according to the key field corresponding to the tag field to be retrieved to obtain the second time series data includes: Determine a target key-value pair including a timestamp field from the stored time series data according to the key field corresponding to the tag field to be retrieved; The value fields corresponding to the timestamp fields in the target key-value pairs are aggregated to obtain second time series data.

8. A time series data processing device, characterized in that: The device comprises: A time series data acquisition module, used to acquire first time series data to be stored, wherein the first time series data includes a key field, a value field, and a timestamp field; the key field includes a time period field; The time series data storage module is used to store the first time series data, and the first time series data is combined with the stored time series data of the same time period field and stored as a key-value pair including a timestamp field.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images