Method and system for scanning data in time series database sorted by time

Abstract

The present invention discloses a method and system for scanning data in a time series database sorted by time, which belongs to the field of data scanning technology. The technical problem to be solved is how to optimize the data scanning method and improve the latest and oldest N data in the time series database sorted by time. The method includes the following steps: data storage: for streaming data, partition storage is performed according to timestamps; data query: when querying data, initialize the timestamp ts called, filter the qualified data from the storage partition according to the timestamp ts, and perform bulk sorting on the filtered data.

Description

Technical Field

[0001] The present invention relates to the technical field of data scanning, and in particular to a method and system for scanning data in a time series database sorted by time. Background Art

[0002] As the amount of data in streaming databases continues to increase, traditional data scanning methods can no longer meet performance requirements. The direction of optimization is to push the screening conditions to the input operators or even to the storage as much as possible.

[0003] How to optimize the data scanning method and improve the latest and oldest N pieces of data in the time series database sorted by time is a technical problem that needs to be solved. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a data scanning method and system for a time series database sorted by time, so as to solve the technical problem of how to optimize the data scanning method and achieve the improvement of the latest and oldest N data in the time series database sorted by time.

[0005] In a first aspect, the present invention provides a method for scanning data in a time series database sorted by time, comprising the following steps:

[0006] Data storage: For streaming data, partition storage is performed according to timestamps;

[0007] Data query: When querying data, initialize the calling timestamp ts, filter the qualified data from the storage partition according to the timestamp ts, and sort the filtered data in a large pile.

[0008] Preferably, the qualified data is filtered from the storage partition according to the timestamp ts. If the qualified data exists, the qualified data is sorted by the root of the heap, the limit data is saved, the timestamp ts is updated to the top value of the heap, and the qualified data is filtered from the storage partition according to the timestamp ts again, and the filtered data is sorted by the root of the heap. If the qualified data does not exist, the step is terminated.

[0009] As a preferred embodiment, when data is stored, three storage partitions are obtained, namely:

[0010] First partition: (1, 5), (5, 7), (4, 6), (3, 2);

[0011] Second partition: (6, 4), (10, 9), (8, 10), (9, 1);

[0012] The third partition: (15, 12), (11, 7), (13, 8), (12, 11);

[0013] Correspondingly, data query includes the following steps:

[0014] Initialize the capacity of the large heap root to 2;

[0015] Initial timestamp ts = INT64_MAX;

[0016] Query data from storage based on timestamps and store the first returned data (1, 5), (5, 7), (4, 6), (3, 2);

[0017] Filter the returned data. The filtering condition is that the second data in each group of data is less than 10, that is, a<10;

[0018] Sort the filtered data (1, 5) and (3, 2) in a large heap, discard the redundant data (5, 7) and (4, 6), and update the data stamp ts to the top value of the heap, ts = 3; if the returned data cannot fill the heap, ts is still INT64_MAX;

[0019] Query the data whose first data is less than the top value of the heap from the storage space. That is, the query condition is to query the data with k_timestamp < 3. The first data of each group of data in the second partition and the third partition is greater than 3. If no data that meets the condition is found, it will be returned directly.

[0020] The operator can output the data in the heap in reverse order.

[0021] In a second aspect, the present invention provides a system for scanning data in a time series database sorted by time, which is used to perform data scanning by scanning data in a time series database sorted by time as described in any one of the first aspects, wherein the system includes a data storage module and a data query module;

[0022] The data storage module is used to perform the following: for streaming data, partition storage is performed according to timestamps;

[0023] The data query module is used to perform the following: when executing data query, initialize the calling timestamp ts, filter the qualified data from the storage partition according to the timestamp ts, and perform bulk sorting on the filtered data.

[0024] Preferably, the data query module is used to perform the following: filter qualified data from the storage partition according to the timestamp ts; if qualified data exists, perform a large heap root sort on the qualified data, save the limit data, update the timestamp ts to the top value of the heap, filter qualified data from the storage partition according to the timestamp ts again, and perform a large heap sort on the filtered data; if qualified data does not exist, end the step.

[0025] Preferably, when the data storage module performs data storage, three storage partitions are obtained, namely:

[0026] First partition: (1, 5), (5, 7), (4, 6), (3, 2);

[0027] Second partition: (6, 4), (10, 9), (8, 10), (9, 1);

[0028] The third partition: (15, 12), (11, 7), (13, 8), (12, 11);

[0029] Correspondingly, the data query module is used to perform the following operations:

[0030] Initialize the capacity of the large heap root to 2;

[0031] Initial timestamp ts = INT64_MAX;

[0032] Query data from storage based on timestamps and store the first returned data (1, 5), (5, 7), (4, 6), (3, 2);

[0033] Filter the returned data. The filtering condition is that the second data in each group of data is less than 10, that is, a<10;

[0034] Sort the filtered data (1, 5) and (3, 2) in a large heap, discard the redundant data (5, 7) and (4, 6), and update the data stamp ts to the top value of the heap, ts = 3; if the returned data cannot fill the heap, ts is still INT64_MAX;

[0035] Query the data whose first data is less than the top value of the heap from the storage space. That is, the query condition is to query the data with k_timestamp < 3. The first data of each group of data in the second partition and the third partition is greater than 3. If no data that meets the condition is found, it will be returned directly.

[0036] The operator can output the data in the heap in reverse order.

[0037] In a third aspect, the present invention provides an electronic device, comprising: at least one memory and at least one processor;

[0038] The at least one memory is used to store a machine-readable program;

[0039] The at least one processor is used to call the machine-readable program to execute any method described in the first aspect.

[0040] In a fourth aspect, the present invention provides a computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the processor executes any of the methods described in the first aspect.

[0041] The method and system for scanning data in a time series database sorted by time of the present invention have the following advantages: relying on the orderly characteristics of storage partitions, data that does not meet the conditions can be quickly filtered out, reducing the number of original data items scanned at the upper level, thereby achieving the effect of accelerating query. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] The present invention is further described below in conjunction with the accompanying drawings.

[0044] Figure 1 This is a flowchart of a method for scanning data in a time series database sorted by time in Example 1;

[0045] Figure 2 This is a flowchart of data query in a method for scanning data in a time series database sorted by time in Example 1. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0047] The embodiments of the present invention provide a method and system for scanning data in a time series database sorted by time, which are used to solve the technical problem of how to optimize the data scanning method and improve the latest and oldest N pieces of data in the time series database sorted by time.

[0048] Embodiment 1:

[0049] The present invention discloses a method for scanning data in a time series database according to time sorting, which includes two steps: data storage and data query.

[0050] Step S100: Data storage: For streaming data, partition storage is performed according to timestamps.

[0051] Step S200 data query: When querying data, initialize the timestamp ts of the call, filter the qualified data from the storage partition according to the timestamp ts, and perform bulk sorting on the filtered data.

[0052] In step S200, qualified data is filtered from the storage partition according to the timestamp ts. If qualified data exists, a large heap root sort is performed on the qualified data, the limit data is saved, the timestamp ts is updated to the top value of the heap, and the qualified data is filtered from the storage partition according to the timestamp ts again, and the filtered data is sorted in a large heap. If no qualified data exists, the step is terminated.

[0053] As a specific example of this method, when data is stored, three storage partitions are obtained, namely:

[0054] First partition: (1, 5), (5, 7), (4, 6), (3, 2);

[0055] Second partition: (6, 4), (10, 9), (8, 10), (9, 1);

[0056] The third partition: (15, 12), (11, 7), (13, 8), (12, 11).

[0057] Correspondingly, data query includes the following steps:

[0058] (1) Initialize the capacity of the large heap root to 2;

[0059] (2) Initial timestamp ts = INT64_MAX;

[0060] (3) Query data from storage based on timestamps and store the first returned data (1, 5), (5, 7), (4, 6), (3, 2);

[0061] (4) Filter the returned data. The filtering condition is that the second data in each group of data is less than 10, that is, a<10;

[0062] (5) Sort the filtered data (1, 5) and (3, 2) in a large heap, discard the redundant data (5, 7) and (4, 6), and update the data stamp ts to the top value of the heap, ts = 3; if the returned data cannot fill the heap, ts is still INT64_MAX;

[0063] (6) Query the data whose first data is less than the top value of the heap from the storage space. That is, the query condition is to query the data with k_timestamp < 3. The first data of each group of data in the second partition and the third partition is greater than 3. If no data that meets the condition is found, it will be returned directly.

[0064] (7) The operator simply outputs the data in the heap in reverse order.

[0065] The method of this embodiment mainly stores and ensures that the partitions are ordered and the upper-level operator is implemented by heap sorting and timestamp push-down. It can rely on the ordered characteristics of the storage partitions to quickly filter out data that does not meet the conditions and reduce the number of original data items scanned by the upper layer, thereby achieving the effect of accelerating the query.

[0066] Embodiment 2:

[0067] The present invention discloses a data scanning system for a time series database sorted by time, comprising a data storage module and a data query module.

[0068] The data storage module is used to perform the following: for streaming data, partition storage is performed according to timestamps.

[0069] The data query module is used to perform the following: when executing data query, initialize the calling timestamp ts, filter the qualified data from the storage partition according to the timestamp ts, and perform bulk sorting on the filtered data.

[0070] As a specific implementation, the data query module is used to perform the following: filter qualified data from the storage partition according to the timestamp ts; if qualified data exists, perform a large heap root sort on the qualified data, save the limit data, update the timestamp ts to the top value of the heap, filter qualified data from the storage partition according to the timestamp ts again, and perform a large heap sort on the filtered data; if qualified data does not exist, end the step.

[0071] As an example of this embodiment, when the data storage module executes data storage, three storage partitions are obtained, namely:

[0072] First partition: (1, 5), (5, 7), (4, 6), (3, 2);

[0073] Second partition: (6, 4), (10, 9), (8, 10), (9, 1);

[0074] The third partition: (15, 12), (11, 7), (13, 8), (12, 11).

[0075] Correspondingly, the data query module is used to perform the following operations:

[0076] (1) Initialize the capacity of the large heap root to 2;

[0077] (2) Initial timestamp ts = INT64_MAX;

[0078] (3) Query data from storage based on timestamps and store the first returned data (1, 5), (5, 7), (4, 6), (3, 2);

[0079] (4) Filter the returned data. The filtering condition is that the second data in each group of data is less than 10, that is, a<10;

[0080] (5) Sort the filtered data (1, 5) and (3, 2) in a large heap, discard the redundant data (5, 7) and (4, 6), and update the data stamp ts to the top value of the heap, ts = 3; if the returned data cannot fill the heap, ts is still INT64_MAX;

[0081] (6) Query the data whose first data is less than the top value of the heap from the storage space. That is, the query condition is to query the data with k_timestamp < 3. The first data of each group of data in the second partition and the third partition is greater than 3. If no data that meets the condition is found, it will be returned directly.

[0082] (7) The operator simply outputs the data in the heap in reverse order.

[0083] The system of this embodiment can execute the method disclosed in Example 1 to achieve data scanning.

[0084] Embodiment 3:

[0085] The present invention discloses an electronic device, comprising: at least one memory and at least one processor; the at least one memory is used to store a machine-readable program; and the at least one processor is used to call the machine-readable program to execute the method disclosed in Embodiment 1 of the present invention.

[0086] Embodiment 4:

[0087] The embodiment of the present invention further provides a computer-readable medium, wherein computer instructions are stored on the computer-readable medium, and when the computer instructions are executed by a processor, the processor executes the method disclosed in Embodiment 1 of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which software program codes implementing the functions of any of the above embodiments are stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0088] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0089] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0090] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0091] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0092] It should be noted that not all steps and modules in the above-mentioned processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0093] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for scanning data in a time series database sorted by time, characterized in that: The steps include: Data storage: For streaming data, partition storage is performed according to timestamps; Data query: When querying data, initialize the calling timestamp ts, filter the qualified data from the storage partition according to the timestamp ts, and sort the filtered data in a large pile.

2. The method for scanning data in a time series database according to claim 1, characterized in that: Filter qualified data from the storage partition according to timestamp ts. If qualified data exists, perform a large heap root sort on the qualified data, save the limit data, update timestamp ts to the top value of the heap, and perform the filtering of qualified data from the storage partition according to timestamp ts again. Perform a large heap sort on the filtered data. If no qualified data exists, end the step.

3. The method for scanning data in a time series database according to claim 1 or 2, characterized in that: When storing data, three storage partitions are obtained, namely: First partition: (1, 5), (5, 7), (4, 6), (3, 2); Second partition: (6, 4), (10, 9), (8, 10), (9, 1); The third partition: (15, 12), (11, 7), (13, 8), (12, 11); Correspondingly, data query includes the following steps: Initialize the capacity of the large heap root to 2; Initial timestamp ts = INT64_MAX; Query data from storage based on timestamps and store the first returned data (1, 5), (5, 7), (4, 6), (3, 2); Filter the returned data. The filtering condition is that the second data in each group of data is less than 10, that is, a<10; Sort the filtered data (1, 5) and (3, 2) in a large heap, discard the redundant data (5, 7) and (4, 6), and update the data stamp ts to the top value of the heap, ts = 3; if the returned data cannot fill the heap, ts is still INT64_MAX; Query the data whose first data is less than the top value of the heap from the storage space. That is, the query condition is to query the data with k_timestamp < 3. The first data of each group of data in the second partition and the third partition is greater than 3. If no data that meets the condition is found, it will be returned directly. The operator can output the data in the heap in reverse order.

4. A system for scanning data in a time series database sorted by time, characterized in that: Used to perform data scanning by scanning data sorted by time in a time series database according to any one of claims 1 to 3, the system comprising a data storage module and a data query module; The data storage module is used to perform the following: for streaming data, partition storage is performed according to timestamps; The data query module is used to perform the following: when executing data query, initialize the calling timestamp ts, filter the qualified data from the storage partition according to the timestamp ts, and perform bulk sorting on the filtered data.

5. The system for scanning data in a time series database according to claim 4, characterized in that: The data query module is used to perform the following: filter qualified data from the storage partition according to the timestamp ts. If qualified data exists, perform a large heap root sort on the qualified data, save the limit data, update the timestamp ts to the top value of the heap, filter qualified data from the storage partition according to the timestamp ts again, and perform a large heap sort on the filtered data. If no qualified data exists, end the step.

6. The system for scanning data in a time series database according to time order according to claim 4 or 5, characterized in that: When the data storage module executes data storage, three storage partitions are obtained, namely: First partition: (1, 5), (5, 7), (4, 6), (3, 2); Second partition: (6, 4), (10, 9), (8, 10), (9, 1); The third partition: (15, 12), (11, 7), (13, 8), (12, 11); Correspondingly, the data query module is used to perform the following operations: Initialize the capacity of the large heap root to 2; Initial timestamp ts = INT64_MAX; Query data from storage based on timestamps and store the first returned data (1, 5), (5, 7), (4, 6), (3, 2); Filter the returned data. The filtering condition is that the second data in each group of data is less than 10, that is, a<10; Sort the filtered data (1, 5) and (3, 2) in a large heap, discard the redundant data (5, 7) and (4, 6), and update the data stamp ts to the top value of the heap, ts = 3; if the returned data cannot fill the heap, ts is still INT64_MAX; Query the data whose first data is less than the top value of the heap from the storage space. That is, the query condition is to query the data with k_timestamp < 3. The first data of each group of data in the second partition and the third partition is greater than 3. If no data that meets the condition is found, it will be returned directly. The operator can output the data in the heap in reverse order.

7. An electronic device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 3.

8. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, enable the processor to execute any one of the methods of claims 1 to 3.

Images