Deep paging and fast query technology for massive time series data

Abstract

This invention discloses a deep pagination-based fast query technology for massive time-series data, comprising the following steps: S1, inputting query conditions; S2, setting the query start time; S3, determining whether the current query start time is less than the initially input start time; if less, setting the current query start time as the initial query start time; S4, using the new start and end times to perform query counting statistics, accumulating the statistical results, and determining whether the time range meets the data extraction requirements according to data extraction rules; if it meets the data extraction requirements, performing a formal query and extracting the data into the returned dataset. This invention avoids direct database aggregation of data within the query time period by dividing data query into a statistical-before-query approach, accurately locating the query interval containing the data to be queried, reducing the amount of data computation, and improving the query speed of time-series data in time aggregation.

Description

Technical Field

[0001] The present invention relates to the field of time series data technology, and in particular to a deep paging fast query technology for massive time series data. Background Art

[0002] The full name of time series data is time series data, which is a series of data points indexed in chronological order. Time series data is almost everywhere, and because of its significant and valuable characteristics, it has become an object of special analysis in network security.

[0003] In the prior art, there are many aggregation queries in time series query scenarios. However, if the amount of data is very large, the machine performance loss is high during deep paging query after aggregation, and the query time is slow. Therefore, there is an urgent need for a deep paging and fast query technology for massive time series data. Summary of the invention

[0004] The present invention provides a deep paging and fast query technology for massive time series data to solve the problems raised in the above background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The deep paging fast query technology for massive time series data includes the following steps:

[0007] S1, input query conditions;

[0008] S2, set the query start time;

[0009] S3, determining whether the current query start time is less than the initial input start time, if so, setting the query start time to the initial query start time;

[0010] S4, use the new start time and end time to perform query count statistics first, accumulate the statistical results, and determine whether the time range meets the data extraction requirements according to the data extraction rules. If it meets the data extraction requirements, perform a formal query and extract data into the returned data set;

[0011] S5: Determine whether the current query start time is equal to the initial query start time. If the times are equal, directly return the assembled data in the data set;

[0012] S6, determine whether the current data query quantity is met, if not, set the current query start time to the end time of a new round of queries and repeat S2, if it is met, return the assembled data in the data set.

[0013] As a further improvement of the technical solution: the specific data extraction rules are: when data is acquired in batches according to time, the following six situations will occur:

[0014] The first one is that the current data query volume does not meet the data query requirements;

[0015] The second type is that the latter part of the current data query volume meets the data query requirements;

[0016] The third type is that the current data query volume fully meets the data query requirements;

[0017] Fourth, the first part of the current data query volume meets the data query requirements;

[0018] Fifth, the current data query volume exceeds the data query demand;

[0019] The sixth type is that part of the current data query volume meets the data query requirements;

[0020] Among them, except the first and fifth types, the other four types are data that meet the data query requirements. Then the amount of data for each query is accumulated as an anchor point to determine which situation the current query data belongs to and extract the data.

[0021] As a further improvement of the technical solution: in S1, the query conditions are the start time, the end time, and the query range.

[0022] As a further improvement of the technical solution: in S2, the query start time is set to use the end time minus the time interval set by the program as the start time of this query.

[0023] As a further improvement of the present technical solution: in S3, if the start time of this query is not less than the start time of the initial input, then directly proceed to the next step S4.

[0024] As a further improvement of the technical solution: in S4, if the time range does not meet the data extraction requirements, the corresponding interval data is obtained according to the data extraction rules and added to the data set, and then the next step S5 is performed.

[0025] As a further improvement of the present technical solution: in S5, if the current query start time is not equal to the initial query start time, proceed directly to the next step S6.

[0026] An embodiment of the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the deep paging and fast query technology for massive time series data described in any one of the above items is implemented.

[0027] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned deep paging and fast query technologies for massive time series data.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The invention divides data query into statistics first and then query, avoiding the database from directly aggregating the data within the query time period, accurately locating the query interval where the data to be queried is located, reducing the amount of data calculation, and improving the query speed of time series data in time aggregation.

[0030] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0032] Figure 1 A logical diagram of the deep paging fast query technology for massive time series data proposed by the present invention;

[0033] Figure 2 A diagram describing the data interval extraction rules proposed by the present invention;

[0034] Figure 3 A schematic structural diagram of a preferred embodiment of a terminal device provided by the present invention. DETAILED DESCRIPTION

[0035] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention. The present invention is described in more detail by way of example with reference to the accompanying drawings in the following paragraphs. The advantages and features of the present invention will become clearer according to the following description and claims. It should be noted that the drawings are all in a very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0036] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may also be a component centered. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may also be a component centered. When a component is considered to be "set on" another component, it may be directly set on the other component or there may also be a component centered. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0038] See also Figures 1 to 3 In an embodiment of the present invention, a deep paging fast query technology for massive time series data includes the following steps:

[0039] S1, enter the query conditions: start time, end time, query range, etc.;

[0040] S2, set the query start time, use the end time minus the time interval set by the program as the start time of this query;

[0041] S3, determine whether the current query start time is less than the initial input start time, if it is less, set the query start time to the initial query start time, if the current query start time is not less than the initial input start time, directly proceed to the next step S4;

[0042] S4, use the new start time and end time to perform query count statistics first, accumulate the statistical results, and determine whether the time range meets the data extraction requirements according to the data extraction rules. If it meets the data extraction requirements, perform a formal query and extract data into the returned data set; if it does not meet the data extraction requirements within the time range, obtain the corresponding interval data according to the data extraction rules and add it to the data set, and then proceed to the next step S5, where the data extraction rules are as follows: Figure 2 Specifically, when data is acquired in batches by time, the following six situations will occur:

[0043] The first one is that the current data query volume does not meet the data query requirements;

[0044] The second type is that the latter part of the current data query volume meets the data query requirements;

[0045] The third type is that the current data query volume fully meets the data query requirements;

[0046] Fourth, the first part of the current data query volume meets the data query requirements;

[0047] Fifth, the current data query volume exceeds the data query demand;

[0048] The sixth type is that part of the current data query volume meets the data query requirements;

[0049] Among them, except the first and fifth types, the other four types are data that meet the data query requirements. Then the amount of data queried each time is accumulated as an anchor point to determine which case the current query data belongs to and extract the data;

[0050] S5: Determine whether the current query start time is equal to the initial query start time. If the time is equal, directly return the assembled data in the data set. If the current query start time is not equal to the initial query start time, directly proceed to the next step S6;

[0051] S6, determine whether the current data query quantity is met, if not, set the current query start time to the end time of a new round of queries and repeat S2, if it is met, return the assembled data in the data set.

[0052] See also Figure 3 , Figure 3 It is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the deep paging fast query technology for massive time series data described in any of the above embodiments when executing the computer program.

[0053] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0054] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0055] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory can also be other volatile solid-state storage devices.

[0056] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 3 The structural diagram is only an example of the above-mentioned terminal device and does not constitute a limitation of the above-mentioned terminal device. It may include more or less components than shown in the figure, or combine certain components, or different components.

[0057] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the deep paging and fast query technology for massive time series data described in any of the above embodiments.

[0058] The working principle of the present invention is:

[0059] The present invention designs a method for improving the speed of deep paging query of aggregated data by performing batch statistics, query and assembly of data according to a set time interval.

[0060] The time query interval is defined in the program, and then the time interval is automatically set according to the depth of the current deep paging. When performing deep paging query with time aggregation, data query is performed according to the specified data query logic. When the queried data volume meets the requirements or reaches the specified time, the current queried data is assembled and returned according to the aggregation rules.

[0061] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any ordinary technician in the industry can smoothly implement the present invention as shown in the drawings and described above. However, any equivalent changes, modifications and evolutions made by technicians familiar with the profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A method for fast querying massive time series data by deep paging, characterized in that: The following steps are involved: S1, input query conditions; S2, set the query start time; S3, determining whether the current query start time is less than the initial input start time, if so, setting the query start time to the initial query start time; S4, use the new start time and end time to perform query count statistics first, accumulate the statistical results, and determine whether the time range meets the data extraction requirements according to the data extraction rules. If it meets the data extraction requirements, perform a formal query and extract data into the returned data set; S5: Determine whether the current query start time is equal to the initial query start time. If the times are equal, directly return the assembled data in the data set; S6, determine whether the current data query quantity is met, if not, set the current query start time to the end time of the new round of query and repeat S2, if it is met, return the assembled data in the data set; The data extraction rules are as follows: When data is acquired in batches according to time, the following six situations may occur: The first one is that the current data query volume does not meet the data query requirements; The second type is that the latter part of the current data query volume meets the data query requirements; The third type is that the current data query volume fully meets the data query requirements; Fourth, the first part of the current data query volume meets the data query requirements; Fifth, the current data query volume exceeds the data query demand; The sixth type is that part of the current data query volume meets the data query requirements; Among them, except the first and fifth types, the other four types are data that meet the data query requirements. Then the amount of data queried each time is accumulated as an anchor point to determine which case the current query data belongs to and extract the data; In S1, the query conditions are the start time, the end time, and the query range; In S2, setting the query start time specifically uses the end time minus the time interval set by the program as the start time of this query; In S3, if the start time of this query is not less than the start time initially input, then the next step S4 is directly entered; In S3, if the time range does not meet the data extraction requirements, the corresponding interval data is obtained according to the data extraction rules and added to the data set, and then the next step S5 is performed; In S4, if the current query start time is not equal to the initial query start time, directly proceed to the next step S6.

2. A terminal device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the deep paging fast query method for massive time series data as claimed in claim 1 is implemented.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the deep paging fast query method for massive time series data as claimed in claim 1.

Images