A distributed database and a data retrieval method thereof

Abstract

The application discloses a kind of distributed database and its data retrieval method, it is related to data retrieval technical field, the distributed database includes processing module, aggregation module and storage module.The application can evaluate the time period, time precision and time heat of time series data section, so that the aggregation file with same time period, time precision and time heat is evenly dispersed on different data storage nodes, realizes the high-density concurrent access of time series data section, solves the problem of load imbalance, effectively improves the response speed and data analysis efficiency when small data amount random retrieval is carried out under the background of mass time series data.

Description

Technical Field

[0001] This invention relates to the field of data retrieval technology, specifically to a distributed database and its data retrieval method. Background Technology

[0002] With the rapid development of information technology, big data is becoming a crucial asset across various industries. Furthermore, the rapid advancements in technologies such as the Internet of Things (IoT) and 5G have led to an expansion of the scope and diversification of big data sources. As global industrial levels rise, industrial big data has become an important member of the big data family, with time-series data being a significant and typical type of industrial big data. In various smart industrial scenarios, the volume of heterogeneous time-series log data from different sources is exploding. Given the ever-increasing scale of time-series data, the storage and management of massive amounts of industrial time-series data has become a major research focus in the field of big data.

[0003] However, existing time-series data storage suffers from load imbalance and low data query efficiency. In some typical business scenarios, such as in ultra-large automated terminal systems, tens of thousands of terminal equipment sensors generate a large amount of equipment operating status time-series data in a very short period of time. For the same sensor or data in similar time periods, writing and reading usually occur on the same storage node, which can easily cause load skew problems caused by uneven data access traffic. This can easily lead to slow data access speeds or even server crashes, which can seriously affect the operation of business processes.

[0004] Therefore, this application provides a distributed database and its data retrieval method to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a distributed database and its data retrieval method, device, and medium.

[0006] Firstly, a distributed database comprising multiple data storage nodes, including:

[0007] The processing module is used to obtain the time period, time precision, and time popularity of the time series data segment, wherein the time popularity is determined based on the proportion of the time period in the preset time period and the preset time popularity of the preset time period.

[0008] The aggregation module is used to aggregate multiple time-series data segments with the same time period, time precision, and time popularity according to the time period, time precision, and time popularity to obtain multiple aggregate files; and to divide the multiple aggregate files into multiple sub-aggregate files according to the number of time data storage nodes.

[0009] The storage module is used to store multiple sub-aggregated files on multiple data storage nodes, using time period, time precision, time popularity, and average fraction as index values.

[0010] Preferably, the processing module includes:

[0011] A time accuracy determination module is used to obtain the sampling time interval and calculate the time accuracy based on the sampling time interval.

[0012] The time precision determination module is used to obtain the time period of the time series data segment; determine a target preset time period from multiple preset time periods based on the time period, and calculate the proportion of the time series data segment in the target preset time period; calculate the time popularity of the time series data segment based on the proportion of the time series data segment in the target preset time period and the preset time popularity of the target preset time period, wherein the preset time popularity is determined based on the historical query frequency of the preset time period within the preset time period.

[0013] Preferably, the duration of the time period is less than the duration of the preset time sequence data segment, and the duration of the preset time sequence data segment is less than the duration of the preset time period.

[0014] Preferably, the storage module includes:

[0015] The acquisition module is used to obtain the byte length of the sub-aggregation file;

[0016] The segmentation module is used to determine whether the byte length is greater than a preset threshold. When the byte length is greater than the preset threshold, the sub-aggregated file is segmented to obtain multiple segmented files and their corresponding segment numbers.

[0017] The sub-storage module stores multiple segmented files in the data storage node using time precision, time heat, average division number, and segment number as index values.

[0018] Secondly, a data retrieval method based on a distributed database, the method comprising:

[0019] Obtain the query instruction for the target time series data segment input by the user, and obtain the query time period, query time precision, and query time popularity based on the query instruction;

[0020] Based on the query time period, query time precision, and query time popularity, multiple target sub-aggregated files are determined from multiple data storage nodes;

[0021] The multiple target sub-aggregated files are processed and returned to the user as query results.

[0022] Preferably, when determining multiple target sub-aggregate files from multiple data storage nodes based on the query time period, query time precision, and query time popularity, if none of the multiple data storage nodes have a target sub-aggregate file with the same query time period, query time precision, or query time popularity, then multiple target sub-aggregate files are determined from the multiple data storage nodes based on the range of the query time period, query time precision, and query time popularity.

[0023] Thirdly, an electronic device includes: a memory for storing a computer program; and a processor for executing the computer program to implement the method as described in any of the preceding claims.

[0024] Fourthly, a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement any of the methods described above.

[0025] The beneficial effects of the present invention are reflected in the following: the embodiments of the present invention can evaluate the time period, time precision and time popularity of time-series data segments, thereby uniformly distributing aggregated files with the same time period, time precision and time popularity on different data storage nodes, realizing high-density concurrent access to time-series data segments, solving the problem of unbalanced load, and effectively improving the response speed and data analysis efficiency when performing random retrieval of small amounts of data in the context of massive time-series data. Attached Figure Description

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0027] Figure 1 This is a schematic diagram of the structure of a distributed database provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram illustrating the workflow of a distributed database provided in an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the index values ​​of the segmented file provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the timing data segment flow provided in an embodiment of the present invention;

[0031] Figure 5This is a flowchart illustrating a data retrieval method based on a distributed database provided in an embodiment of the present invention.

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0034] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0035] Example 1

[0036] By analyzing the system time-series data query logs in various specific industrial scenarios, we found that industrial systems have relatively fixed time-series data access patterns under different business scenarios, which are characterized by frequent queries for a specific time period. Therefore, we evaluate the popularity of the written time-series data segments based on the query requests entered by users, and distribute the evaluated data evenly across different data storage nodes to achieve load balancing.

[0037] refer to Figure 1-4 Based on the above findings, this application provides a distributed database. The distributed database 100 includes multiple data storage nodes, and also includes a processing module 101, an aggregation module 102, and a storage module 103.

[0038] In this embodiment of the application, the processing module 101 is used to obtain the time period, time precision and time popularity of the time series data segment, wherein the time popularity is determined according to the proportion of the time period in the preset time period and the preset time popularity of the preset time period.

[0039] To facilitate the retrieval of time-series data, the data can be preprocessed before passing through the processing module. Longer time-series data can be divided into smaller time-series data segments, and shorter time-series data can be divided into larger time-series data segments. Specifically, a preset time period is used as a parameter to ensure that the duration of each time-series data segment is less than the duration of the preset time period.

[0040] Specifically, the processing module includes: a time precision determination module, used to obtain the sampling time interval; calculate the time precision based on the sampling time interval; a time precision determination module, used to obtain the time period of the time series data segment; determine a target preset time period from multiple preset time periods based on the time period, and calculate the proportion of the time series data segment in the target preset time period; calculate the time popularity of the time series data segment based on the proportion of the time series data segment in the target preset time period and the preset time popularity of the target preset time period, wherein the preset time popularity is determined based on the historical query frequency of the preset time period within the preset time period.

[0041] The preset time period is longer than the preset time interval, and the preset time interval is longer than the time interval. It should be understood that the preset time period, preset time interval, and time interval are all set by the user according to actual circumstances. For example, the preset time period may be a week, and the preset time interval may be weekdays and rest days, with weekday time-series data segments having higher time-volume activity and rest day time-volume activity; or, the preset time period may be a day, and the preset time interval may be the working hours and rest hours within a day, with working hours time-series data segments having higher time-volume activity and rest hours time-volume activity. Furthermore, the preset time period and preset time interval need to be updated periodically to adapt to changes in user query frequency, thereby achieving better performance.

[0042] In this embodiment of the application, the aggregation module 102 is used to aggregate multiple time-series data segments with the same time period, time precision, and time popularity according to the time period, time precision, and time popularity to obtain multiple aggregate files; and to divide the multiple aggregate files into multiple sub-aggregate files according to the number of time data storage nodes.

[0043] Time series data segments with the same time period, time precision, and time popularity are aggregated to further facilitate retrieval.

[0044] In this embodiment, the storage module 103 is used to store multiple sub-aggregated files on multiple data storage nodes using time period, time precision, time popularity and average number as index values.

[0045] In this embodiment, the storage module includes: an acquisition module for acquiring the byte length of the sub-aggregated file; a segmentation module for determining whether the byte length is greater than a preset threshold, and if the byte length is greater than the preset threshold, segmenting the sub-aggregated file to obtain multiple segmented files and their corresponding segment numbers; and a sub-storage module for storing the multiple segmented files in a data storage node using time precision, time popularity, average number, and segment number as index values. A schematic diagram of the index values ​​of the segmented files in this embodiment is shown below. Figure 3 As shown.

[0046] The aggregated file is segmented to ensure that the size of the segmented file meets the preset file size, which further facilitates retrieval. At the same time, compared with the traditional storage method, it can effectively reduce information redundancy and compress storage space.

[0047] Specifically, the distributed database provided in this application embodiment is HBase. It should be noted that HBase is a NoSQL database built on top of HDFS. It organizes data in the form of Key (row key) and Value, arranged in lexicographical order of the Key values. The resulting HFile files are stored in HDFS. The main query methods are of two types: Get using a single or multiple Key values ​​to obtain the corresponding record, or Scan using a range of Key values ​​to obtain multiple records. When the background data volume is large, random read / write performance is superior for small data volumes.

[0048] In summary, the embodiments of the present invention can evaluate the time period, time precision, and time popularity of time-series data segments, thereby uniformly distributing aggregated files with the same time period, time precision, and time popularity across different data storage nodes. This enables high-density concurrent access to time-series data segments, solves the problem of unbalanced load, and effectively improves the response speed and data analysis efficiency when performing random retrieval of small amounts of data in the context of massive time-series data.

[0049] Example 2

[0050] refer to Figure 5 Based on the above embodiments, this application provides a data retrieval method based on a distributed database. The method includes: obtaining a query instruction for a target time-series data segment input by a user; obtaining a query time period, query time precision, and query time popularity based on the query instruction; determining multiple target sub-aggregate files from multiple data storage nodes based on the query time period, query time precision, and query time popularity; and processing the multiple target sub-aggregate files and returning them to the user as query results.

[0051] In this embodiment, multiple target sub-aggregate files are determined from multiple data storage nodes based on the query time period, query time precision, and query time popularity. If no target sub-aggregate file with the same query time period, query time precision, or query time popularity exists on any of the multiple data storage nodes, multiple target sub-aggregate files are determined from the multiple data storage nodes based on the range of the query time period, query time precision, and query time popularity, thereby improving applicability.

[0052] The method provided in this application uses the query time period, query time precision, and query time popularity as row keys to start scanning, thereby quickly locating the range of required data and continuously reading the data within that range. The method provided in this application enables parallel scanning of multiple data nodes, avoiding scanning all time-series data and achieving rapid data retrieval in the context of massive time-series background data.

[0053] Example 3

[0054] refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device 300 provided in an embodiment of this application. The electronic device 300 includes: at least one processor 301, at least one communication interface 302, at least one memory 303, and at least one bus 304. The bus 304 is used to enable direct communication between these components. The communication interface 302 is used for signaling or data communication with other node devices. The memory 303 stores machine-readable instructions executable by the processor 301. When the electronic device 300 is running, the processor 301 communicates with the memory 303 via the bus 304. When the machine-readable instructions are invoked by the processor 301, the data retrieval method provided in the above embodiment is executed.

[0055] Processor 301 can be an integrated circuit chip with signal processing capabilities. The processor 301 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0056] The memory 303 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0057] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof.

[0058] Furthermore, embodiments of the present invention also provide a computer storage medium storing a computer program, which, when run by a computer, performs the steps of the data retrieval method provided in the above embodiments.

[0059] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0060] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0061] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0062] It should be noted that if the function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A distributed database, comprising multiple data storage nodes, characterized in that, Also includes: The processing module is used to obtain the time period, time precision, and time popularity of the time series data segment, wherein the time popularity is determined based on the proportion of the time period in the preset time period and the preset time popularity of the preset time period. The aggregation module is used to aggregate multiple time-series data segments with the same time period, time precision, and time popularity according to the time period, time precision, and time popularity to obtain multiple aggregate files; and to divide the multiple aggregate files into multiple sub-aggregate files according to the number of time data storage nodes. The storage module is used to store multiple sub-aggregated files on multiple data storage nodes, using time period, time precision, time popularity and average fraction as index values; The storage module includes: The acquisition module is used to obtain the byte length of the sub-aggregation file; The segmentation module is used to determine whether the byte length is greater than a preset threshold. When the byte length is greater than the preset threshold, the sub-aggregated file is segmented to obtain multiple segmented files and their corresponding segment numbers. The sub-storage module uses time precision, time heat, average division number, and segment number as index values ​​to store multiple segmented files in the data storage node; The system starts scanning with the query time period, query time precision, and query time popularity as row keys to quickly locate the range of the required data and continuously read out the data within that range. The processing module includes: A time accuracy determination module is used to obtain the sampling time interval and calculate the time accuracy based on the sampling time interval. The time popularity determination module is used to obtain the time period of the time series data segment; determine a target preset time period from multiple preset time periods based on the time period, and calculate the proportion of the time series data segment in the target preset time period; calculate the time popularity of the time series data segment based on the proportion of the time series data segment in the target preset time period and the preset time popularity of the target preset time period, wherein the preset time popularity is determined based on the historical query frequency of the preset time period within the preset time period.

2. A distributed database according to claim 1, characterized in that, The duration of the time period is less than the duration of the preset time sequence data segment, and the duration of the preset time sequence data segment is less than the duration of the preset time period.

3. A data retrieval method based on a distributed database, applicable to a distributed database as described in claims 1-2, characterized in that, The method includes: Obtain the query instruction for the target time series data segment input by the user, and obtain the query time period, query time precision, and query time popularity based on the query instruction; Based on the query time period, query time precision, and query time popularity, multiple target sub-aggregated files are determined from multiple data storage nodes; The multiple target sub-aggregated files are processed and returned to the user as query results.

4. The data retrieval method based on a distributed database according to claim 3, characterized in that, When determining multiple target sub-aggregate files from multiple data storage nodes based on the query time period, query time precision, and query time popularity, if none of the multiple data storage nodes have a target sub-aggregate file with the same query time period, query time precision, or query time popularity, then multiple target sub-aggregate files are determined from the multiple data storage nodes based on the range of the query time period, query time precision, and query time popularity.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of a data retrieval method based on a distributed database as described in claims 3-4 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the data retrieval method based on a distributed database as described in claims 3-4.

Images