A method and apparatus for in-memory table matching and filtering in time-series databases
By initializing a memory table and building a hash table in the time-series database, and using a hash function to calculate the hash value of attribute value combinations for matching and filtering, the problem of excessively long response time when writing time-series data to the time-series database is solved, and the effect of fast batch writing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 山东浪潮数据库技术有限公司
- Filing Date
- 2023-11-24
- Publication Date
- 2026-07-31
AI Technical Summary
Time-series databases need to access metadata when writing time-series data, resulting in excessively long response times and failing to meet the requirements for fast batch writing.
By initializing an in-memory table and building a hash table, the FNV32 hash function is used to calculate the hash value of the attribute value combination, and matching and filtering are performed to quickly locate the target sub-table and write the data, reducing the metadata parsing time.
It improves the write performance of time-series databases, meets the need for fast batch writing of time-series data, and reduces response time.
Smart Images
Figure CN117591525B_ABST
Abstract
Description
Technical Field
[0001] This invention discloses a method and apparatus, relating to the field of time-series database technology, specifically a memory table matching and filtering method and apparatus for time-series databases. Background Technology
[0002] Time series data are mainly collected and generated by various types of real-time monitoring, inspection and analysis equipment in industries such as power, chemical, meteorology and geographic information. The typical characteristics of this industrial data are: high generation frequency, that is, multiple data points can be generated per second at each monitoring point; heavy dependence on the acquisition time, that is, each data point is required to correspond to a unique time; and a large amount of information from many monitoring points.
[0003] Currently, time series databases need to provide the ability to write time series data in batches. When writing time series data, it is necessary to access the metadata of time series objects. When dealing with the metadata of time series objects with a large amount of data, parsing the metadata of time series objects during writing will consume a lot of time, resulting in a long write request response time, which cannot meet the needs of time series databases to quickly write time series data in batches. Summary of the Invention
[0004] This invention addresses the problems of existing technologies by providing a memory table matching and filtering method and apparatus for time-series databases, reducing write request response time and meeting the needs of time-series databases for rapid batch writing of time-series data.
[0005] The specific solution proposed in this invention is as follows:
[0006] This invention provides a memory table matching and filtering method for time-series databases, comprising:
[0007] Initialize the memory table: Retrieve information on all child tables of at least one parent table in the time-series database, and store the information of all child tables of the parent table in a cached memory table.
[0008] Concurrent hash table construction: Retrieve the in-memory table of the parent table from the cache, obtain information on all child tables based on the in-memory table, construct a hash table based on the information of all child tables, and use the hash value of the attribute value combination of each child table as the key of the hash table, and the attribute value combination and child table ID as the value of the hash table.
[0009] Perform matching and filtering: When the time-series database writes time-series data to the parent table, it calculates a hash value based on the specified attribute value combination.
[0010] The hash value is used to search for the value in the constructed hash table. If the value is not found, it means that no attribute combination in the sub-table satisfies the attribute value combination specified at the time of writing, and the time-series data is not written.
[0011] If a value is found, check if the attribute value combination of the corresponding sub-table in the value matches the attribute value combination specified when writing. If not, no time series data is written. If there are two or more attribute value combinations in the value that match the attribute value combination specified when writing, no time series data is written. If there is only one attribute value combination in the value that matches the attribute value combination specified when writing, the matching is complete, and the time series data is written to the sub-table.
[0012] Furthermore, the initialization of the memory table in the memory table matching and filtering method for a time-series database includes: obtaining all child table information of the parent table, wherein the child table information includes child table ID and child table attribute value, and storing all child table information in a cached memory table according to a two-dimensional array data structure.
[0013] Furthermore, the concurrent hash table construction method in the memory table matching and filtering method for a time-series database includes: obtaining all sub-table information based on the memory table, obtaining attribute values in each sub-table information, combining the attribute values of each sub-table separately, calculating the hash value of the attribute value combination of each sub-table using the FNV32 hash function, constructing a hash table, with the hash value of the attribute value combination of each sub-table used as the key of the hash table, and the attribute value combination and sub-table ID used as the value of the hash table.
[0014] Furthermore, the concurrent hash table construction method in the memory table matching and filtering method for a time-series database includes: first, grouping all sub-tables according to twice the number of system cores of the database; then, simultaneously calculating the hash value of the attribute value combination of each sub-table within each group to construct a hash table.
[0015] The present invention also provides a memory table matching and filtering device for a time-series database, including an initialization module, a hash table construction module, and a matching and filtering module.
[0016] The initialization module initializes the memory table: it retrieves information about all child tables of at least one parent table in the time-series database and stores this information in a cached memory table.
[0017] The hash table construction module concurrently constructs hash tables: it retrieves the in-memory table of the parent table from the cache, obtains information on all child tables based on the in-memory table, and constructs a hash table based on the information of all child tables. The hash value of the attribute value combination of each child table is used as the key of the hash table, and the attribute value combination and the child table ID are used as the value of the hash table.
[0018] The matching and filtering module performs matching and filtering: when the time-series database writes time-series data to the parent table, it calculates the hash value based on the specified attribute value combination.
[0019] The hash value is used to search for the value in the constructed hash table. If the value is not found, it means that no attribute combination in the sub-table satisfies the attribute value combination specified at the time of writing, and the time-series data is not written.
[0020] If a value is found, check if the attribute value combination of the corresponding sub-table in the value matches the attribute value combination specified when writing. If not, no time series data is written. If there are two or more attribute value combinations in the value that match the attribute value combination specified when writing, no time series data is written. If there is only one attribute value combination in the value that matches the attribute value combination specified when writing, the matching is complete, and the time series data is written to the sub-table.
[0021] Furthermore, in the memory table matching and filtering device for a time-series database, the initialization module initializes the memory table, including: obtaining all sub-table information of the parent table, wherein the sub-table information includes sub-table ID and sub-table attribute values, and storing all sub-table information in a cached memory table according to a two-dimensional array data structure.
[0022] Furthermore, the hash table construction module in the memory table matching and filtering device for a time-series database concurrently constructs a hash table, including: obtaining all sub-table information based on the memory table, obtaining attribute values in each sub-table information, combining the attribute values of each sub-table separately, calculating the hash value of the attribute value combination of each sub-table using the FNV32 hash function, constructing a hash table, with the hash value of the attribute value combination of each sub-table used as the key of the hash table, and the attribute value combination and sub-table ID used as the value of the hash table.
[0023] Furthermore, the hash table construction module in the memory table matching and filtering device for a time-series database concurrently constructs a hash table, including: first grouping all sub-tables according to twice the number of system cores of the database, and simultaneously calculating the hash value of the attribute value combination of each sub-table within each group to construct a hash table.
[0024] The advantages of this invention are:
[0025] This invention provides a memory-based table matching and filtering method for time-series databases. It is used to match and filter target sub-tables when responding to write requests in a time-series database. By storing all sub-table information of the parent table in a cache, the time spent parsing time-series object metadata can be significantly reduced, thereby improving the write performance of time-series data. Furthermore, by concurrently constructing hash tables and using hash calculations to achieve fast matching and location of instance sub-tables, matching efficiency is improved. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.
[0029] A time series database is a database that primarily processes data with time labels (data that changes in chronological order, i.e., time-series data). Data with time labels is also called time series data.
[0030] This invention provides a memory table matching and filtering method for time-series databases, comprising:
[0031] Initialize the memory table: Retrieve information on all child tables of at least one parent table in the time-series database, and store the information of all child tables of the parent table in a cached memory table.
[0032] Concurrent hash table construction: Retrieve the in-memory table of the parent table from the cache, obtain information on all child tables based on the in-memory table, construct a hash table based on the information of all child tables, and use the hash value of the attribute value combination of each child table as the key of the hash table, and the attribute value combination and child table ID as the value of the hash table.
[0033] Perform matching and filtering: When the time-series database writes time-series data to the parent table, it calculates a hash value based on the specified attribute value combination.
[0034] The hash value is used to search for the value in the constructed hash table. If the value is not found, it means that no attribute combination in the sub-table satisfies the attribute value combination specified at the time of writing, and the time-series data is not written.
[0035] If a value is found, check if the attribute value combination of the corresponding sub-table in the value matches the attribute value combination specified when writing. If not, no time series data is written. If there are two or more attribute value combinations in the value that match the attribute value combination specified when writing, no time series data is written. If there is only one attribute value combination in the value that matches the attribute value combination specified when writing, the matching is complete, and the time series data is written to the sub-table.
[0036] The method of this invention can reduce the response time of write requests, meet the needs of time series databases for fast batch writing of time series data, and further provide the ability to write time series data in batches.
[0037] In specific applications, the process in some embodiments of the method of the present invention can be referred to as follows:
[0038] Step 1: Initialize the in-memory table: Obtain information on all child tables of at least one parent table in the time-series database, and store the information of all child tables of the parent table in a cached in-memory table. This avoids initializing the same parent table's in-memory table every time, improving the parsing speed when writing to the parent table.
[0039] The initialization of the memory table includes: obtaining all child table information of the parent table, wherein the child table information includes child table ID and child table attribute value, and storing all child table information in a cached memory table in a two-dimensional array data structure.
[0040] Step 2: Concurrently build a hash table: Retrieve the memory table of the parent table from the cache, obtain all child table information based on the memory table, and build a hash table based on all child table information. The hash value of the attribute value combination of each child table is used as the key of the hash table, and the attribute value combination and child table ID are used as the value of the hash table.
[0041] During the concurrent construction of the hash table, information on all sub-tables can be obtained from the cached memory table. Attribute values from each sub-table are retrieved from the memory table. Each sub-table can have one or more attributes. The attribute values of each sub-table are combined, and the FNV32 hash function is used to calculate the hash value of each sub-table attribute value combination. The hash value of each sub-table attribute value combination serves as the key of the hash table, and the attribute value combination and the sub-table ID serve as the value. The data structure storing the attribute value combination and the sub-table ID in the value can be two one-dimensional arrays.
[0042] When two or more sub-tables have the same attribute values, their attribute value combinations also have the same hash value. The sub-table IDs and attribute value combinations with the same attribute values are stored in the value array under the same key in the hash table. Another scenario is when two sub-tables have different attribute value combinations, but a hash collision occurs, resulting in two equal hash values. In this case, the sub-table IDs and attribute value combinations with the hash collision are stored in the value array under the same key in the hash table. The process is similar to the case of identical attribute values.
[0043] Furthermore, when a parent table has a large number of child tables, all child tables can be grouped according to twice the number of database system cores. At the same time, the hash value of the attribute value combination of each child table in each group is calculated, which speeds up the construction of hash tables and improves write performance.
[0044] Step 3: Perform matching and filtering: When the time-series database writes time-series data to the parent table, it calculates the hash value based on the specified attribute value combination. For example, assuming that the specified attributes when writing to the parent table are tag1 and tag2, it is necessary to retrieve the attribute values of tag1 and tag2 corresponding to all child tables from the memory table, combine the two attribute values of each child table separately, and use the FNV32 hash function to calculate the hash value of the attribute value combination of each child table.
[0045] The hash value is used to search for the value in the constructed hash table. If the value is not found, it means that no attribute combination in the sub-table satisfies the attribute value combination specified at the time of writing, and the time-series data is not written.
[0046] If a value is found, check if the attribute value combination of the corresponding sub-table in the value matches the attribute value combination specified when writing. If not, no time series data is written. If there are two or more attribute value combinations in the value that match the attribute value combination specified when writing, no time series data is written. If there is only one attribute value combination in the value that matches the attribute value combination specified when writing, the matching is complete, and the time series data is written to the sub-table.
[0047] The present invention also provides a memory table matching and filtering device for a time-series database, including an initialization module, a hash table construction module, and a matching and filtering module.
[0048] The initialization module initializes the memory table: it retrieves information about all child tables of at least one parent table in the time-series database and stores this information in a cached memory table.
[0049] The hash table construction module concurrently constructs hash tables: it retrieves the in-memory table of the parent table from the cache, obtains information on all child tables based on the in-memory table, and constructs a hash table based on the information of all child tables. The hash value of the attribute value combination of each child table is used as the key of the hash table, and the attribute value combination and the child table ID are used as the value of the hash table.
[0050] The matching and filtering module performs matching and filtering: when the time-series database writes time-series data to the parent table, it calculates the hash value based on the specified attribute value combination.
[0051] The hash value is used to search for the value in the constructed hash table. If the value is not found, it means that no attribute combination in the sub-table satisfies the attribute value combination specified at the time of writing, and the time-series data is not written.
[0052] If a value is found, check if the attribute value combination of the corresponding sub-table in the value matches the attribute value combination specified when writing. If not, no time series data is written. If there are two or more attribute value combinations in the value that match the attribute value combination specified when writing, no time series data is written. If there is only one attribute value combination in the value that matches the attribute value combination specified when writing, the matching is complete, and the time series data is written to the sub-table.
[0053] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.
[0054] Similarly, the device of this invention is used to match and filter target sub-tables when responding to write requests in a time-series database. By storing all sub-table information of the parent table in a cache, the time for parsing time-series object metadata can be greatly reduced, thereby improving the write performance of time-series data. By concurrently building a hash table and using hash calculations to achieve fast matching and locating of instance sub-tables, matching efficiency is improved.
[0055] It should be noted that not all steps and modules in the above processes and device structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structure described in the above embodiments can be a physical structure or a logical structure. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.
[0056] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.
Claims
1. A method for memory table matching filtering of a time series database, characterized in that include: Initialize the memory table: Retrieve information on all child tables of at least one parent table in the time-series database, and store the information of all child tables of the parent table in a cached memory table. Concurrent hash table construction: Retrieve the memory table of the parent table from the cache, obtain all child table information based on the memory table, construct a hash table based on all child table information, use the hash value of the attribute value combination of each child table as the key of the hash table, and use the attribute value combination and child table ID as the value of the hash table. Specifically, obtain all child table information from the memory table, obtain the attribute values in the information of each child table, combine the attribute values of each child table separately, use the FNV32 hash function to calculate the hash value of the attribute value combination of each child table, construct the hash table, use the hash value of the attribute value combination of each child table as the key of the hash table, and use the attribute value combination and child table ID as the value of the hash table. Perform matching and filtering: When the time-series database writes time-series data to the parent table, it calculates a hash value based on the specified attribute value combination. The hash value is used to search for the value in the constructed hash table. If the value is not found, it means that no attribute combination in the sub-table satisfies the attribute value combination specified at the time of writing, and the time-series data is not written. If a value is found, check if the attribute value combination of the corresponding sub-table in the value matches the attribute value combination specified when writing. If not, no time series data is written. If there are two or more attribute value combinations in the value that match the attribute value combination specified when writing, no time series data is written. If there is only one attribute value combination in the value that matches the attribute value combination specified when writing, the matching is complete, and the time series data is written to the sub-table.
2. The memory table matching and filtering method for a time-series database according to claim 1, characterized in that: The initialization of the memory table includes: obtaining all child table information of the parent table, wherein the child table information includes child table ID and child table attribute value, and storing all child table information in a cached memory table in a two-dimensional array data structure.
3. The memory table matching and filtering method for a time-series database according to claim 1, characterized in that: The concurrent hash table construction includes: first, grouping all sub-tables into groups based on twice the number of database system cores; then, simultaneously calculating the hash value of the attribute value combination of each sub-table within each group to construct the hash table.
4. A memory table matching and filtering device for a time-series database, characterized in that: It includes an initialization module, a hash table construction module, and a matching and filtering module. The initialization module initializes the memory table: it retrieves information about all child tables of at least one parent table in the time-series database and stores this information in a cached memory table. The hash table construction module concurrently constructs hash tables: it retrieves the memory table of the parent table from the cache, obtains all child table information based on the memory table, constructs a hash table based on all child table information, uses the hash value of the attribute value combination of each child table as the key of the hash table, and uses the attribute value combination and child table ID as the value of the hash table. Specifically, it obtains all child table information from the memory table, obtains the attribute values in the information of each child table, combines the attribute values of each child table separately, uses the FNV32 hash function to calculate the hash value of the attribute value combination of each child table, constructs the hash table, and uses the hash value of the attribute value combination of each child table as the key of the hash table, and uses the attribute value combination and child table ID as the value of the hash table. The matching and filtering module performs matching and filtering: when the time-series database writes time-series data to the parent table, it calculates the hash value based on the specified attribute value combination. The hash value is used to search for the value in the constructed hash table. If the value is not found, it means that no attribute combination in the sub-table satisfies the attribute value combination specified at the time of writing, and the time-series data is not written. If a value is found, check if the attribute value combination of the corresponding sub-table in the value matches the attribute value combination specified when writing. If not, no time series data is written. If there are two or more attribute value combinations in the value that match the attribute value combination specified when writing, no time series data is written. If there is only one attribute value combination in the value that matches the attribute value combination specified when writing, the matching is complete, and the time series data is written to the sub-table.
5. The memory table matching and filtering device for a time-series database according to claim 4, characterized in that: The initialization module initializes the memory table, including: obtaining all child table information of the parent table, wherein the child table information includes child table ID and child table attribute value, and storing all child table information in the cache memory table in a two-dimensional array data structure.
6. The memory table matching and filtering device for a time-series database according to claim 4, characterized in that: The hash table construction module constructs hash tables concurrently, including: first, grouping all sub-tables into groups based on twice the number of database system cores; then, simultaneously calculating the hash value of the attribute value combination of each sub-table within each group, and constructing the hash table.