Data query method and device of wal log, time series database and medium

Abstract

This invention relates to the field of electronic digital data processing technology, specifically to a data query method, apparatus, time-series database, and medium for WAL logs. The method includes: receiving a log information query instruction to obtain a query target; responding to the log information query instruction, searching for the metadata of the corresponding data table in the global metadata of the time-series database, and determining the write location information of the query target based on the metadata; and locating the query target based on the write location information and a pre-constructed WAL log encoding structure, wherein the WAL log encoding structure includes multiple data blocks, multiple data pages, and data segments. This solves the technical problem in related technologies where unnecessary data page locking occurs during data querying, resulting in low cache utilization, a large amount of useless cache, and consequently affecting database query performance.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a data query method, apparatus, time-series database, and medium for WAL logs. Background Technology

[0002] Time-series data is written to the database at high speed, in large quantities, and continuously.

[0003] In related technologies, during data reading, if the data has not yet been flushed to the local disk, the data page remains locked and cannot be swapped out, even if the data to be read does not contain it. This leads to decreased cache utilization, the generation of a large amount of useless cache, and the inability to cache other data that should be cached, impacting database query performance and urgently requiring improvement. Summary of the Invention

[0004] This invention provides a data query method, apparatus, time-series database, and medium for WAL logs, to solve the technical problems in related technologies, such as unnecessary data page locking during data query, low cache utilization, easy generation of a large amount of useless cache, and thus affecting database query performance.

[0005] A first aspect of the present invention provides a data query method for WAL logs, applied to a time-series database, wherein the method includes the following steps: receiving a log information query instruction to obtain a query target; responding to the log information query instruction, searching for the metadata of the corresponding data table in the global metadata of the time-series database, and determining the write location information of the query target based on the metadata; locating the query target based on the write location information and a pre-constructed WAL log encoding structure, wherein the WAL log encoding structure includes multiple data blocks, multiple data pages, and data segments.

[0006] Optionally, in one embodiment of the present invention, before receiving the log information query instruction, the method further includes: receiving a time-series data import instruction and obtaining the WAL log generated during the time-series data import; mapping the log information of the WAL log to a corresponding data table, and writing the data table and the log information into memory composed of the WAL log encoding structure.

[0007] Optionally, in one embodiment of the present invention, each data block stores WAL log information of a data table, and each data block stores multiple log messages, each log message including a data header and data, wherein the data header is the timestamp and sequence number of each log message; each data page includes the multiple data blocks, and the header of each data page records pre-allocated storage information, sequence number, number of data blocks, identifier of the data table corresponding to each data block, timestamp range, and offset in the data page; each data segment includes the multiple data pages.

[0008] Optionally, in one embodiment of the present invention, writing the data table and the log information into the memory composed of the pre-constructed WAL log encoding structure includes: filtering memory that meets preset storage conditions, and initializing each data segment in the memory to obtain pre-allocated storage information for each data segment; based on the pre-allocated storage information, writing a corresponding number of log information into each data segment in a preset order until all log information has been written.

[0009] Optionally, in one embodiment of the present invention, initializing each data segment in the memory to obtain pre-allocated storage information for each data segment includes: storing multiple log messages from the corresponding number of log messages into the current data block of the current data page of the current data segment until the current data block reaches the pre-allocated number of storage messages; storing the remaining log messages into the next data block until all data blocks in the current data page reach the pre-allocated number of storage messages; storing the remaining log messages into the next data page until all data pages in the current data segment reach the pre-allocated storage limit; determining that the corresponding number of log messages have been written into the current data segment; and updating the storage location of each log message in the metadata of the corresponding data table.

[0010] Optionally, in one embodiment of the present invention, the expression for calculating the actual storage capacity of the current data page is:

[0011] ,

[0012] in, This indicates the actual storage capacity. This indicates the number of pre-allocated storage entries. Indicates the corresponding first n The size of each data table Indicates the corresponding first n A data table.

[0013] Optionally, in one embodiment of the present invention, locating the query target based on the write location information and the pre-built WAL log encoding structure includes: locating at least one data segment to be scanned in the WAL log encoding structure based on the write location information, and traversing the at least one data segment to be scanned using the identifier or timestamp of the at least one data table to find the corresponding data page; traversing the corresponding data page using the timestamp to locate the corresponding data block, and storing the corresponding location information in a candidate list; searching the candidate list for log information that meets the preset query conditions until all data segments to be scanned have been scanned.

[0014] A second aspect of the present invention provides a data query device for WAL logs, applied to a time-series database, wherein the device includes: a receiving module for receiving a log information query instruction to obtain a query target; a searching module for responding to the log information query instruction to search for the metadata of the corresponding data table in the global metadata of the time-series database, and determining the write location information of the query target based on the metadata; and a locating module for locating the query target based on the write location information and a pre-constructed WAL log encoding structure, wherein the WAL log encoding structure includes multiple data blocks, multiple data pages, and data segments.

[0015] Optionally, in one embodiment of the present invention, it further includes: an acquisition module, configured to receive a time-series data import instruction and acquire the WAL log generated during the time-series data import; and a writing module, configured to map the log information of the WAL log to a corresponding data table and write the data table and the log information into memory composed of the WAL log encoding structure.

[0016] Optionally, in one embodiment of the present invention, each data block stores WAL log information of a data table, and each data block stores multiple log messages, each log message including a data header and data, wherein the data header is the timestamp and sequence number of each log message; each data page includes the multiple data blocks, and the header of each data page records pre-allocated storage information, sequence number, number of data blocks, identifier of the data table corresponding to each data block, timestamp range, and offset in the data page; each data segment includes the multiple data pages.

[0017] Optionally, in one embodiment of the present invention, the writing module includes: an initialization unit, configured to filter memory that meets preset storage conditions and initialize each data segment in the memory to obtain pre-allocated storage information for each data segment; and a writing unit, configured to write a corresponding number of log information into each data segment in a preset order based on the pre-allocated storage information until all log information has been written.

[0018] Optionally, in one embodiment of the present invention, the initialization unit includes: a first storage subunit, configured to store multiple log messages from the corresponding number of log messages into the current data block of the current data page of the current data segment, until the current data block reaches the pre-allocated number of storage messages; a second storage subunit, configured to store the remaining log messages into the next data block, until all data blocks in the current data page reach the pre-allocated number of storage messages; and a third storage subunit, configured to store the remaining log messages into the next data page, until all data pages in the current data segment reach the pre-allocated storage limit, determine that the corresponding number of log messages have been written into the current data segment, and update the storage location of each log message in the metadata of the corresponding data table.

[0019] Optionally, in one embodiment of the present invention, the expression for calculating the actual storage capacity of the current data page is:

[0020] ,

[0021] in, This indicates the actual storage capacity. This indicates the number of pre-allocated storage entries. Indicates the corresponding first n The size of each data table Indicates the corresponding first n A data table.

[0022] Optionally, in one embodiment of the present invention, the positioning module includes: a first positioning unit, configured to locate at least one data segment to be scanned in the WAL log encoding structure based on the write location information, and to traverse the at least one data segment to be scanned using the identifier or timestamp of the at least one data table to find the corresponding data page; a second positioning unit, configured to traverse the corresponding data page using the timestamp to locate the corresponding data block, and to store the corresponding positioning information in a candidate list; and a scanning unit, configured to search for log information that meets preset query conditions in the candidate list until all data segments to be scanned have been scanned.

[0023] A third aspect of the present invention provides a time-series database, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data query method for WAL logs as described in the above embodiments.

[0024] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the WAL log data query method as described in the above embodiments.

[0025] A fifth aspect of the present invention provides a computer program product, including a computer program, which, when executed, is used to implement the data query method for WAL logs as described above.

[0026] This invention, in response to a log information query command, searches for the metadata of the corresponding data table in the global metadata of the time-series database, performing only necessary data page locking to improve cache utilization. Based on the metadata, it determines the write location information of the query target. Combining this write location information with the WAL (Write-Ahead Logging) encoding structure in memory, it locates the query target, supporting direct database queries on the WAL log while ensuring query performance. This solves the technical problem in related technologies where unnecessary data page locking during data queries leads to low cache utilization, the generation of a large amount of useless cache, and consequently, affects database query performance.

[0027] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0028] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 This is a schematic diagram of the encoding structure of a WAL log provided according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the encoding structure of a WAL log according to an embodiment of the present invention;

[0031] Figure 3 A flowchart illustrating a WAL log data query method according to an embodiment of the present invention;

[0032] Figure 4A schematic diagram illustrating the writing principle of a WAL log data query method according to an embodiment of the present invention;

[0033] Figure 5 A schematic diagram illustrating the query principle of a WAL log data query method according to an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of the structure of a WAL log data query device according to an embodiment of the present invention;

[0035] Figure 7 This is a schematic diagram of the structure of a time-series database provided according to an embodiment of the present invention.

[0036] Among them, 10-WAL log encoding structure, 101-data block, 102-data page, 103-data segment; 20-WAL log data query device, 201-receiving module, 202-search module, 203-location module; 701-memory, 702-processor, 703-communication interface. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0038] The following describes a data query method, apparatus, time-series database, and medium for WAL logs according to embodiments of the present invention with reference to the accompanying drawings. Addressing the technical problems mentioned in the background art, such as unnecessary data page locking during data querying, low cache utilization, and the generation of a large amount of useless cache, which negatively impacts database query performance, the present invention provides a data query method for WAL logs. This method responds to log information query commands by searching for the metadata of the corresponding data table in the global metadata of the time-series database, performing only necessary data page locking to improve cache utilization, and determining the write location information of the query target based on the metadata. By combining the write location information with the WAL log encoding structure in memory, the query target is located, supporting direct database queries on WAL logs while ensuring query performance. Thus, the technical problems of unnecessary data page locking, low cache utilization, and the generation of a large amount of useless cache, which negatively impact database query performance, are solved in the related art.

[0039] Before describing the data query method for WAL logs in this embodiment of the invention, the memory structure involved in this embodiment of the invention will be explained first to facilitate subsequent understanding.

[0040] In this embodiment of the invention, the memory can be composed of multiple WAL log encoding structures, such as... Figure 1 As shown, the WAL log encoding structure 10 of this embodiment may include: data block 101, data page 102 and data segment 103.

[0041] Specifically, there are multiple data blocks 101, each data block 101 stores the WAL log information of a data table, and each data block 101 stores multiple log messages. Each log message includes a data header and data, wherein the data header is the timestamp and sequence number stored for each log message.

[0042] Multiple data pages 102, each data page 102 includes multiple data blocks 101, and the header of each data page records the pre-allocated storage information, sequence number, number of data blocks, identifier of the data table corresponding to each data block 101, timestamp range and offset in the data page.

[0043] Data segment 103, each data segment 103 includes multiple data pages 102.

[0044] Understandably, WAL (Write-Ahead Log) is a widely used technology in databases and other data storage systems, primarily used to ensure data persistence and transaction atomicity. Its core principle is to record modifications to the actual data storage in a log file before any changes are made. The advantage of this is that even in the event of system failures (such as crashes or power outages), data can be recovered based on the records in the log file, ensuring that data is not lost or corrupted, and that transactions either complete entirely or are not executed at all.

[0045] like Figure 2 As shown, the WAL log encoding structure of this embodiment consists of a three-level data structure. The first-level data structure is a block (data block 101). Each data block 101 contains WAL log information from the same data table. Each log information (i.e., a slot) contains two parts: a data header and data. The data header contains the timestamp of the data point plus the corresponding LSN (Log sequence number) information (i.e., the sequence number).

[0046] The second-level data structure is a page (data page 102). Each data page 102 contains n (n>= 1) data blocks 101, and each data block 101 belongs to a different data table. The page header record of each data table contains the pre-allocated slot, LSN, and the number of data blocks 101. Each data block 101 corresponds to the data table ID, timestamp range, and offset in data page 102.

[0047] The third-level structure is a segment (data segment 103), which contains n data pages 102. Data segment 103 is the smallest unit for storing and allocating WAL logs in memory and on disk.

[0048] In this embodiment of the invention, data segments 103 and data pages 102 can be of fixed length, while data blocks 101 can be of floating length. Each data segment is set to a size of 1 MiB (MB), and each data page has a maximum capacity of 64 KiB. Each page header contains the following content:

[0049] 1. The data page 102 stores the data corresponding to the data table id array (i.e., the data table identifier array).

[0050] 2. The range of the global LSN of the data stored in this data page 102.

[0051] For each data block 101 stored in the data page 102, the information of the data block 101 includes the identifier of the corresponding data table, the timestamp range, the global LSN range, and the position of the data block 101 in the data page 102 (offset information).

[0052] The WAL log encoding structure proposed in this embodiment of the invention can be implemented in a block-page-segment format. A data block, as the smallest storage unit, stores log information and records the log header and data. A data page can include multiple data blocks, and the contents of these data blocks are summarized to form a page header. A data segment can include multiple data pages. By partitioning and sorting according to the data table, fragmentation of the same table in the WAL log is reduced, improving data loading and insertion efficiency and reducing space usage. This allows for layer-by-layer storage during data import using a data block-data page-data segment approach, allocating contiguous space in memory and reducing random read / write operations and fragmentation on memory and disk. Furthermore, data queries or retrieval can be performed layer-by-layer using a data segment-data page-data block approach. This solves the technical problems in related technologies where data import requires simultaneous writing to both the WAL log and cache / disk, resulting in high costs and impacting overall import performance. Additionally, storing different data tables in different files during import leads to repeated random read / write operations on memory and disk, causing severe storage fragmentation and affecting subsequent file retrieval.

[0053] Specifically, Figure 3 This is a flowchart illustrating a WAL log data query method provided in an embodiment of the present invention.

[0054] like Figure 3 As shown, this WAL log data query method, applied to a time-series database, includes the following steps:

[0055] In step S301, a log information query instruction is received to obtain the query target.

[0056] As we can understand, time series data is a set of data points recorded in chronological order, with each data point associated with a specific point in time. This type of data typically reflects the changes of a variable at different times, and is an ordered, time-series collection of data. This type of data usually has the following characteristics: Timestamps: Each data point has a timestamp to identify the time the data was generated. Continuity: The data is usually generated continuously at fixed or variable time intervals. High frequency: The data points are generated at a high frequency, such as per second, per minute, or per hour. Multidimensionality: The data can contain multiple dimensions, such as temperature, humidity, and pressure.

[0057] A time-series database is a database system specifically designed for processing time-series data. Time-series data refers to a series of data points arranged in chronological order, with each data point typically representing a measurement or state record at a specific moment.

[0058] The time-series database in this embodiment of the invention can be an Internet of Things (IoT) based time-series database to process and analyze time-series data collected from various IoT devices.

[0059] Based on the characteristics of time-series data, embodiments of the present invention can achieve high-quality querying of log information.

[0060] In actual execution, embodiments of the present invention can receive log information query instructions to clarify the query target in the log information query instructions, namely the log information and corresponding timestamps corresponding to the time-series data stored in the time-series database, so as to locate the log information in subsequent processes and return the results to the user.

[0061] In step S302, in response to the log information query instruction, the metadata of the corresponding data table is searched in the global metadata of the time series database, and the write location information of the query target is determined based on the metadata.

[0062] Furthermore, in this embodiment of the invention, the relevant data tables can be locked in the global metadata of the time-series database according to the log information query instruction, so as to perform metadata query on the locked data tables and obtain the write location information of the query target. The global metadata may include the storage location of each log information.

[0063] In step S303, the query target is located based on the write location information and the pre-built WAL log encoding structure, wherein the WAL log encoding structure includes multiple data blocks, multiple data pages and data segments.

[0064] Based on the write location information and WAL log encoding structure, embodiments of the present invention can locate the query target to determine the data block where the query target is located, thereby realizing data query.

[0065] Optionally, in one embodiment of the present invention, locating the query target based on the write location information and a pre-built WAL log encoding structure includes: locating at least one data segment to be scanned in the WAL log encoding structure based on the write location information, and traversing at least one data segment to be scanned using the identifier or timestamp of at least one data table to find the corresponding data page; traversing the corresponding data page using the timestamp to locate the corresponding data block, and storing the corresponding location information in a candidate list; searching for log information that meets the preset query conditions in the candidate list until all data segments to be scanned have been scanned.

[0066] Understandably, in IoT-based time-series databases, users query stored time-series data. After receiving the query, the time-series database scans and filters the data, then returns the results to the user. Typically, time-series database queries include a time window, meaning only data whose timestamps fall within that window is selected for further processing. Therefore, optimizing and processing the timestamp selection of time-series data has a decisive impact on the performance of the time-series database.

[0067] This invention can simultaneously issue scan requests and return scanned data as needed. The scanning process is as follows:

[0068] Step S1. Obtain the identifier of the WAL data segment corresponding to the data in the table through the table's metadata information;

[0069] Step S2. Locate the position of the data segment to be scanned;

[0070] Step S3. Obtain the header information and perform the following judgments:

[0071] (1) If the identifier array of the data table does not contain the identifiers of the data tables that need to be scanned, then skip the page;

[0072] (2) If the timestamp required for the query is not within the range of the timestamps identified in the corresponding data table, then skip this page;

[0073] (3) For data block information that meets the conditions, record its offset position and store it in the candidate list for scanning;

[0074] Step S4. Read data block information from the candidate list. Read each log entry. If its sequence number meets the database visibility requirements and the corresponding timestamp belongs to a query access, return the data; otherwise, skip it.

[0075] Repeat steps S2-4 until all data segments have been scanned.

[0076] Optionally, in one embodiment of the present invention, before receiving the log information query instruction, the method further includes: receiving a time-series data import instruction and obtaining the WAL log generated during the time-series data import; mapping the log information of the WAL log to the corresponding data table, and writing the data table and the log information into memory composed of the WAL log encoding structure.

[0077] Understandably, time-series databases need to write log information before they can query log information. Here, we will explain the process of writing log information.

[0078] Data ingestion refers to the process of collecting data files from various sources and importing them into a database for storage, processing, and analysis. Data ingestion aims to cleanse data and store it in an accessible and consistent central repository for use within the organization. Data sources include financial systems, third-party data providers, social media platforms, IoT devices, SaaS (Software as a Service) applications, and on-premises business applications such as enterprise resource planning and customer relationship management. These data sources contain both structured and unstructured data. After ingestion, data can be stored in data lakes, data warehouses, integrated data lake warehouses, data marketplaces, relational databases, and document storage systems. Organizations ingest data for business intelligence tasks, as well as for machine learning, predictive modeling, and artificial intelligence applications.

[0079] In this embodiment of the invention, time-series data, such as IoT time-series data, can be obtained through data ingestion, and the time-series data can be mapped to a data table. The data table and the time-series data can then be imported into the generated log information for writing.

[0080] Optionally, in one embodiment of the present invention, writing data tables and log information into memory composed of a pre-built WAL log encoding structure includes: filtering memory that meets preset storage conditions and initializing each data segment in memory to obtain pre-allocated storage information for each data segment; based on the pre-allocated storage information, writing a corresponding number of log information into each data segment in a preset order until all log information has been written.

[0081] Furthermore, in this embodiment of the invention, a whole block of memory can be allocated first and stored using the WAL log writing structure. For each data segment in memory, this embodiment of the invention can initialize the first page and the corresponding data block area.

[0082] In actual execution, embodiments of the present invention can write log information based on the pre-allocated storage information obtained after initialization. For example, based on the pre-allocated storage information, the storage capacity of the current data segment is determined, and the corresponding number of log information is stored. When the remaining space of the current data segment is insufficient to accommodate the new page data structure, a new data segment space is requested.

[0083] The preset order can be a chronological order or other orders, and can be set by those skilled in the art according to the actual situation.

[0084] Optionally, in one embodiment of the present invention, initializing each data segment in memory to obtain pre-allocated storage information for each data segment includes: storing multiple log messages from a corresponding number of log messages into the current data block of the current data page of the current data segment, until the current data block reaches the pre-allocated number of storage messages; storing the remaining log messages into the next data block, until all data blocks in the current data page reach the pre-allocated number of storage messages; storing the remaining log messages into the next data page, until all data pages in the current data segment reach the pre-allocated storage limit, determining that a corresponding number of log messages have been written into the current data segment, and updating the storage location of each log message in the metadata of the corresponding data table, wherein the calculation expression for the actual storage capacity of the current data page is:

[0085] ,

[0086] in, Indicates the actual storage capacity. Indicates the number of pre-allocated storage entries. Indicates the corresponding first n The size of each data table Indicates the corresponding first n A data table.

[0087] For page initialization, the following formula can be used to calculate the number of slots in this embodiment of the invention.

[0088] First, calculate the number of data sources currently being imported (time series data). n Frequency of each data source f and the size of each data entry. mSince IoT time-series data is structured data, the size m of each data entry is equal to the size of each row in the corresponding data table of the data source. This size can be easily obtained from metadata, and it is a commonly used data in databases, so it will not cause additional performance loss.

[0089] ,

[0090] in, The maximum size of the data page is 64KB, and the denominator is the largest single data record among all data sources. tn Indicates the first n One data table, Let be the number of slots. Then you can calculate the maximum number of items that the page can store.

[0091] Subsequently, embodiments of the present invention can use the following formula to calculate the size of the data block corresponding to each table, that is, the number of records that each data block can store:

[0092] ,

[0093] in, tn Indicates the corresponding first n Each data table, meaning the entire data page space, is divided according to the frequency of each data source. This represents the sampling frequency for the nth data table, in Hz, which is the number of samples collected per second. Finally, record the actual size of the page used.

[0094] .

[0095] After obtaining the actual size, the embodiments of the present invention can be carried out according to... The size of the page header determines the actual memory allocated to that page. The allocator then writes the identifier array that needs to be written to the data table. The value, actual size, offset of each data block (i.e., actual position), is entered into the page header of that data page. After completion, the state of the data page is changed to writable.

[0096] Based on the encoding structure of the WAL log, the data writing process in this embodiment of the invention can be as follows: the log information is directly written into the data block data structure of the corresponding table in the allocated page.

[0097] Step S1. Obtain the latest writable data segment;

[0098] Step S2. Update the metadata information of the corresponding table with the information of the data segment;

[0099] Step S3. Locate the data page whose data segment is not full;

[0100] Step S4. Read the page header to obtain the address of the corresponding data block;

[0101] Step S5. Write the latest sequence number, the timestamp of the log information, and the data row, while retaining the address pointer of the current data block;

[0102] Continue writing until the data block is full, then proceed to step S1.

[0103] After the log information is written, this embodiment of the invention can also track the memory usage status of the WAL log and perform the following operations:

[0104] Step S1. After the specified time is met, append the data blocks of the data segments containing log information to the WAL file on the disk;

[0105] Step S2. After the data block of the data segment is full, append the data block to the WAL file on the disk;

[0106] Step S3. Notify the WAL allocator to reclaim the memory of the data blocks that have been written to the disk.

[0107] This invention allows for the conversion of data in the corresponding WAL file into a database standard file and the creation of a checkpoint after the database issues a checkpoint command. The converted WAL file is then archived.

[0108] Combination Figure 4 and Figure 5 The working principle of the log data processing method of the present invention will be described in detail using an example.

[0109] In actual implementation, embodiments of the present invention can implement the writing and querying of log information based on a WAL writer, a WAL allocator, a WAL synchronizer, a WAL asynchronous writer, and a WAL log scanner.

[0110] The writing process can be as follows: Figure 4 As shown.

[0111] The WAL allocator can pre-allocate entire blocks of memory for storing the data structures of data segments. For each data segment, the allocator initializes the first page and the corresponding data block area.

[0112] The WAL writer can directly write log information into the data structure of the corresponding table in the allocated page. Each thread of the writer corresponds to one data table imported from one data source.

[0113] The WAL synchronizer and asynchronous writer are responsible for tracking the memory usage status of the current WAL log. The asynchronous writer ensures that after the database issues a checkpoint command, the data in the corresponding WAL file is converted into the database standard file, and a checkpoint is created. The WAL file that has been converted into the database standard file is then archived.

[0114] Based on the above architecture, the time-series data imported in this embodiment of the invention only needs to be written once, that is, only to the WAL log, and then subsequently written to disk asynchronously, reducing memory copying and waiting time. Leveraging the known characteristics of IoT time-series data, the allocator can pre-calculate the required memory size and allocate whole blocks of memory as needed, reducing memory fragmentation and avoiding memory waste. Each thread can guarantee sequential writing within a data block, without random memory I / O. Based on the known characteristics of IoT time-series data, the WAL writer can ensure that only one thread is writing to the same memory address at a time, eliminating the need to lock the corresponding data structures and reducing the overhead of concurrent writes and lock waiting. The WAL log to database standard file conversion supports mainstream asynchronous writing methods, improving data dumping performance.

[0115] The WAL log data query process can be as follows: Figure 5 As shown.

[0116] By adding an additional scanner for the WAL log, this embodiment of the invention can directly filter pages that do not need to be scanned based on the query timestamp, without the need for additional indexes, thus improving query efficiency and eliminating the extra space consumption caused by indexes. The WAL scanner only scans data from the necessary tables, automatically skipping data from tables not included in the scan, without incurring additional overhead.

[0117] Furthermore, regarding the WAL log data recovery process, the new WAL logs generated by this embodiment of the invention are compatible with the current standard WAL log recovery mechanism, requiring no additional changes for log recovery and reducing database upgrade costs.

[0118] In summary, the encoding method in this embodiment of the invention partitions and sorts data tables, reducing fragmentation of the same table in the WAL log, improving data loading and insertion efficiency, and reducing space consumption. Simultaneously, the writer employs multi-threading, with each thread corresponding to a data block. This ensures that each thread writes sequentially within the data block, eliminating random memory I / O. Furthermore, leveraging the known characteristics of IoT time-series data, the WAL writer guarantees that only one thread is writing to the same memory address at any given time, eliminating the need for locking the corresponding data structure and reducing the overhead of concurrent writes and lock waiting. Additionally, compared to existing technologies, the WAL scanner can directly filter pages that do not need scanning by querying timestamps, eliminating the need for additional indexes, improving query efficiency, and avoiding the additional space consumption caused by indexes. Compared to existing technologies, the WAL scanner only scans data from the necessary tables, automatically skipping data from tables not included in the scan, thus avoiding additional overhead.

[0119] The new WAL log encoding eliminates the need for storing an additional copy of the data, while ensuring that the ACID (Atomicity, Consistency, Isolation, Durability) performance of the database remains unchanged. It supports direct queries of the WAL log while maintaining query performance. Furthermore, it supports asynchronous archiving of the WAL log, allowing the WAL log data files to be converted back to the database file format for persistent storage. Compared to existing technologies, imported time-series data only needs to be written once—to the WAL log—and subsequently persisted to disk asynchronously, reducing memory copying and waiting time.

[0120] The new WAL log uses space pre-allocation and append mode for writing. It pre-allocates contiguous space in memory and writes to disk for persistent storage via append mode, reducing random read / write operations and fragmentation of memory and disk. Based on the known characteristics of IoT time-series data, the allocator can pre-calculate the required memory size and request whole blocks of memory as needed, reducing memory fragmentation and avoiding memory waste. Using the popular append mode also ensures that the WAL log supports mainstream asynchronous writing methods, improving data dumping performance. Furthermore, the new WAL log generated by this invention is compatible with current standard WAL log recovery mechanisms, requiring no additional changes for log recovery and reducing database upgrade costs.

[0121] The WAL log data query method proposed in this embodiment of the invention can, in response to a log information query command, search for the metadata of the corresponding data table in the global metadata of the time-series database, perform only necessary data page locking to improve cache utilization, and determine the write location information of the query target based on the metadata. By combining the write location information with the WAL log encoding structure in memory, the query target can be located. This method supports direct database queries on the WAL log while ensuring query performance. Therefore, it solves the technical problems in related technologies where unnecessary data page locking during data querying leads to low cache utilization, the generation of a large amount of useless cache, and consequently, affects database query performance.

[0122] Next, the data query apparatus for WAL logs according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0123] Figure 6 This is a block diagram of a WAL log data query device according to an embodiment of the present invention.

[0124] like Figure 6 As shown, the WAL log data query device 20 includes: a receiving module 201, a searching module 202, and a locating module 203.

[0125] Specifically, the receiving module 201 is used to receive log information query instructions in order to obtain the query target.

[0126] The lookup module 202 is used to respond to the log information query command, search for the metadata of the corresponding data table in the global metadata of the time series database, and determine the write location information of the query target based on the metadata.

[0127] The positioning module 203 is used to locate the query target based on the write location information and the pre-built WAL log encoding structure, wherein the WAL log encoding structure includes multiple data blocks, multiple data pages and data segments.

[0128] Optionally, in one embodiment of the present invention, the WAL log data query device 20 further includes an acquisition module and a writing module.

[0129] The acquisition module is used to receive time series data import instructions and acquire the WAL logs generated during the time series data import process.

[0130] The write module is used to map the log information of the WAL log to the corresponding data table, and write the data table and log information into memory composed of the WAL log encoding structure.

[0131] Optionally, in one embodiment of the present invention, each data block stores the WAL log information of a data table, and each data block stores multiple log messages. Each log message includes a data header and data, wherein the data header is the timestamp and sequence number stored in each log message.

[0132] Each data page includes multiple data blocks. The header of each data page records the pre-allocated storage information, sequence number, number of data blocks, identifier of the data table corresponding to each data block, timestamp range, and offset within the data page.

[0133] Each data segment consists of multiple data pages.

[0134] Optionally, in one embodiment of the present invention, the writing module includes an initialization unit and a writing unit.

[0135] The initialization unit is used to filter memory that meets the preset storage conditions and initialize each data segment in the memory to obtain the pre-allocated storage information of each data segment.

[0136] The write unit is used to write a corresponding number of log messages into each data segment in a preset order based on pre-allocated storage information, until all log messages have been written.

[0137] Optionally, in one embodiment of the present invention, the initialization unit includes: a first storage subunit, a second storage subunit, and a third storage subunit.

[0138] The first storage subunit is used to store multiple log messages from a corresponding number of log messages into the current data block of the current data page of the current data segment, until the current data block reaches the pre-allocated number of storage messages.

[0139] The second storage sub-unit is used to store the remaining log information into the next data block until all data blocks in the current data page have reached the pre-allocated number of storage entries.

[0140] The third storage subunit is used to store the remaining log information into the next data page until all data pages in the current data segment reach the pre-allocated storage limit, determine that the corresponding number of log information has been written into the current data segment, and update the storage location of each log information into the metadata of the corresponding data table.

[0141] Optionally, in one embodiment of the present invention, the expression for calculating the actual storage capacity of the current data page is:

[0142] ,

[0143] in, Indicates the actual storage capacity. Indicates the number of pre-allocated storage entries. Indicates the corresponding first n The size of each data table Indicates the corresponding first n A data table.

[0144] Optionally, in one embodiment of the present invention, the positioning module 203 includes: a first positioning unit, a second positioning unit, and a scanning unit.

[0145] The first positioning unit is used to locate at least one data segment to be scanned in the WAL log encoding structure based on the write location information, and to traverse at least one data segment to be scanned using the identifier or timestamp of at least one data table in order to find the corresponding data page.

[0146] The second positioning unit is used to traverse the corresponding data pages using timestamps to locate the corresponding data blocks and store the corresponding positioning information in a candidate list.

[0147] The scanning unit is used to search for log information that meets the preset query conditions in the candidate list until all data segments to be scanned have been scanned.

[0148] It should be noted that the foregoing explanation of the WAL log data query method embodiment also applies to the WAL log data query device of this embodiment, and will not be repeated here.

[0149] The WAL log data query device proposed in this embodiment of the invention can, in response to a log information query command, search for the metadata of the corresponding data table in the global metadata of the time-series database, perform only necessary data page locking to improve cache utilization, and determine the write location information of the query target based on the metadata. By combining the write location information with the WAL log encoding structure in memory, the query target can be located, thus supporting direct database queries on the WAL log while ensuring query performance. This solves the technical problem in related technologies where unnecessary data page locking during data querying leads to low cache utilization, the generation of a large amount of useless cache, and consequently, affects database query performance.

[0150] Figure 7 A schematic diagram of the structure of a time-series database provided in an embodiment of the present invention. The time-series database may include:

[0151] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0152] When the processor 702 executes the program, it implements the data query method for the WAL log provided in the above embodiments.

[0153] Furthermore, time-series databases also include:

[0154] Communication interface 703 is used for communication between memory 701 and processor 702.

[0155] The memory 701 is used to store computer programs that can run on the processor 702.

[0156] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0157] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0158] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0159] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0160] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data query method for the WAL log as described above.

[0161] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the WAL log data query method provided in this invention.

[0162] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0163] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0164] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0165] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0166] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0167] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0168] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0169] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for querying data from WAL logs, characterized in that, When applied to time-series databases, the following steps are included: Receive log information query commands to obtain the query target; In response to the log information query command, the metadata of the corresponding data table is searched in the global metadata of the time series database, and the write location information of the query target is determined based on the metadata. Based on the write location information and the pre-built WAL log encoding structure, the query target is located, wherein the WAL log encoding structure includes multiple data blocks, multiple data pages, and data segments; Before receiving the log information query instruction, the method further includes: receiving the time-series data import instruction and obtaining the WAL log generated during the time-series data import; mapping the log information of the WAL log to the corresponding data table, and writing the data table and the log information into the memory composed of the WAL log encoding structure. Each data block stores the WAL log information of one data table, and each data block stores multiple log messages. Each log message includes a data header and data, wherein the data header contains the timestamp and sequence number of each log message. Each data page includes the multiple data blocks, and the header of each data page records the pre-allocated storage information, sequence number, number of data blocks, identifier of the data table corresponding to each data block, timestamp range, and offset within the data page. Each data segment includes the multiple data pages.

2. The WAL log data query method according to claim 1, characterized in that, The step of writing the data table and the log information into memory composed of the pre-built WAL log encoding structure includes: Filter the memory that meets the preset storage conditions, and initialize each data segment in the memory to obtain the pre-allocated storage information of each data segment; Based on the pre-allocated storage information, a corresponding number of log messages are written into each data segment in a preset order until all log messages have been written.

3. The WAL log data query method according to claim 2, characterized in that, The initialization of each data segment in the memory to obtain pre-allocated storage information for each data segment includes: Store multiple log entries from the corresponding number of log entries into the current data block of the current data page of the current data segment until the current data block reaches the pre-allocated storage number; Store the remaining log information into the next data block until all data blocks in the current data page have reached the pre-allocated number of storage entries; The remaining log information is stored in the next data page until all data pages in the current data segment reach the pre-allocated storage limit. Once the corresponding number of log information has been written into the current data segment, the storage location of each log information is updated in the metadata of the corresponding data table.

4. The WAL log data query method according to claim 3, characterized in that, The expression for calculating the actual storage capacity of the current data page is: ， in, This indicates the actual storage capacity. This indicates the number of pre-allocated storage entries. Indicates the corresponding first n The size of each data table Indicates the corresponding first n A data table.

5. The WAL log data query method according to claim 1, characterized in that, The process of locating the query target based on the write location information and the pre-built WAL log encoding structure includes: Based on the write location information, locate at least one data segment to be scanned in the WAL log encoding structure, and use the identifier or timestamp of the at least one data table to traverse the at least one data segment to be scanned in order to find the corresponding data page; The corresponding data pages are traversed using the timestamps to locate the corresponding data blocks, and the corresponding location information is stored in a candidate list. Search the candidate list for log information that meets the preset query conditions until all data segments to be scanned have been scanned.

6. A data query device for WAL logs, characterized in that, Applied to time series databases, including: The receiving module is used to receive log information query commands in order to obtain the query target; The lookup module is used to respond to the log information query command, search for the metadata of the corresponding data table in the global metadata of the time series database, and determine the write location information of the query target based on the metadata; The positioning module is used to locate the query target based on the written location information and the pre-built WAL log encoding structure, wherein the WAL log encoding structure includes multiple data blocks, multiple data pages and data segments; The acquisition module is used to receive a time-series data import instruction and acquire the WAL log generated during the time-series data import; the writing module is used to map the log information of the WAL log to the corresponding data table and write the data table and the log information into the memory composed of the WAL log encoding structure. Each data block stores the WAL log information of one data table, and each data block stores multiple log messages. Each log message includes a data header and data, wherein the data header contains the timestamp and sequence number of each log message. Each data page includes the multiple data blocks, and the header of each data page records the pre-allocated storage information, sequence number, number of data blocks, identifier of the data table corresponding to each data block, timestamp range, and offset within the data page. Each data segment includes the multiple data pages.

7. A time-series database, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the data query method for WAL logs as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the data query method for WAL logs as described in any one of claims 1-5.

Images