Time series data read-write method, system and device under double-structure memory table and medium

Abstract

The invention provides a time sequence data read-write method, system and device under a double-structure memory table and a medium, and the method comprises the steps: constructing a memory table structure which comprises a writable list and a non-writable list; if a data writing request is received, creating a first memory table and a first writable list to perform data writing until the data volume reaches a data volume threshold value, sequencing the data in the first writable list, then converting the data into a non-writable list, and creating a second writable list to continue to write the data; if the first memory table meets the disk refreshing condition, refreshing the data in the first memory table, and creating a second memory table to continue to write the data; and if a memory table query request is received, generating first ordered data according to the current writable list or the current writable list and at least one unwritable list, and returning the first ordered data. By optimizing the memory table structure, the writable disordered data list and the unwritable ordered data list are maintained, and the read-write performance of the time sequence database is improved.

Description

Technical Field

[0001] This application relates to the field of database management technology, and in particular to a method, system, device and medium for reading and writing time-series data under a dual-structure memory table. Background Technology

[0002] Time-series data refers to data sequences recorded in chronological order. It is widely used in fields such as the Industrial Internet of Things (IIoT) and smart manufacturing. Typical applications include high-frequency data collection from sensors of equipment operating parameters (such as temperature, pressure, and vibration). With the continuous improvement of automation and informatization levels, data acquisition frequencies can reach millisecond or even microsecond levels, resulting in the generation of massive amounts of time-series data in a short period, placing extremely high write throughput requirements on the underlying storage system. Simultaneously, time-series data queries are typically based on a time range, requiring query results to strictly maintain temporal order and efficiently handle data with duplicate timestamps. Therefore, time-series databases commonly adopt storage engines based on the LSM (Log Structured Merge Tree) architecture, such as TDengine (a data platform developed by TaoData), IoTDB (Internet of Things Database), and DolphinDB (a high-performance distributed time-series database).

[0003] In related technologies, MemTable is a crucial component of the LSM tree storage engine, used to store data written by clients but not yet written to disk; that is, data written by clients is first written to the MemTable in memory. Time-series databases typically use SkipList as the underlying data structure of the in-memory table, supporting ordered insertion and efficient searching. Some in-memory tables use the unordered List data structure to ensure write throughput. However, in high-concurrency read-write mixed scenarios, the above solutions still face the dual challenge of balancing write performance and query efficiency. Using an ordered structure sacrifices write speed, while using an unordered structure increases query overhead and lock contention, resulting in low read-write performance of time-series databases in high-concurrency read-write mixed scenarios. Summary of the Invention

[0004] This application discloses a method, system, device, and medium for reading and writing time-series data under a dual-structure memory table, which is used to solve the technical problem of low read and write performance of time-series databases in high-concurrency read and write mixed scenarios.

[0005] This application provides a time-series data read / write method under a dual-structure memory table. The method includes: constructing a memory table structure, the memory table structure including a writable list and a non-writable list, wherein the writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion; if a data write request is received, a first memory table and a first writable list in the first memory table are created to write data to the first writable list until the data volume in the first writable list reaches the data volume threshold, the data in the first writable list is sorted by time and converted into the non-writable list, and a second writable list is created to continue writing data; if the first memory table meets a preset disk flushing condition, all data in the first memory table is flushed to disk, and a second memory table is created to continue writing data; if a memory table query request is received, first ordered data is generated according to the current writable list, or the current writable list and at least one non-writable list are merged and sorted to generate first ordered data, and the first ordered data is returned as the query result.

[0006] In one embodiment of this application, flushing all data in the first memory table to disk includes: calling a flushing thread, blocking new memory table query requests, the flushing thread holding a write lock on the memory table; if the first memory table consists entirely of non-writable lists, then sorting all non-writable lists chronologically to generate second ordered data, and writing the second ordered data to disk; if the first memory table contains a target writable list currently being written to, then determining whether the data in the target writable list is ordered; if not ordered, then sorting the data in the target writable list chronologically, and sorting the sorted target writable list and all non-writable lists chronologically to generate third ordered data, and writing the third ordered data to disk; if ordered, then directly sorting the target writable list and all non-writable lists chronologically to generate fourth ordered data, and writing the fourth ordered data to disk.

[0007] In one embodiment of this application, after flushing all data in the first memory table to disk, the method further includes: detecting whether the first memory table is in a queried state; if not, directly releasing the memory resources of the first memory table; if it is, continuously detecting whether the first memory table is in a queried state until it is no longer in a queried state, and then releasing the memory resources of the first memory table.

[0008] In one embodiment of this application, the step of merging and sorting the current writable list and at least one non-writable list to generate the first ordered data includes: calling a query thread to determine whether the data in the current writable list is ordered, wherein the query thread holds a memory table read lock and a reference queue; if ordered, directly adding the current writable list and the at least one non-writable list to the reference queue; if unordered, copying the current writable list to obtain a copy list, and adding the copy list and the at least one non-writable list to the reference queue; releasing the memory table read lock, and sorting the data lists in the reference queue according to time sequence to generate the first ordered data.

[0009] In one embodiment of this application, after returning the first ordered data as a query result, the method further includes: if the copy list exists in the reference queue, then releasing the memory resources of the copy list; if the target memory table queried by the memory table query request has been flushed to disk and the target memory table is not in a queried state, then releasing the memory resources of the target memory table.

[0010] In one embodiment of this application, the adjustment method of the data volume threshold includes: monitoring the number of times the writable list is converted into the non-writable list within a preset time period; if the number is greater than a preset first number threshold, then the data volume threshold is gradually increased until a preset maximum data volume threshold is reached; if the number is less than a preset second number threshold, then the data volume threshold is gradually decreased until a preset minimum data volume threshold is reached; wherein, the second number threshold is less than the first number threshold, and the minimum data volume threshold and the maximum data volume threshold are set according to the memory table flush threshold.

[0011] In one embodiment of this application, writing data into the first writable list includes: calling a write thread to generate a timestamp for each piece of data to be written, wherein the write thread holds a memory table write lock; writing the data with the timestamp into the first writable list, and generating a table identifier for the first writable list based on the timestamp sequence of the data written in the first writable list, wherein the table identifier is used to mark whether the data in the first writable list is ordered.

[0012] This application also provides a time-series data read / write system under a dual-structure memory table. The system includes: a construction module for constructing a memory table structure, the memory table structure including a writable list and a non-writable list, wherein the writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion; a writing module for creating a first memory table and a first writable list in the first memory table if a data write request is received, to write data to the first writable list until the data volume in the first writable list reaches the data volume threshold, then sorting the data in the first writable list by time and converting it into the non-writable list, and simultaneously creating a second writable list to continue writing data; a disk flushing module for flushing all data in the first memory table to disk if the first memory table meets preset disk flushing conditions, and simultaneously creating a second memory table to continue writing data; and a query module for generating first ordered data based on the current writable list, or merging and sorting the current writable list with at least one non-writable list to generate first ordered data, and returning the first ordered data as the query result if a memory table query request is received.

[0013] This application also provides an electronic device, including: a processor; and a storage device for storing a program, which, when executed by the processor, causes the electronic device to implement the timing data read / write method under the dual-structure memory table as described above.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a computer's processor, causes the computer to perform a timing data read / write method under a dual-structure memory table as described above.

[0015] The beneficial effects of this application: This application provides a method, apparatus, device, and medium for time-series data read / write under a dual-structure memory table. First, a memory table structure is constructed, including a writable list and a non-writable list. The writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion. Then, if a data write request is received, a first memory table and a first writable list within the first memory table are created to write data to the first writable list until the data volume in the first writable list reaches the data volume threshold. Then, the data in the first writable list is sorted by time sequence and converted to a non-writable list. Simultaneously, a second writable list is created to continue writing data. Then, if the first memory table meets preset disk flushing conditions, all data in the first memory table is flushed to disk. Simultaneously, a second memory table is created to continue writing data. If a data write request is received... When a query request is received in the memory table, the first ordered data is generated based on the current writable list, or the current writable list is merged and sorted with at least one non-writable list to generate the first ordered data, which is then returned as the query result. By optimizing the memory table structure of the time-series database storage engine, a writable unordered data list and multiple non-writable ordered data lists are maintained. The writable list's support for unordered writing improves the data writing speed, while the ordered nature of the non-writable lists ensures that the data is ordered in the time dimension during data flushing or reading through multi-way merging. Especially in the case of parallel data read and write, it can maintain high write throughput while reducing the time for data sorting during queries and the time for query threads to hold read locks, effectively improving the read and write performance of the time-series database in high-concurrency read and write mixed scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] In the attached diagram:

[0018] Figure 1 This is a schematic diagram illustrating the implementation environment of a timing data read / write system under a dual-structure memory table, as shown in an exemplary embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating a timing data read / write method under a dual-structure memory table, as shown in an exemplary embodiment of this application.

[0020] Figure 3 This is a schematic diagram illustrating a memory table structure in an exemplary embodiment of this application;

[0021] Figure 4 This is a flowchart illustrating a time-series database data writing process, as shown in an exemplary embodiment of this application;

[0022] Figure 5 This is a block diagram illustrating a timing data read / write system under a dual-structure memory table, as shown in an exemplary embodiment of this application.

[0023] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0024] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0026] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0027] MemTable is a crucial component of the LSM tree storage engine, used to store data written by clients but not yet written to disk; that is, data written by clients is first written to the MemTable in memory. Time-series databases typically use SkipList as the underlying data structure for in-memory tables, supporting ordered insertion and efficient searching. Some in-memory tables use the unordered List data structure to ensure write throughput. However, the inventors of this application have found that in high-concurrency read-write mixed scenarios, there is still a dual challenge between write performance and query efficiency. Using an ordered structure sacrifices write speed, while using an unordered structure does not guarantee order in the time dimension, thus requiring sorting of all data in the in-memory table during queries, increasing query overhead. Furthermore, if there are concurrent writes to the in-memory table during data queries, the data list needs to be cloned, and each thread needs to clone and sort the data list during concurrent queries. The query thread holds a read lock while cloning data, and the write thread needs to acquire a write lock, causing lock contention and degrading write performance. Therefore, time-series databases have low read-write performance in high-concurrency read-write mixed scenarios.

[0028] Therefore, please see Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation environment of a timing data read / write system under a dual-structure memory table, as shown in an exemplary embodiment of this application. Figure 1 As shown, the implementation environment may include a time-series data read / write system 110 with a dual-structure memory table and a computer device 120. The time-series data read / write system 110 with a dual-structure memory table can be set up within the computer device 120 for reading and writing data in the time-series database. The computer device 120 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, or a neural network computer. This time-series data read / write system 110 with a dual-structure memory table optimizes the memory table structure of the time-series database storage engine, maintaining a writable unordered data list and multiple non-writable ordered data lists. The writable list's support for unordered writing improves the data write speed, while the ordered nature of the non-writable lists ensures that data is ordered in a temporal dimension during data flushing or reading through multi-way merging. Especially in parallel data read / write scenarios, it maintains high write throughput while reducing the time spent sorting data during queries and the time the query thread holds read locks, effectively improving the read / write performance of the time-series database in high-concurrency read / write scenarios.

[0029] Please see Figure 2 , Figure 2 This is a flowchart illustrating a timing data read / write method under a dual-structure memory table, as shown in an exemplary embodiment of this application. This method can be applied to... Figure 1The implementation environment shown is specifically executed by the timing data read / write system 110 under the dual-structure memory table in that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which this method is applicable.

[0030] like Figure 2 As shown, in an exemplary embodiment, the timing data read / write method under a dual-structure memory table includes at least steps S210 to S240, which are described in detail below:

[0031] Step S210: Construct a memory table structure, which includes a writable list and a non-writable list. The writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion.

[0032] Unlike other time-series databases that use a single skip list or list structure, the in-memory table structure maintains a writable list and multiple non-writable lists, i.e., a dual-structure (one writable structure part and the other non-writable structure part) in-memory table. The writable list does not guarantee the order of written data, that is, it may be unordered or ordered, while the data in the non-writable list is ordered in the time dimension and cannot be modified.

[0033] In addition, each data list in the memory table structure has a flag to record whether the data in the data list is ordered, and there is also a set to record the query threads that reference the data list.

[0034] Step S220: If a data write request is received, a first memory table and a first writable list in the first memory table are created to write data to the first writable list until the amount of data in the first writable list reaches the data amount threshold. Then, the data in the first writable list is sorted by time and converted into an unwritable list. At the same time, a second writable list is created to continue writing data.

[0035] The first memory table is a variable memory table, meaning it does not meet the conditions for flushing to disk; the data volume threshold can be dynamically adjusted based on the number of times the writable list is converted to the non-writable list within a preset time period.

[0036] In this embodiment, when the amount of data written to the first writable list reaches a data volume threshold, the written data is sorted and converted into an unwritable list, and a new second writable list is created to continue providing writing services. The second writable list and the subsequently created writable lists are also converted into unwritable lists after sorting the written data when the amount of data written reaches the data volume threshold.

[0037] Specifically, writing data to the first writable list includes: calling a write thread to generate a timestamp for each piece of data to be written, with the write thread holding a write lock on the memory table; writing the timestamped data to the first writable list, and generating a table identifier for the first writable list based on the timestamp sequence of the data written to the first writable list, wherein the table identifier is used to mark whether the data in the first writable list is ordered.

[0038] It's important to note that the writing thread holds a write lock on the memory table, allowing it to insert data into the writable list. The timestamps of data to be written can be repeated, and when data with the same timestamp is queried, the last written value will be returned. Additionally, the table identifier can be written to the flag in the first writable list.

[0039] In this embodiment, the time characteristics of the data are strictly preserved through a timestamp mechanism, which supports subsequent data sorting. The write lock mechanism ensures the security of data writing in high-concurrency read-write mixed scenarios. The dynamic table identifier can intelligently identify the order of the writable list, significantly reducing unnecessary subsequent sorting operations. Furthermore, the entire writing process supports unordered data writing, maintaining a high write throughput and optimizing the memory table writing mechanism of the time-series database, making it particularly suitable for high-frequency time-series data writing scenarios.

[0040] For example, the data in the first writable list is sorted chronologically based on the timestamps attached to the data in the first writable list.

[0041] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating a memory table structure as shown in an exemplary embodiment of this application. Figure 3 As shown, the in-memory table structure contains a writable list and two sorted non-writable lists. The data sequence written to the writable list is 11, 15, 12, 14, 16, 20, 13...17, which is an unordered data sequence. The data sequences in the two non-writable lists are ordered data sequences (1-7 and 6-12). When the amount of data in the writable list reaches a set data volume threshold, its unordered data sequence is sorted. The data in the unordered data list (11-20) is converted into a new non-writable list, which together with the original two non-writable lists to form a complete ordered dataset. At the same time, a new writable list is also created to continue providing data writing services. This design refines the in-memory table into writable and non-writable lists, which not only ensures writing efficiency but also ensures that ordered data can be quickly merged and returned during queries.

[0042] In step S230, if the first memory table meets the preset disk flushing conditions, all data in the first memory table is flushed to disk, and at the same time, a second memory table is created to continue writing data.

[0043] The conditions for triggering the disk flushing operation include, but are not limited to, the total amount of data in the first memory table reaching a preset disk flushing threshold, the lifespan of the memory table objects in the first memory table reaching a preset maximum lifespan, the memory usage rate reaching a preset warning threshold, and receiving a disk flushing command manually triggered by the administrator; and the first memory table that meets the disk flushing conditions is an immutable memory table.

[0044] In this embodiment, when the first memory table meets the preset disk flushing conditions, all data in the first memory table is written to the disk. The data in the disk file is ordered in the time dimension, and a new second memory table is created to continue providing writing services. The second memory table and the subsequently created memory tables also flush all the data written to them when the preset disk flushing conditions are met.

[0045] Specifically, flushing all data in the first memory table to disk includes: calling the flushing thread, blocking new memory table query requests, and the flushing thread holding a write lock on the memory table; if the first memory table consists entirely of non-writable lists, then sorting all non-writable lists by time to generate second ordered data, and writing the second ordered data to disk; if the first memory table contains a target writable list where data is being written, then determining whether the data in the target writable list is ordered; if it is not ordered, then sorting the data in the target writable list by time, and sorting the sorted target writable list and all non-writable lists by time to generate third ordered data, and writing the third ordered data to disk; if it is ordered, then directly sorting the target writable list and all non-writable lists by time to generate fourth ordered data, and writing the fourth ordered data to disk.

[0046] It should be noted that the impact of concurrent queries needs to be considered when flushing the memory table to disk. Therefore, all new query threads must wait for the flush to finish. If a query thread is already accessing the writable list and the writable list is marked as unordered, the current writable list in the memory table is cloned and replaced.

[0047] In this embodiment, when flushing all data in the first memory table to disk, since the unwritable list in the first memory table is an ordered list of data, it is only necessary to sort the data in the writable list. Furthermore, the sorting status of the writable list is dynamically identified, and only the unordered data is sorted. Then, the unordered data is merged and sorted together with the unwritable list. The entire flushing process ensures that the data is reliably written to disk while minimizing the impact of sorting operations on system performance, achieving a good balance between write throughput and query efficiency.

[0048] Furthermore, after flushing all data in the first memory table to disk, the process also includes: checking whether the first memory table is in a queried state; if not, directly releasing the memory resources of the first memory table; if it is, continuously checking whether the first memory table is in a queried state until it is no longer in a queried state, and then releasing the memory resources of the first memory table.

[0049] In this embodiment, intelligent reclamation of memory resources is achieved through a reference counting mechanism. That is, after flushing to disk, the query status of the memory table is automatically detected to avoid forcibly releasing data that is being queried. This not only prevents memory leaks but also avoids query interruption, effectively ensuring the reliability of query results.

[0050] Please see Figure 4 , Figure 4 This is a flowchart illustrating a time-series database data writing process, as shown in an exemplary embodiment of this application. Figure 4 As shown, the steps for writing data to a time-series database are detailed below: First, the writing module writes data to the writable list in the memory table and generates a timestamp for each data entry. As the data volume grows, a table type conversion operation is triggered based on a threshold control, converting the writable list to a non-writable list. Then, a disk flushing operation is triggered based on the disk flushing conditions. The disk flushing module flushes the memory tables that meet the flushing conditions to the disk, persisting the data in the memory tables to a multi-level disk structure (level 0, level 1, level 2, etc.). During this process, the non-writable list is marked as read-only to ensure data consistency and query performance. At the same time, new data continues to be written to the updated writable list, forming an efficient cyclical collaborative workflow between memory and disk, achieving high-throughput data writing.

[0051] Step S240: If a memory table query request is received, generate first ordered data based on the current writable list, or merge and sort the current writable list with at least one non-writable list to generate first ordered data, and return the first ordered data as the query result.

[0052] The memory table query request may be executed concurrently with the corresponding data write operation of the memory table being queried.

[0053] In this embodiment, when a memory table query request is received, the memory table may only contain a writable list. In this case, the first ordered data is generated only based on the current writable list. If the data in the current writable list is ordered, the first ordered data is generated directly. If the data in the current writable list is unordered, the data in the current writable list is copied and sorted to generate the first ordered data. However, if the memory table contains both a writable list and a non-writable list, the current writable list and at least one non-writable list need to be merged and sorted before generating the first ordered data.

[0054] Specifically, the process of merging and sorting the current writable list and at least one non-writable list to generate the first ordered data includes: calling a query thread to determine whether the data in the current writable list is ordered; the query thread holding a memory table read lock and a reference queue; if ordered, directly adding the current writable list and at least one non-writable list to the reference queue; if unordered, copying the current writable list to obtain a copy list, and adding the copy list and at least one non-writable list to the reference queue; releasing the memory table read lock, and sorting the data lists in the reference queue according to time sequence to generate the first ordered data.

[0055] In this embodiment, when querying data in the memory table, since the non-writable lists in the memory table are all ordered data lists, it is only necessary to sort the data in the writable lists. Furthermore, the sorting status of the writable lists is dynamically identified, and only the unordered data in the writable lists is copied and sorted. Then, it is merged and sorted together with the non-writable lists. This not only avoids the overall copying and sorting of data in the memory table during the query, but also does not block the write operation. In addition, the query thread holds the memory table read lock for a short time during the preparation phase, and the execution phase is a lock-free operation, that is, it only needs to hold the read lock briefly to obtain a data snapshot, which greatly reduces lock contention. Therefore, it can significantly optimize the concurrent read and write performance and achieve the optimal balance of read and write performance of time-series databases in high-concurrency read and write mixed scenarios.

[0056] Furthermore, after returning the first ordered data as the query result, the method also includes: if a copy list exists in the reference queue, then release the memory resources of the copy list; if the target memory table queried by the memory table query request has been flushed to disk and the target memory table is not in a queried state, then release the memory resources of the target memory table.

[0057] In this embodiment, an intelligent memory management mechanism is used to achieve efficient resource reclamation. Specifically, the temporary copy of the list memory is released immediately after the query is completed, and the abandoned memory table that has been flushed to disk is automatically reclaimed when there is no query reference, which significantly reduces the consumption of memory resources.

[0058] In one embodiment, the method for adjusting the data volume threshold includes: monitoring the number of times the writable list is converted to the non-writable list within a preset time period; if the number is greater than a preset first threshold, the data volume threshold is gradually increased until a preset maximum data volume threshold is reached; if the number is less than a preset second threshold, the data volume threshold is gradually decreased until a preset minimum data volume threshold is reached; wherein, the second threshold is less than the first threshold, and the minimum and maximum data volume thresholds are set according to the memory table flush threshold.

[0059] In this embodiment, considering that if the data volume threshold is too large, there will be too much data in the writable list, resulting in high overhead for query thread cloning and sorting in the case of disorder, while if the data volume threshold is too small, there will be too many unwritable lists, increasing the overhead of merge sorting, the data volume threshold for converting the writable list to the unwritable list is dynamically adjusted based on the number of times the writable list is converted to the unwritable list within a preset time (i.e., the data write speed of the memory table). A gradual adjustment strategy is adopted to avoid performance fluctuations caused by threshold abrupt changes. At the same time, an upper and lower bound protection mechanism is set to prevent the threshold from going out of bounds, ensuring system stability and achieving a dynamic balance between high throughput writes and low latency queries.

[0060] For example, the minimum data volume threshold and the maximum data volume threshold are set according to the memory table flush threshold. For example, the minimum data volume threshold is set to the memory table flush threshold * 0.001, and the maximum data volume threshold is set to the memory table flush threshold * 0.2. The values ​​0.001 and 0.2 can be adjusted according to specific circumstances and requirements. This application embodiment does not impose any restrictions. In addition, the initial data volume threshold is set to the minimum data volume threshold.

[0061] For example, if the number of times the writable list is converted to the non-writable list within a preset time (e.g., within 5 seconds) is greater than 10, the threshold is doubled until the upper limit of the threshold is reached. This increases the threshold during peak write periods so that there are not too many non-writable lists in the memory table. If the number of times the writable list is converted to the non-writable list within a preset time (e.g., 5 seconds) is less than 2, the threshold is halved until the lower limit of the threshold is reached. This decreases the threshold during off-peak write periods so that there is not too much data accumulating in the writable list, which helps to reduce query overhead.

[0062] The aforementioned time-series data read / write method under a dual-structure memory table first constructs a memory table structure, including a writable list and a non-writable list. The writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion. Next, if a data write request is received, a first memory table and a first writable list within the first memory table are created to write data to the first writable list until the data volume in the first writable list reaches the data volume threshold. Then, the data in the first writable list is sorted chronologically and converted to a non-writable list. Simultaneously, a second writable list is created to continue writing data. Then, if the first memory table meets preset disk flushing conditions, all data in the first memory table is flushed to disk. Simultaneously, a second memory table is created to continue writing data. If a memory table query request is received, then... The current writable list generates the first ordered data, or the current writable list and at least one non-writable list are merged and sorted to generate the first ordered data, which is then returned as the query result. By optimizing the memory table structure of the time-series database storage engine, a writable unordered data list and multiple non-writable ordered data lists are maintained. The writable list's support for unordered writing improves the data writing speed, while the ordered nature of the non-writable lists ensures that the data is ordered in a temporal dimension during data flushing or reading through multi-way merging. Especially in the case of parallel data read and write, it can maintain high write throughput while reducing the time for data sorting during queries and the time for query threads to hold read locks, effectively improving the read and write performance of the time-series database in high-concurrency read and write mixed scenarios.

[0063] Please see Figure 5 , Figure 5 This is a block diagram illustrating a timing data read / write system under a dual-structure memory table, as shown in an exemplary embodiment of this application. This system can be applied to... Figure 1 The implementation environment shown is intended to illustrate the system, but it should be understood that the system can also be applied to other exemplary implementation environments. This embodiment does not limit the implementation environment to which the system is applicable.

[0064] like Figure 5 As shown, in an exemplary embodiment, the time-series data read / write system 500 under a dual-structure memory table includes at least a construction module 510, a writing module 520, a disk flushing module 550, and a query module 530, which are described in detail below:

[0065] Module 510 is used to construct an in-memory table structure. The in-memory table structure includes a writable list and a non-writable list. The writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion.

[0066] The writing module 520 is used to create a first memory table and a first writable list in the first memory table if a data writing request is received, so as to write data into the first writable list until the amount of data in the first writable list reaches the data amount threshold. Then, the data in the first writable list is sorted by time and converted into an unwritable list. At the same time, a second writable list is created to continue writing data.

[0067] The disk flushing module 530 is used to flush all data in the first memory table to disk if the first memory table meets the preset disk flushing conditions, and at the same time, create a second memory table to continue writing data.

[0068] The query module 540 is used to generate first ordered data based on the current writable list if a memory table query request is received, or to merge and sort the current writable list with at least one non-writable list to generate first ordered data, and return the first ordered data as the query result.

[0069] It should be noted that the timing data read / write system under the dual-structure memory table provided in the above embodiments and the timing data read / write method under the dual-structure memory table provided in the above embodiments belong to the same concept. The content of the operation performed by each module has been described in detail in the method embodiments, and will not be repeated here.

[0070] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0071] like Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from Storage Unit 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0072] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0073] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0074] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a computer's processor, the computer program causes the computer to perform a timing data read / write method under a dual-structure memory table as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not deployed within that electronic device.

[0075] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0076] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for sequential data read / write under a dual-structure memory table, characterized in that, The method includes: Construct a memory table structure, which includes a writable list and a non-writable list. The writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion. If a data write request is received, a first memory table and a first writable list in the first memory table are created to write data to the first writable list until the amount of data in the first writable list reaches the data amount threshold. Then, the data in the first writable list is sorted by time and converted into the unwritable list. At the same time, a second writable list is created to continue writing data. If the first memory table meets the preset flushing conditions, then all data in the first memory table will be flushed to disk, and at the same time, a second memory table will be created to continue writing data. If a memory table query request is received, first ordered data is generated based on the current writable list, or the current writable list and at least one non-writable list are merged and sorted to generate first ordered data, and the first ordered data is returned as the query result.

2. The timing data read / write method under a dual-structure memory table according to claim 1, characterized in that, The step of flushing all data in the first memory table to disk includes: The disk flushing thread is invoked, blocking new memory table query requests. The disk flushing thread holds a write lock on the memory table. If all items in the first memory table are non-writable lists, then sort all non-writable lists in chronological order to generate second ordered data, and write the second ordered data to disk. If the first memory table contains a target writable list where data is being written, then determine whether the data in the target writable list is ordered; If there is no order, the data in the target writable list is sorted in chronological order, and the sorted target writable list and all non-writable lists are sorted in chronological order to generate third ordered data, and the third ordered data is written to disk. If ordered, the target writable list and all non-writable lists are directly sorted by time to generate the fourth ordered data, and the fourth ordered data is written to the disk.

3. The timing data read / write method under a dual-structure memory table according to claim 2, characterized in that, After flushing all data from the first memory table to disk, the process further includes: Check whether the first memory table is in a queried state; If not, the memory resources of the first memory table are released directly; If it is, then continuously check whether the first memory table is in a queried state until it is no longer in a queried state, and then release the memory resources of the first memory table.

4. The timing data read / write method under a dual-structure memory table according to claim 1, characterized in that, The step of merging and sorting the current writable list with at least one non-writable list to generate the first ordered data includes: The query thread is invoked to determine whether the data in the currently writable list is ordered. The query thread holds a memory table read lock and a reference queue. If ordered, the currently writable list and the at least one unwritable list are directly added to the reference queue; If there is no order, the currently writable list is copied to obtain a copy list, and the copy list and the at least one unwritable list are added to the reference queue; Release the memory table read lock, sort the data lists in the reference queue according to time sequence, and generate the first ordered data.

5. The timing data read / write method under a dual-structure memory table according to claim 4, characterized in that, After returning the first ordered data as the query result, the method further includes: If the copy list exists in the reference queue, then release the memory resources of the copy list; If the target memory table requested by the memory table query request has been flushed to disk and the target memory table is not in a queried state, then the memory resources of the target memory table are released.

6. The timing data read / write method under a dual-structure memory table according to claim 1, characterized in that, The adjustment methods for the data volume threshold include: Monitor the number of times the writable list is converted into the non-writable list within a preset time period; If the number of times exceeds the preset first threshold, the data volume threshold is gradually increased until the preset maximum data volume threshold is reached. If the number of times is less than the preset second number threshold, the data volume threshold is gradually reduced until the preset minimum data volume threshold is reached. Wherein, the second number threshold is less than the first number threshold, and the minimum data volume threshold and the maximum data volume threshold are set according to the memory table flush threshold.

7. The timing data read / write method under a dual-structure memory table according to any one of claims 1 to 6, characterized in that, The step of writing data into the first writable list includes: The write thread is invoked to generate a timestamp for each piece of data to be written, and the write thread holds a write lock on the memory table. The data with the timestamp is written into the first writable list, and a table identifier for the first writable list is generated based on the timestamp sequence of the data written into the first writable list, wherein the table identifier is used to mark whether the data in the first writable list is ordered.

8. A timing data read / write system under a dual-structure memory table, characterized in that, The system includes: A construction module is used to construct an in-memory table structure, which includes a writable list and a non-writable list. The writable list supports unordered writing and is configured with a data volume threshold that triggers table type conversion. The writing module is used to create a first memory table and a first writable list in the first memory table if a data write request is received, so as to write data into the first writable list until the amount of data in the first writable list reaches the data amount threshold. Then, the data in the first writable list is sorted by time and converted into the unwritable list. At the same time, a second writable list is created to continue writing data. The disk flushing module is used to flush all data in the first memory table to disk if the first memory table meets the preset disk flushing conditions, and at the same time, create a second memory table to continue writing data. The query module is used to generate first ordered data based on the current writable list if a memory table query request is received, or to merge and sort the current writable list with at least one non-writable list to generate first ordered data, and return the first ordered data as the query result.

9. An electronic device, characterized in that, include: processor; A storage device for storing a program, which, when executed by the processor, causes the electronic device to implement the timing data read / write method under a dual-structure memory table as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by the computer's processor, causes the computer to perform the timing data read / write method under a dual-structure memory table as described in any one of claims 1 to 7.

Images