Embedded ring time series data storage method based on key-value retrieval

Abstract

The application discloses a key value retrieval-based embedded ring-shaped time sequence data storage method and relates to the technical field of data storage. A global storage space is determined, a second local storage space is traversed, an idle local storage space with a null key value pair existing in the head is searched, and a current control local storage space is determined. A first key value pair head address of the current control local storage space is traversed, a write control address is determined based on a preset condition, and when a key value pair is written in the write control address, if the remaining space of the current local storage space is insufficient, the next local storage space is erased. The local storage space is stored in a ring shape, the earliest data in the warehouse can be automatically reset, when data occurring within a specified time is queried, the data can be serialized according to a specified predicate logic, the earliest data can be erased, the local storage space is used evenly, and the wear of the flash memory can be balanced.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and more specifically to an embedded circular time series data storage method based on key-value retrieval. Background Technology

[0002] Traditional embedded time-series data uses a linear sequential storage model with no circular reuse mechanism. Data is only appended in one direction and cannot be automatically reset or cleaned up older data. Log data is continuously collected and generated around the clock, and old data permanently occupies storage space while new data accumulates. Embedded devices generally have limited storage space, and after long-term operation, they are prone to storage exhaustion and memory overflow, leading to device shutdown, interrupted data collection, and inability to write new data. Furthermore, this model relies on manual cleanup of old data periodically. However, the sheer number of unattended embedded terminals in the field results in extremely high manual maintenance costs and low efficiency, and problems such as untimely or missed cleanup can occur, seriously affecting the long-term stable operation of embedded devices. Summary of the Invention

[0003] The purpose of this invention is to provide an embedded circular time series data storage method based on key-value retrieval to address the shortcomings in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: an embedded circular time series data storage method based on key-value retrieval, comprising the following steps: Step 1: Determine the global storage space, which consists of at least two local storage spaces. Multiple local storage spaces are arranged in a closed loop according to the beginning and end logic. Within a single local storage space, multiple key-value pairs are arranged with their beginning and end addresses consecutively. Step 2: Starting from the second local storage space, traverse the local storage space, search for free local storage spaces with null key-value pairs in the header, and determine the current control local storage space; Step 3: Traverse the first key-value pair header address in the current control local storage space and determine the write control address based on preset conditions; Step 4: When writing key-value pairs at the write control address, if there is insufficient remaining space in the current local storage space, erase the next local storage space and switch the write control address to the header address of the next local storage space; if the current local storage space is just full, update the global read / write control flag; if there is sufficient remaining space, write the key-value pairs directly. Step 5: Using the global storage starting address as the initial read control address, the write address retrieval logic is reused to traverse valid key-value pairs. Time-series data is filtered and serialized by key matching predicates. After reading, the read control address is updated. If all local storage spaces have not been read, the read control address is jumped to the next local storage space header address.

[0005] In a preferred embodiment, step one includes: Key-value pairs are composed of key data and value data. The key data serves as a unique data identifier for key-value retrieval, while the value data is stored in business time-series logs. Local storage space is the smallest erase unit of flash memory, and multiple local storage spaces are arranged in a closed loop according to the first and last logical sequence. In a single local storage space, multiple key-value pairs are arranged with their first and last addresses consecutively. If the remaining space at the end of the local storage space cannot hold a null key-value pair, then the corresponding key-value pair is the last data in that local storage space.

[0006] In a preferred embodiment, step two includes: Starting from the second local storage space, traverse and search for free local storage spaces where there are null key-value pairs in the header; If a free local storage space is found, its previous local storage space is used as the current control local storage space; if there is still no free space after traversing to the first local storage space, the global storage space is erased and reset, and the first local storage space is set as the current control local storage space.

[0007] In a preferred embodiment, step three includes: Traverse the first key-value pair header address in the current controlled local storage space and determine the write control address based on preset conditions; The preset conditions are as follows: if there is not enough space after the end address of the current key-value pair to store an empty key-value pair, then it is determined whether there is an empty key-value pair at the beginning of the next local storage space. If there is, then the address is determined to be a valid write control address. If not, the control address is iterated to the beginning address of the next local storage space for loop searching. If there is enough space remaining after the current key-value pair, the control address is iterated to the end address of the current key-value pair and the traversal continues until the write control address is determined.

[0008] In a preferred embodiment, step four includes: A response point is configured for each global storage space, and each response point is connected to every local storage space in the global storage space. When writing key-value pairs to the write control address, the system first checks whether the remaining space in the current local storage space is insufficient. If it is insufficient, the system controls the erasure of the next local storage space through the response point and switches the write control address to the header address of the next local storage space. If the current local storage space is just full, update the global read / write control flag; if there is enough remaining space, write the key-value pairs directly.

[0009] In a preferred embodiment, the step of configuring response points for corresponding global storage spaces, wherein each response point is connected to every local storage space within the global storage space, is as follows: Multiple linkage points are configured for the global storage space. The number of linkage points is the same as the number of local storage spaces, and there is a one-to-one correspondence between the linkage points and the local storage spaces. Multiple linkage points are connected according to the logical closed-loop connection relationship of the beginning and end of the local storage space. Each linkage point stores multiple connection factors to obtain the response point.

[0010] In a preferred embodiment, the step of first determining whether the remaining space of the current local storage space is insufficient when writing key-value pairs at the write control address, and if insufficient, controlling the erasure of the next local storage space through the response point and switching the write control address to the header address of the next local storage space: The remaining space of the local storage space is recorded in the corresponding linkage point. When writing key-value pairs to the control address, the key-value pairs are loaded through the linkage point corresponding to the current local storage space. If the amount of data written to the key-value pairs is greater than the remaining space, the data will be transmitted to the linkage point of the next connection through the connection factor. Upon receiving the connection factor, the next linked point immediately triggers an erase command for the next local storage space, completing the data erasure process. Simultaneously, the key-value pair is loaded into the connection factor and transmitted to the next local storage space through the linkage point corresponding to the current local storage space. The write control address is switched to the header address of the next local storage space through the linkage point, and the key-value pair is accessed based on the linkage point corresponding to the next local storage space.

[0011] In a preferred embodiment, step five includes: Initialize the read control address to the global memory space header address, and reuse the write control address retrieval logic to traverse valid key-value pairs; Based on key data matching, a custom predicate logic is used to filter time-series data within a specified time range and complete data serialization. After reading, the read control address is updated. If the cumulative number of unread local storage spaces equals the total number of global local storage spaces, the read control address is jumped to the header address of the next local storage space after the current local storage space.

[0012] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention uses a circular storage space for local storage, which can automatically reset the earliest data entered into the database. When querying data that occurred within a specified time period, the data can be serialized according to the specified predicate logic, the oldest data can be erased, the local storage space can be used in a balanced manner, and the wear of flash memory can be evened out. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Example 1, please refer to Figure 1 As shown in this embodiment, the embedded circular time series data storage method based on key-value retrieval includes the following steps: Step 1: Determine the global storage space, which consists of at least two local storage spaces. Multiple local storage spaces are arranged in a closed loop according to the beginning and end logic. Within a single local storage space, multiple key-value pairs are arranged with their beginning and end addresses consecutively. Step 2: Starting from the second local storage space, traverse the local storage space, search for free local storage spaces with null key-value pairs in the header, and determine the current control local storage space; Step 3: Traverse the first key-value pair header address in the current control local storage space and determine the write control address based on preset conditions; Step 4: When writing key-value pairs at the write control address, if there is insufficient remaining space in the current local storage space, erase the next local storage space and switch the write control address to the header address of the next local storage space; if the current local storage space is just full, update the global read / write control flag; if there is sufficient remaining space, write the key-value pairs directly. Step 5: Using the global storage starting address as the initial read control address, the write address retrieval logic is reused to traverse valid key-value pairs. Time-series data is filtered and serialized by key matching predicates. After reading, the read control address is updated. If all local storage spaces have not been read, the read control address is jumped to the next local storage space header address.

[0017] As described in the steps above, circular storage automatically resets the earliest data stored; it meets the business needs of data storage in embedded microcontroller scenarios where storage and computing resources are limited; it allows querying data occurring within a specified time period and serializing the data according to specified predicate logic; it can level out flash memory wear; and it has good cross-platform compatibility. It can be expanded into a time-series database.

[0018] In one embodiment, step one includes: Key-value pairs are composed of key data and value data. The key data serves as a unique data identifier for key-value retrieval, while the value data is stored in the business time-series log (the key data is a 16-bit enumeration type, the value data is a variable-length binary data stream, and the value data embeds a timestamp field to achieve time series filtering). Local storage space is the smallest erase unit of flash memory. Multiple local storage spaces are arranged in a logical closed loop. The block before the first local storage space is the last local storage space, and the block after the last local storage space is the first local storage space. (The global storage space is a 1024K-byte flash memory partition, and a single local storage space is a 4K-byte flash local storage space (sector). The storage space uses flash logical address addressing. Flash memory only supports erasing the entire local storage space and does not support overwriting single key-value pairs.) In a single local storage space, multiple key-value pairs are arranged with their beginning and end addresses consecutively. If the remaining space at the end of the local storage space is insufficient to hold a null key-value pair (a null key-value pair is a fixed-length placeholder data block used to mark the free position of the local storage space and verify the remaining storage capacity of the local storage space), then the corresponding key-value pair is the last data in that local storage space (key-value pairs are arranged consecutively in the local storage space: the end address of the nth key-value pair = the beginning address of the (n+1)th key-value pair. Valid key-value pairs are written sequentially from the beginning of the local storage space, occupying a segment of address space after each pair is written; when a valid key-value pair is written, if the remaining free space from the end of that data to the end of the local storage space is insufficient to hold a null key-value pair, then this just-written valid key-value pair is defined as the last data in that local storage space).

[0019] In one embodiment, step two includes: Starting from the second local storage space, traverse and search for free local storage spaces where there are null key-value pairs in the header; If a free local storage space is found, its previous local storage space is used as the current control local storage space; if there is still no free space after traversing to the first local storage space, the global storage space is erased and reset, and the first local storage space is set as the current control local storage space (the step of erasing the next local storage space achieves flash wear leveling, evenly distributes the erase and write frequency of each flash local storage space, and avoids frequent wear of a single local storage space. In addition, when the global storage space is full, the local storage space corresponding to the earliest stored timing data is automatically erased to achieve first-in-first-out circular overlay and automatically cleans up expired timing logs).

[0020] In one embodiment, step three includes: Traverse the first key-value pair header address in the current controlled local storage space and determine the write control address based on preset conditions; The preset conditions are as follows: if there is not enough space after the end address of the current key-value pair to store an empty key-value pair, then it is determined whether there is an empty key-value pair at the beginning of the next local storage space. If there is, then the address is determined to be a valid write control address. If not, the control address is iterated to the beginning address of the next local storage space for loop searching. If there is enough space remaining after the current key-value pair, the control address is iterated to the end address of the current key-value pair and the traversal continues until the write control address is determined.

[0021] In one embodiment, step four includes: A response point is configured for each global storage space, and each response point is connected to every local storage space in the global storage space. When writing key-value pairs to the write control address, the system first checks whether the remaining space in the current local storage space is insufficient. If it is insufficient, the system controls the erasure of the next local storage space through the response point and switches the write control address to the header address of the next local storage space. If the current local storage space is just full, update the global read / write control flag; if there is enough remaining space, write the key-value pairs directly.

[0022] In one embodiment, the step of configuring response points for corresponding global storage spaces, where each response point is connected to every local storage space within the global storage space, is as follows: Multiple linkage points are configured for the global storage space. The number of linkage points is the same as the number of local storage spaces, and there is a one-to-one correspondence between the linkage points and the local storage spaces. Multiple linkage points are connected according to the logical closed-loop connection relationship of the beginning and end of the local storage space. Each linkage point stores multiple connection factors to obtain the response point.

[0023] In one embodiment, the step of first determining whether the remaining space of the current local storage space is insufficient when writing key-value pairs at the write control address, and if insufficient, controlling the erasure of the next local storage space through the response point and switching the write control address to the header address of the next local storage space: The remaining space of the local storage space is recorded in the corresponding linkage point. When writing key-value pairs to the control address, the key-value pairs are loaded through the linkage point corresponding to the current local storage space. If the amount of data written to the key-value pairs is greater than the remaining space, the data will be transmitted to the linkage point of the next connection through the connection factor. Upon receiving the connection factor, the next linked point immediately triggers an erase command for the next local storage space, completing the data erasure process. Simultaneously, the key-value pair is loaded into the connection factor and transmitted to the next local storage space through the linkage point corresponding to the current local storage space. The write control address is switched to the header address of the next local storage space through the linkage point, and the key-value pair is accessed based on the linkage point corresponding to the next local storage space.

[0024] As described above, within the data storage space, multiple linkage points are configured corresponding to the global storage space. These linkage points are virtual machines, and each linkage point internally stores multiple connection factors, which are also virtual machines. Each connection factor has a preset amount of storage space. Multiple linkage points are connected according to the logical closed-loop connection relationship of the beginning and end of the local storage space to obtain response points. Writes are all loaded through these response points to avoid data erroneous erasure due to inaccurate changes during direct writes. One linkage point corresponds to one local storage space. To better arrange the writing of key-value pairs, the remaining space in the local storage space is recorded in the corresponding linkage point. When writing a key-value pair at the write control address, the key-value pair is loaded through the linkage point corresponding to the current local storage space. If the amount of data to be written for the key-value pair is greater than the remaining space, a direct comparison is made in the linkage point, without the need for a "trial and error" comparison of the key-value pair to be written in the local storage space. This ensures the stability of the data stored in the global storage space. The data is then transmitted to the next linkage point via the connection factors through the linkage point. Here, the connection factor is only used to trigger the data erasure of the next connection's linkage point. The connection factor without data represents data erasure. When the next connection's linkage point receives the connection factor, it immediately triggers the erasure command for the next local storage space, completing the data erasure. At the same time, the key-value pair is loaded into the connection factor and transmitted to the next local storage space through the linkage point corresponding to the current local storage space. The data storage capacity of the connection factor is limited. The operation of loading the key-value pair into the connection factor is as follows: multiple connection factors are activated to connect based on the data volume of the key-value pair, which together serve as the carrier of the key-value pair, forming a connection factor. The connection factor storing the key-value pair is transmitted to the linkage of the next connection. The write control address is switched to the header address of the next local storage space through the linkage point. Based on the linkage point corresponding to the next local storage space, the key-value pair is written into the local storage space. The entire key-value pair input process is data-scheduled through the linkage point. It does not need to directly face the global storage space during scheduling, which can ensure the stability of key-value pair writing to local storage space and make data writing scheduling more flexible and accurate.

[0025] In one embodiment, step five includes: Initialize the read control address to the global memory space header address, and reuse the write control address retrieval logic to traverse valid key-value pairs; Based on key data matching, a custom predicate logic is used to filter time-series data within a specified time range and complete data serialization (the predicate logic includes time interval filtering, key identifier matching, and device event type filtering; after filtering, binary serialization parsing is performed on the matched value data to output business logs). After reading, the read control address is updated; if the cumulative number of unread local storage spaces is equal to the total number of global local storage spaces, the read control address is jumped to the header address of the next local storage space under the current local storage space.

[0026] As described in step five above, the local storage space here is stored in a ring, which can automatically reset the data that was first entered into the database; it meets the business needs of data storage in embedded microcontroller scenarios where storage and computing resources are limited; when querying data that occurred within a specified time period, the data can be serialized according to the specified predicate logic; it can level out the wear of flash memory; and it has good cross-platform compatibility.

[0027] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An embedded circular time series data storage method based on key-value retrieval, characterized in that, Includes the following steps: Step 1: Determine the global storage space, which consists of at least two local storage spaces. Multiple local storage spaces are arranged in a closed loop according to the beginning and end logic. Within a single local storage space, multiple key-value pairs are arranged with their beginning and end addresses consecutively. Step 2: Starting from the second local storage space, traverse the local storage space, search for free local storage spaces with null key-value pairs in the header, and determine the current control local storage space; Step 3: Traverse the first key-value pair header address in the current control local storage space and determine the write control address based on preset conditions; Step 4: When writing key-value pairs at the write control address, if there is insufficient remaining space in the current local storage space, erase the next local storage space and switch the write control address to the header address of the next local storage space; if the current local storage space is just full, update the global read / write control flag; if there is sufficient remaining space, write the key-value pairs directly. Step 5: Using the global storage starting address as the initial read control address, the write address retrieval logic is reused to traverse valid key-value pairs. Time-series data is filtered and serialized by key matching predicates. After reading, the read control address is updated. If all local storage spaces have not been read, the read control address is jumped to the next local storage space header address.

2. The embedded circular time series data storage method based on key-value retrieval according to claim 1, characterized in that, Step one includes: Key-value pairs are composed of key data and value data. The key data serves as a unique data identifier for key-value retrieval, while the value data is stored in business time-series logs. Local storage space is the smallest erase unit of flash memory, and multiple local storage spaces are arranged in a closed loop according to the first and last logical sequence. In a single local storage space, multiple key-value pairs are arranged with their first and last addresses consecutively. If the remaining space at the end of the local storage space cannot hold a null key-value pair, then the corresponding key-value pair is the last data in that local storage space.

3. The embedded circular time series data storage method based on key-value retrieval according to claim 1, characterized in that, Step two includes: Starting from the second local storage space, traverse and search for free local storage spaces where there are null key-value pairs in the header; If a free local storage space is found, its previous local storage space is used as the current control local storage space; if there is still no free space after traversing to the first local storage space, the global storage space is erased and reset, and the first local storage space is set as the current control local storage space.

4. The embedded circular time series data storage method based on key-value retrieval according to claim 1, characterized in that, Step three includes: Traverse the first key-value pair header address in the current controlled local storage space and determine the write control address based on preset conditions; The preset conditions are as follows: if there is not enough space after the end address of the current key-value pair to store an empty key-value pair, then it is determined whether there is an empty key-value pair at the beginning of the next local storage space. If there is, then the address is determined to be a valid write control address. If not, the control address is iterated to the beginning address of the next local storage space for loop searching. If there is enough space remaining after the current key-value pair, the control address is iterated to the end address of the current key-value pair and the traversal continues until the write control address is determined.

5. The embedded circular time series data storage method based on key-value retrieval according to claim 1, characterized in that, Step four includes: A response point is configured for each global storage space, and each response point is connected to every local storage space in the global storage space. When writing key-value pairs to the write control address, the system first checks whether the remaining space in the current local storage space is insufficient. If it is insufficient, the system controls the erasure of the next local storage space through the response point and switches the write control address to the header address of the next local storage space. If the current local storage space is just full, update the global read / write control flag; if there is enough remaining space, write the key-value pairs directly.

6. The embedded circular time series data storage method based on key-value retrieval according to claim 5, characterized in that, The step of configuring response points in the corresponding global storage space, where each response point is connected to every local storage space in the global storage space: Multiple linkage points are configured for the global storage space. The number of linkage points is the same as the number of local storage spaces, and there is a one-to-one correspondence between the linkage points and the local storage spaces. Multiple linkage points are connected according to the logical closed-loop connection relationship of the beginning and end of the local storage space. Each linkage point stores multiple connection factors to obtain the response point.

7. The embedded circular time series data storage method based on key-value retrieval according to claim 6, characterized in that, The steps described above are as follows: When writing key-value pairs to the write control address, firstly, the response point is used to determine whether the remaining space in the current local storage space is insufficient. If it is insufficient, the response point is used to control the erasure of the next local storage space and switch the write control address to the header address of the next local storage space. The remaining space of the local storage space is recorded in the corresponding linkage point. When writing key-value pairs to the control address, the key-value pairs are loaded through the linkage point corresponding to the current local storage space. If the amount of data written to the key-value pairs is greater than the remaining space, the data will be transmitted to the linkage point of the next connection through the connection factor. Upon receiving the connection factor, the next linked point immediately triggers an erase command for the next local storage space, completing the data erasure process. Simultaneously, the key-value pair is loaded into the connection factor and transmitted to the next local storage space through the linkage point corresponding to the current local storage space. The write control address is switched to the header address of the next local storage space through the linkage point, and the key-value pair is accessed based on the linkage point corresponding to the next local storage space.

8. The embedded circular time series data storage method based on key-value retrieval according to claim 1, characterized in that, Step five includes: Initialize the read control address to the global memory space header address, and reuse the write control address retrieval logic to traverse valid key-value pairs; Based on key data matching, a custom predicate logic is used to filter time-series data within a specified time range and complete data serialization. After reading, the read control address is updated. If the cumulative number of unread local storage spaces equals the total number of global local storage spaces, the read control address is jumped to the header address of the next local storage space after the current local storage space.

Images