Device label information hierarchical storage method and device based on prefix tree structure

Abstract

The invention relates to the field of data management, and provides an equipment label information hierarchical storage method and device based on a prefix tree structure, the method comprises the following steps: caching equipment label information into a memory in the prefix tree structure, the prefix tree structure being a data structure combining a prefix tree and a linked list, each cached node holds a pointer pointing to a child node storage space on a disk; when the amount of data cached in the memory reaches a preset threshold value, newly adding or modifying the equipment label information cached in the prefix tree into a physical log file; after the newly-added or modified operation is completely written into the physical log file, the equipment label information in the memory is stored in the prefix tree file, and the prefix tree file is stored on a disk by adopting a structure of fusing a prefix tree with a B + tree. According to the invention, the problem of overlarge memory consumption of a full-memory storage scheme in the prior art is solved, and the memory occupation is obviously reduced.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a method and apparatus for hierarchical storage of device tag information based on a prefix tree structure. Background Technology

[0002] With the rapid development of IoT technology, the time-series data generated by massive numbers of devices is experiencing explosive growth. Device identifiers are typically composed of hierarchical tags, which are used for efficient retrieval and aggregation operations in time-series databases. However, in scenarios with tens of millions or even hundreds of millions of devices, traditional identifier management methods face problems such as excessive memory resource consumption and low disk storage efficiency, severely restricting the scalability and query performance of time-series databases.

[0003] Currently, mainstream time-series databases typically manage device identifiers using two methods: full in-memory storage and a combination of internal and external storage with indexes. Full in-memory storage loads all identifier tags into memory, enabling millisecond-level queries through hash tables or inverted indexes. However, as the number of devices grows, memory usage far exceeds hardware limits, hindering system scalability. The combination of internal and external storage with indexes stores some identifiers on disk, but current methods involve creating a separate index for each tag. For example, the tag "country-city-device" is split into independent nodes, leading to duplicate storage of tags with the same prefix, wasting disk space and causing redundant I / O during queries.

[0004] Therefore, to meet the need for efficient management of massive device identifiers in IoT scenarios, there is an urgent need for a persistence method that can achieve fast hierarchical retrieval while ensuring low storage overhead. Summary of the Invention

[0005] This invention provides a hierarchical storage method and apparatus for device tag information based on a prefix tree structure, which solves the problem of excessive memory consumption in existing full-memory storage schemes and significantly reduces memory usage.

[0006] This invention provides a hierarchical storage method for device tag information based on a prefix tree structure, comprising the following steps: Device tag information is cached in memory in a prefix tree structure, which is a data structure combining a prefix tree and a linked list. Each cached node holds a pointer to the storage space of its child nodes on the disk. When the amount of data cached in the memory reaches a preset threshold, the addition or modification operation of the device tag information cached in the prefix tree is stored in the physical log file; After the addition or modification operation is completely written to the physical log file, the device tag information in memory is then stored in the prefix tree file, which is stored on the disk using a prefix tree fused with a B+ tree structure.

[0007] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided. Before caching the device tag information in memory in a prefix tree structure, the method further includes: storing the device tag information in a logical log file; the logical log file is used to restore device tag information not stored on disk when the system restarts, and to clear the logical log file after the device tag information is completely written into the prefix tree file.

[0008] According to the hierarchical storage method for device tag information based on a prefix tree structure provided by the present invention, after the device tag information in memory is completely written into the prefix tree file, the method further includes: deleting the device tag information with the least recent use in memory as needed according to the least recently used algorithm mechanism.

[0009] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided. The storage structure of the prefix tree file includes pages, segments, and records, wherein: the page is a basic disk I / O unit, including a page header, a segment, and free space; the segment is used to manage sibling nodes with a common parent node, and multiple records are stored in a slotted structure; the record stores the name of the prefix tree node and a pointer to the child node segment.

[0010] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided, wherein the sibling nodes are stored in the form of a B+ tree.

[0011] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided. The step of storing the device tag information in memory into a prefix tree file specifically includes: performing a pre-root traversal on the prefix tree cache in memory, sequentially accessing the root node and its child nodes; pre-allocating disk storage space for the child nodes of the current traversed node, and generating a pointer to the storage space; serializing the tag name, attribute information, and child node pointers of the current node into a record format; and writing the serialized node record to the disk in the prefix tree file format, storing it in the corresponding page and segment.

[0012] The present invention also provides a hierarchical storage device for device tag information based on a prefix tree structure, comprising the following modules: The memory caching module is used to cache device tag information in memory in a prefix tree structure. The prefix tree structure is a data structure that combines a prefix tree and a linked list. Each cached node holds a pointer to the storage space of its child nodes on the disk. The physical log storage module is used to store the addition or modification operation of the device tag information cached in the prefix tree to the physical log file when the amount of data cached in the memory reaches a preset threshold. The disk persistence module is used to store the device tag information in memory into a prefix tree file after the addition or modification operation is completely written to the physical log file. The prefix tree file is stored on the disk and uses a prefix tree fused with a B+ tree structure.

[0013] According to the present invention, a hierarchical storage device for device tag information based on a prefix tree structure is provided, the device further comprising a logical log storage module for: Before caching the device tag information into memory in a prefix tree structure, the device tag information is stored in a logical log file. The logical log file is used to restore the device tag information that has not been stored on disk when the system restarts, and to clear the logical log file after the device tag information has been completely written into the prefix tree file.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the hierarchical storage method for device tag information based on a prefix tree structure as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hierarchical storage method for device tag information based on a prefix tree structure as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the hierarchical storage method for device tag information based on a prefix tree structure as described above.

[0017] This invention provides a hierarchical storage method and apparatus for device tag information based on a prefix tree structure, which has the following advantages: By using a composite data structure combining prefix trees and linked lists to cache device tag information, and having each memory node hold a pointer to a disk child node, the storage redundancy problem in traditional hierarchical tag management is effectively solved—the prefix tree structure in memory naturally achieves hierarchical tag compression, avoiding duplicate storage of the same prefix, while the linked list structure optimizes node access efficiency; furthermore, by setting a cache threshold to trigger a two-level storage mechanism of physical log recording and disk persistence, data security is ensured, and memory usage pressure is significantly reduced by writing to disk on demand; finally, the fusion storage structure of prefix trees and B+ trees on disk enables the system to simultaneously possess the efficient hierarchical retrieval capability of prefix trees and the batch scanning advantage of B+ trees, thereby reducing memory consumption and improving query response time compared to traditional full-memory or flat index schemes in scenarios with massive numbers of devices. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the hierarchical storage method for device tag information based on a prefix tree structure provided by the present invention.

[0020] Figure 2 This is a flowchart of the hierarchical prefix tree persistence method provided by the present invention.

[0021] Figure 3 This is the physical structure diagram of the prefix tree file provided by the present invention.

[0022] Figure 4 This is the application form of the B+ tree in the prefix tree file provided by this invention.

[0023] Figure 5 This is a schematic diagram of the hierarchical storage device for device tag information based on a prefix tree structure provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] In IoT scenarios, real-time data generated by devices is widely used to monitor device operating status and analyze changing patterns. These time-series data, organized chronologically, are called time series. The identifier for a time series consists of a set of labels describing the device and sensor names; these labels are called identifier tags and are used for time series identifier retrieval and data aggregation during time series querying and analysis. Identifier tags often exhibit a clear hierarchical structure. With the continuous development of IoT technology, the number of IoT devices is increasing daily, and the number of time series data is growing accordingly, often reaching tens or even hundreds of millions. This necessitates the design of efficient disk persistence methods to support the needs of device identifier management.

[0027] The efficiency of tag management has a significant impact on the data query performance of time-series databases. Therefore, existing time-series database systems often load all tags into memory to ensure high query performance. With the increasing number of IoT devices, in massive time-series scenarios, the storage space occupied by tag information often exceeds the system's memory limit. Therefore, mainstream time-series databases are gradually shifting from a pure memory approach to a combined internal and external storage approach for tag management.

[0028] However, existing tag-based indexing management schemes fail to fully utilize the hierarchical nature of tags, thus failing to leverage the prefix sharing and hierarchical indexing features inherent in hierarchical relationships, leading to disk usage and query performance issues. For example, treating each tag as independent and creating an index for each tag and its associated time-series tag results in each hierarchical tag requiring independent indexing of its corresponding time-series tag, exhibiting problems such as duplicate storage, redundant indexing, and inefficient reading. Therefore, proposing a hierarchical prefix tree persistence method can significantly increase the upper limit of the number of devices that a time-series database can manage under the same hardware resources, which is an important prerequisite for supporting the management of massive IoT devices.

[0029] This hierarchical prefix tree persistence method primarily addresses the time-series data management of IoT devices. Designed to meet the needs of managing massive numbers of devices, it satisfies the write and query performance requirements of time-series databases when managing large numbers of devices. The method includes: In memory, identifiers are cached using a composite structure of prefix trees and linked lists. Each cached node holds a pointer to its child node's storage space on disk, supporting fast subtree access. While ensuring prefix compression and hierarchical indexing, it also supports on-demand loading of associated tags from disk and eviction of temporarily unused tags at runtime. On disk, device identifiers are persistently stored using a prefix tree fused with a B+ tree. Each prefix tree node holds a pointer to its child node's storage space, achieving decoupling of management methods between levels. Sibling nodes are stored in the form of a B+ tree, supporting efficient single-point queries and batch scans. Logical and physical logs are introduced to ensure the recoverability and integrity of the prefix tree cache in memory and the prefix tree file on disk, respectively.

[0030] This invention relates to a hierarchical prefix tree persistence method. First, device tag information is persisted to a logical log file via append-only writing to prevent information loss during system restarts. Next, the device tag information is written to a prefix tree cache in memory. The tag information is cached using a composite structure of prefix trees and linked lists. Each cached node holds a pointer to the storage space of its child nodes on disk to support fast subtree access. When the amount of buffered data reaches a certain threshold, a disk write operation is triggered, persisting the addition or modification of the cached device tag information in the physical log file. Finally, the information is persisted to a prefix tree file and stored on disk in a query-friendly prefix tree format. Each prefix tree node has a corresponding record stored on disk, and the hierarchical relationship between nodes is maintained through pointers within the storage file. This structure achieves prefix compression of device identifiers on disk, ensuring that information read during each disk I / O operation can be used for branch retrieval.

[0031] In IoT applications, device identifiers typically present a hierarchical tag structure, and the efficient management of these devices directly determines the database query efficiency. Current time-series databases mostly employ a design strategy of storing identifiers entirely in memory to maintain millisecond-level response capabilities. However, when faced with the explosive growth of smart terminals resulting in hundreds of millions of devices, this memory-based storage architecture faces bottlenecks such as excessive hardware resource consumption and limited system scalability. This invention designs a massive device identifier management scheme combining internal and external memory, proposing a low-overhead storage structure that integrates prefix trees and B+ trees. It supports runtime on-demand, hierarchical loading of associated identifier tags and the eviction of temporarily unused identifier tags from memory. The proposed hierarchical prefix tree persistence method is suitable for massive device identifier management.

[0032] The following is combined Figures 1-6 The embodiments of the present invention are described in detail.

[0033] Figure 1 This is a flowchart illustrating the hierarchical storage method for device tag information based on a prefix tree structure provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: S110. Cache the device tag information in memory using a prefix tree structure. The prefix tree structure is a data structure combining a prefix tree and a linked list, where each cached node holds a pointer to the storage space of its child nodes on disk.

[0034] S120. When the amount of data cached in memory reaches a preset threshold, the addition or modification operation of the device tag information cached in the prefix tree is stored in the physical log file.

[0035] S130. After writing the complete addition or modification operation to the physical log file, the device tag information in memory is then stored in the prefix tree file. The prefix tree file is stored on disk, using a prefix tree fused with a B+ tree structure.

[0036] Specifically, the present invention includes four main steps, such as Figure 2 As shown: Step 1: Persist the device tag information to the logical log file by appending it.

[0037] According to the hierarchical storage method of device tag information based on prefix tree structure provided by the present invention, the device tag information is stored in a logical log file before being cached in memory in the form of a prefix tree structure; the logical log file is used to restore the device tag information not stored on disk when the system restarts, and to clear the logical log file after the device tag information is completely written into the prefix tree file.

[0038] Specifically, device tag information is initially buffered in memory until a certain threshold is reached, at which point a disk write operation is triggered. This design fully leverages the high-speed read / write capabilities of memory to achieve rapid data access and query response. The memory buffering mechanism efficiently integrates device tag data generated over a period of time, forming a logically coherent and structured data set, preparing for subsequent data persistence operations.

[0039] However, as a volatile storage medium, memory is at risk of data loss during system restarts or failures. To effectively address this challenge and ensure data security and integrity, this invention introduces a logical log mechanism. As a temporary persistent storage method, the logical log records changes to device tag information in real-time or periodically while the information is buffered in memory. In this way, even during a system restart, device tag information not yet persisted to disk can be recovered from memory based on the records in the logical log, thus preventing data loss.

[0040] The introduction of logical logs not only enhances the system's fault tolerance but also optimizes the data persistence process. Specifically, whenever all buffered device tag information in memory is successfully written to the prefix tree file, the corresponding record in the logical log is considered to have fulfilled its purpose and is automatically cleared by the system. This design ensures that only non-persistent data records are retained in the logical log, reducing log file redundancy and space consumption, while also simplifying log management complexity. Through the collaborative work of logical logs, memory buffers, and prefix tree file persistence, this invention achieves an efficient, secure, and reliable data management process, providing strong support for the application of IoT time-series databases in scenarios involving massive device tag management.

[0041] Step 2: Write the device tag information into the prefix tree cache in memory.

[0042] Specifically, to significantly improve the efficiency of prefix tree file construction and hotspot information retrieval, while ensuring the preservation of the prefix compression effect and high memory utilization of a pure in-memory prefix tree, this invention specifically designs and introduces a prefix tree caching mechanism. This mechanism serves a dual purpose: disk persistence caching and query caching. In practice, when the system receives device tag information, it carefully organizes this information into a prefix tree structure based on the hierarchical structure of the tag and caches it in memory. This design not only enables the buffered data to respond to query requests instantly, greatly improving query efficiency, but also lays a high-efficiency data foundation for subsequent disk persistence operations.

[0043] In terms of memory cache management, this invention employs an intelligent buffering strategy. When the amount of device tag information data buffered in memory accumulates to a preset threshold, the system automatically triggers a disk write operation to ensure data security and persistence. During the disk write process, the system uses a pre-root traversal approach to perform a depth-first traversal of the prefix tree cache in memory, sequentially accessing the root node and all its child nodes to ensure that the data of each node is accurately and completely written to the disk. This pre-root traversal disk write method not only guarantees data consistency and integrity but also allows data on the same disk page to be processed centrally, thereby effectively reducing the overhead of repeated read / write operations on the same disk page during disk write and improving disk I / O efficiency.

[0044] Furthermore, this mechanism possesses powerful data integration capabilities. It can efficiently integrate device tag information over a period of time in memory, forming a logically interconnected prefix tree structure. This integration not only simplifies subsequent data processing but also enables faster location of required data during queries, further improving query efficiency. Simultaneously, due to the presence of the prefix tree cache, the system can effectively manage memory resources without sacrificing query performance, ensuring high memory utilization and providing strong support for the management of massive numbers of IoT devices.

[0045] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided. After the device tag information in memory is completely written into the prefix tree file, the device tag information with the least recent use in memory is deleted as needed according to the least recently used algorithm mechanism.

[0046] Specifically, after the device tag information is written from memory to the prefix tree file and stored on disk, this data is not immediately and completely removed from memory. Instead, it is selectively stored in the memory cache to maintain a fast response capability for frequently accessed data. This design strategy fully leverages the fact that memory access is much faster than disk I / O. By retaining a portion of recently or frequently accessed data in memory, it significantly reduces the reliance on the disk in subsequent query operations, thereby improving overall query efficiency.

[0047] To efficiently manage this temporarily stored data in memory and avoid excessive memory resource consumption and waste, this invention introduces an LRU (Least Recently Used) cache eviction mechanism. The LRU mechanism is based on a core assumption: data that has been accessed least recently is unlikely to be accessed frequently in the future. Therefore, when memory space is insufficient or space needs to be freed up for new data, the system prioritizes evicting the least recently accessed data items. This mechanism ensures that only the data most likely to be accessed again is always retained in memory, thereby maximizing query performance with limited memory resources.

[0048] In practice, the system maintains an access timestamp or counter for each data item in memory, recording the time or frequency of its most recent access. When data needs to be evicted, the system selects and evicts the least recently accessed data items based on these timestamps or counters. This process is fully automated, requiring no manual intervention, ensuring both efficient memory utilization and system stability and responsiveness when dealing with massive amounts of data. By combining the LRU cache eviction mechanism with a memory retention strategy, this invention achieves fine-grained management of memory resources, providing robust performance support for IoT time-series databases when processing massive amounts of device identification information.

[0049] Step 3: When the amount of buffered data reaches a certain threshold, trigger the disk write operation to persist the addition or modification of device tag information cached in memory to the physical log file.

[0050] Specifically, a key technical challenge in writing buffered information to the prefix tree file lies in ensuring data integrity and consistency during complex write operations that may involve modifications to multiple disk pages. Specifically, when the system writes device tag information accumulated in memory to disk in batches, a single write operation may modify the contents of multiple disk pages simultaneously. This parallel modification of multiple pages increases the risk of data corruption, especially in unforeseen circumstances such as system crashes or restarts. This could lead to some disk pages being in an inconsistent or corrupted state, thus affecting the overall structure of the prefix tree file and subsequent data queries.

[0051] To effectively address this challenge and ensure the integrity and consistency of the prefix tree file at any given time, this invention innovatively introduces a physical log mechanism. As a log file that records data prior to actual data modifications, the core function of the physical log is to pre-record the specific content and scope of each disk write operation, including which disk pages will be modified and the expected data state after modification. In this way, the physical log provides the system with a baseline for "rollback" or "recovery," enabling the system to accurately identify and recover disk page data affected in the last operation before the crash, based on the records in the physical log, in the event of an anomaly such as a system crash or restart.

[0052] In practice, the system strictly adheres to the principle of "write logs first, then modify data." That is, before each disk write operation begins, the system first writes all relevant information about the operation completely to the physical log file. Only after confirming that the physical log has been successfully written and is error-free will the system proceed with the actual modification operations on the prefix tree file, including updating the corresponding disk page content. This process ensures that even in the most adverse scenarios of a crash or restart, a complete and consistent backup of the disk page data affected by the last operation before the crash can always be found on the disk, thus providing strong support for rapid system recovery and continuous data availability.

[0053] Step 4: After the addition or modification of device tag information is fully written to the physical log, it is then written to the prefix tree file.

[0054] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided, which stores device tag information in memory into a prefix tree file. Specifically, the method includes: performing a pre-root traversal on the prefix tree cache in memory, sequentially accessing the root node and its child nodes; pre-allocating disk storage space for the child nodes of the current traversed node and generating pointers to the storage space; serializing the tag name, attribute information, and child node pointers of the current node into a record format; and writing the serialized node record to the disk in the prefix tree file format and storing it in the corresponding page and segment.

[0055] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided. The storage structure of the prefix tree file includes pages, segments, and records, wherein: a page is a basic disk I / O unit, including a page header, a segment, and free space; a segment is used to manage sibling nodes with a common parent node, and multiple records are stored in a slotted structure; a record stores the name of the prefix tree node and a pointer to the child node segment.

[0056] Specifically, such as Figure 3As shown, the physical storage structure of the prefix tree file on disk is carefully designed to ensure efficient data access and management. Its main physical units are divided into three levels: pages, segments, and records. In this hierarchical structure, each node of the prefix tree is mapped to an independent record on the disk. These records not only contain the key attribute information of the node, but also additionally store a pointer to the storage space of its child nodes to support the hierarchical indexing and traversal of the prefix tree structure.

[0057] According to the present invention, a hierarchical storage method for device tag information based on a prefix tree structure is provided, wherein sibling nodes are stored in the form of a B+ tree.

[0058] Furthermore, to optimize disk space utilization and improve data access efficiency, a group of sibling node records sharing a common parent node are organized together to form a segment. As the management unit for sibling node records in the prefix tree, the segment employs a slotted storage structure, enabling multiple child node records under the same parent node to be efficiently read in a single disk I / O operation. In addition, to handle potential space allocation issues arising from new node additions, the segment size is designed to increase incrementally in increments of 1 / 16, 1 / 8, 1 / 4, 1 / 2, and 1 page. This design avoids frequent space allocation and effectively prevents disk space waste. When the segment size reaches 1 page, subsequent segment allocations due to new sibling nodes will always allocate 1 page of space in the new page, and the new segment and the existing segment will be connected by pointers to form a B+ tree structure. This B+ tree structure further improves the performance of single-point queries and batch scans of child nodes at specific levels.

[0059] As the most basic disk I / O unit, the page plays a crucial role in the physical storage structure. A prefix tree file is physically divided into a file header and several fixed-length pages. The file header stores the root node information of the prefix tree for fast location and access. Each page consists of a page header, multiple segments, and free space. The page header stores segment identifiers and page address information for fast location and access of segments within the page; segment information is stored starting from the end of the page, organized in a slotted structure. This design allows the segment information within a page to be deserialized as needed after it is read into memory, thus improving the flexibility and efficiency of data access. Simultaneously, to meet the storage needs of new segments during sequence registration, the system dynamically allocates existing pages with sufficient internal free space for data storage based on segment size requirements; if existing pages lack sufficient internal free space, a new page is added at the end of the file to ensure continuous data storage and efficient access.

[0060] The following provides a detailed explanation of pages, paragraphs, and records.

[0061] Page: To align with the characteristics of disk I / O, pages are introduced as the basic unit of disk I / O. A prefix tree file is physically divided into a file header and several fixed-length pages. The file header stores the root node information of the prefix tree, and the pages store information about the remaining nodes. A page internally consists of a page header and multiple variable-length segments, using a slotted structure. Segment information is stored starting from the end of the page, the page header stores the segment identifier and its internal address, and the middle of the page is free space. This structure allows the segment information within a page to be deserialized as needed after it has been read into memory. When a new segment needs to be stored during sequence registration, existing pages with sufficient internal free space are allocated for data storage, depending on the size requirement. If existing pages do not have enough internal free space, a new page is allocated, i.e., a new page is added at the end of the file.

[0062] Segment: Because devices with the same identifier tag are easily retrieved simultaneously when querying device tag information, a segment structure is designed as a management unit for a group of sibling nodes with a common parent node in the prefix tree. A segment stores multiple records representing prefix tree nodes, also using a slotted structure. When a device is added, or a new sibling node is added to the prefix tree, space is allocated within a page first, ensuring that sibling nodes can be read simultaneously in a single disk I / O. To avoid frequent space allocation and prevent disk space waste, the segment size increases incrementally in the form of 1 / 16, 1 / 8, 1 / 4, 1 / 2, and 1 page. When the segment size reaches 1 page, subsequent segment allocations due to the addition of sibling nodes always allocate 1 page of space in a new page. The new segment and the existing segments form a B+ tree structure, with the key of the B+ tree being the name of the prefix tree node. For example... Figure 4 This demonstrates the application of B+ trees in prefix tree files. Using a B+ tree structure can simultaneously guarantee the performance of retrieving single child nodes and batch scanning of child nodes, thereby ensuring the performance of metadata queries in retrieving tag values ​​at the corresponding levels.

[0063] Records: Each node in the prefix tree corresponds to a record within a segment, containing attribute information such as the node's name. To better support hierarchical indexing of the prefix tree, each record additionally stores a pointer to the first segment managing the child node; this pointer represents the segment's storage address. The address is represented by a 64-bit long value, with the lower 59 bits representing the page containing the segment and the higher 5 bits representing the identifier of the target segment within the page, used for in-page retrieval. This method decouples the management methods between different sibling node groups, facilitating the implementation of different management methods as needed based on load. Since the prefix tree data is processed using a pre-order traversal method for disk persistence, each node, when persisted as a record, is pre-allocated a child node storage space and its corresponding address is stored in the record.

[0064] In summary, this invention solves the problem of excessive hardware resource consumption caused by traditional all-in-memory storage of tags through a storage scheme combining internal and external memory, significantly increasing the upper limit of the number of devices that a time-series database can manage under the same hardware resources. Utilizing the hierarchical nature of tag names, a composite structure of prefix trees and linked lists is used for memory caching. Combined with a low-overhead storage structure integrating prefix trees and B+ trees, efficient data storage and retrieval are achieved, significantly improving write and query performance. The introduction of logical and physical log mechanisms ensures the recoverability and integrity of the prefix tree cache in memory and the prefix tree file on disk, effectively preventing data corruption or loss due to system crashes or restarts. Through a pre-order traversal disk write method and a hierarchical storage structure of pages, segments, and records, device tag information within a certain period is integrated in memory, reducing the overhead of repeated read / write operations on the same disk page during disk writes and improving disk I / O efficiency. An LRU cache eviction mechanism is used to evict temporarily unused tags as needed, supporting dynamic loading of associated tag names at runtime, achieving efficient management and dynamic expansion of massive device tags.

[0065] The following describes the hierarchical storage device for device tag information based on a prefix tree structure provided by the present invention. The hierarchical storage device for device tag information based on a prefix tree structure described below and the hierarchical storage method for device tag information based on a prefix tree structure described above can be referred to in correspondence with each other.

[0066] like Figure 5 The diagram shows a hierarchical storage device for device tag information based on a prefix tree structure provided by the present invention, comprising: The memory cache module 510 is used to cache device tag information in memory in a prefix tree structure. The prefix tree structure is a data structure that combines a prefix tree and a linked list. Each cached node holds a pointer to the storage space of its child nodes on the disk. The physical log storage module 520 is used to store the addition or modification operation of the device tag information cached in the prefix tree to the physical log file when the amount of data cached in the memory reaches a preset threshold. The disk persistence module 530 is used to store the device tag information in memory into a prefix tree file after the addition or modification operation is completely written to the physical log file. The prefix tree file is stored on the disk and uses a prefix tree fused with a B+ tree structure.

[0067] According to the present invention, a hierarchical storage device for device tag information based on a prefix tree structure is provided. The device further includes a logical log storage module, which is used to: store the device tag information in a logical log file before caching the device tag information in memory in a prefix tree structure; the logical log file is used to restore the device tag information not stored on the disk when the system restarts, and to clear the logical log file after the device tag information is completely written to the prefix tree file.

[0068] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a hierarchical storage method for device tag information based on a prefix tree structure. This method includes: caching device tag information in memory using a prefix tree structure, where the prefix tree structure is a data structure combining a prefix tree and a linked list, with each cached node holding a pointer to the storage space of its child nodes on the disk; when the amount of data cached in memory reaches a preset threshold, storing the addition or modification operations of the device tag information cached in the prefix tree to a physical log file; after completely writing the addition or modification operations to the physical log file, storing the device tag information in memory to the prefix tree file, which is stored on the disk using a prefix tree fused with a B+ tree structure.

[0069] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the hierarchical storage method for device tag information based on a prefix tree structure provided by the above methods. The method includes: caching device tag information in memory in a prefix tree structure, wherein the prefix tree structure is a data structure combining a prefix tree and a linked list, and each cached node holds a pointer to the storage space of its child nodes on the disk; when the amount of data cached in memory reaches a preset threshold, storing the addition or modification operation of the device tag information cached in the prefix tree into a physical log file; after the addition or modification operation is completely written into the physical log file, storing the device tag information in memory into the prefix tree file, wherein the prefix tree file is stored on the disk, using a prefix tree fused with a B+ tree structure for storage.

[0071] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the hierarchical storage method for device tag information based on a prefix tree structure provided by the above methods. The method includes: caching device tag information in memory in a prefix tree structure, wherein the prefix tree structure is a data structure combining a prefix tree and a linked list, and each cached node holds a pointer to the storage space of its child nodes on the disk; when the amount of data cached in memory reaches a preset threshold, storing the addition or modification operation of the device tag information cached in the prefix tree into a physical log file; after the addition or modification operation is completely written into the physical log file, storing the device tag information in memory into the prefix tree file, wherein the prefix tree file is stored on the disk, using a prefix tree fused with a B+ tree structure for storage.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical storage method for device tag information based on a prefix tree structure, characterized in that, include: Device tag information is cached in memory in a prefix tree structure, which is a data structure combining a prefix tree and a linked list. Each cached node holds a pointer to the storage space of its child nodes on the disk. When the amount of data cached in the memory reaches a preset threshold, the addition or modification operation of the device tag information cached in the prefix tree is stored in the physical log file; After the addition or modification operation is completely written to the physical log file, the device tag information in memory is then stored in the prefix tree file, which is stored on the disk using a prefix tree fused with a B+ tree structure.

2. The hierarchical storage method for device tag information based on a prefix tree structure according to claim 1, characterized in that, Before caching the device tag information in memory as a prefix tree structure, the method further includes: Store device tag information in a logical log file; The logical log file is used to restore device tag information that has not been stored on disk when the system restarts, and to clear the logical log file after the device tag information has been completely written to the prefix tree file.

3. The hierarchical storage method for device tag information based on a prefix tree structure according to claim 1, characterized in that, After writing the complete device tag information in memory into the prefix tree file, the method further includes: Based on the Least Recently Used (LRU) algorithm, device tag information that is used least frequently in memory is deleted as needed.

4. The hierarchical storage method for device tag information based on a prefix tree structure according to claim 1, characterized in that, The storage structure of the prefix tree file includes pages, segments, and records, wherein: The page is a basic disk I / O unit, which includes a page header, a segment, and free space; The segment is used to manage sibling nodes that share a common parent node, and multiple records are stored using a slotted structure. The record stores the name of the prefix tree node and pointers to its child node segments.

5. The hierarchical storage method for device tag information based on a prefix tree structure according to claim 4, characterized in that, The sibling nodes are stored in the form of a B+ tree.

6. The hierarchical storage method for device tag information based on a prefix tree structure according to claim 4, characterized in that, The step of storing the device tag information in memory into a prefix tree file specifically includes: Perform a pre-root traversal on the prefix tree cache in memory, visiting the root node and its child nodes in turn; Pre-allocate disk storage space for the child nodes of the current traversed node, and generate a pointer to the storage space; Serialize the label name, attribute information, and child node pointers of the current node into a record format; The serialized node records are written to disk in a prefix tree file format and stored in the corresponding pages and segments.

7. A hierarchical storage device for device tag information based on a prefix tree structure, characterized in that, include: The memory caching module is used to cache device tag information in memory in a prefix tree structure. The prefix tree structure is a data structure that combines a prefix tree and a linked list. Each cached node holds a pointer to the storage space of its child nodes on the disk. The physical log storage module is used to store the addition or modification operation of the device tag information cached in the prefix tree to the physical log file when the amount of data cached in the memory reaches a preset threshold. The disk persistence module is used to store the device tag information in memory into a prefix tree file after the addition or modification operation is completely written to the physical log file. The prefix tree file is stored on the disk and uses a prefix tree fused with a B+ tree structure.

8. The hierarchical storage device for device tag information based on a prefix tree structure according to claim 7, characterized in that, The device further includes a logical log storage module for: Before caching the device tag information into memory in a prefix tree structure, the device tag information is stored in a logical log file. The logical log file is used to restore the device tag information that has not been stored on disk when the system restarts, and to clear the logical log file after the device tag information has been completely written into the prefix tree file.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the hierarchical storage method for device tag information based on a prefix tree structure as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hierarchical storage method for device tag information based on a prefix tree structure as described in any one of claims 1 to 6.

Images