Tree structure data storage method and device, terminal equipment and computer program product

Abstract

The invention relates to a tree structure data storage method and device, terminal equipment and a computer program product. The method comprises the steps of obtaining to-be-stored tree structure data; matching a first prefix of a first sub-tree in the tree structure data with the number of a preset memory, distributing the first sub-tree to the memory with the number corresponding to the first prefix, and rewriting the first sub-tree according to a preset period; after the rewrite operation of the current period of the first sub-tree is completed, if it is detected that the first sub-tree meets a preset capacity expansion rule, the first sub-tree is split into a plurality of second sub-trees in the rewrite operation of the next period of the first sub-tree, and each second sub-tree corresponds to a second prefix; and matching the second prefix of the second sub-tree with the serial number of the memory, and allocating the second sub-tree to the memory with the serial number corresponding to the second prefix. By means of the method and device, the problem that the read-write efficiency of the tree structure data is low due to the fact that a single memory is used for storing the tree structure data in the related technology is solved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a tree structure data storage method, apparatus, terminal equipment and computer program product. Background Technology

[0002] Tree data structures (also known as tree structures) are a fundamental data structure in computer science. They simulate the structure of trees in nature and are widely used in file systems, database indexes, compiler design, and other fields.

[0003] Current methods for storing tree structure data typically use a single memory to store the tree structure data. When the tree structure data is large, the storage space of a single memory is limited, which reduces the efficiency of reading and writing the stored tree structure data.

[0004] Currently, no effective solution has been proposed to address the issue of low read / write efficiency of tree structure data due to the use of a single memory to store tree structure data in related technologies. Summary of the Invention

[0005] This application provides a tree structure data storage method, apparatus, terminal device, and computer program product to at least solve the problem of low read / write efficiency of tree structure data caused by using a single memory to store tree structure data in related technologies.

[0006] In a first aspect, embodiments of this application provide a tree structure data storage method, comprising: acquiring tree structure data to be stored, wherein the tree structure data includes multiple first subtrees; matching a first prefix of the first subtree with a preset memory number, and allocating the first subtree to the memory corresponding to the number and the first prefix, wherein the first subtree is rewritten according to a preset period; after the rewrite operation of the first subtree in the current period is completed, if it is detected that the first subtree meets a preset expansion rule, then in the rewrite operation of the first subtree in the next period, splitting the first subtree into multiple second subtrees, wherein each second subtree corresponds to a second prefix; matching the second prefix of the second subtree with the memory number, and allocating the second subtree to the memory corresponding to the number and the second prefix.

[0007] In some embodiments, splitting the first subtree into multiple second subtrees includes: setting the first prefix of the first subtree to a first value, setting the same number of second values ​​according to the number of second subtrees, wherein each second value is unique; sequentially adding the second value after the first value to obtain multiple second prefixes; and writing the data of the first file corresponding to the first subtree into the second file corresponding to each second prefix to obtain the multiple second subtrees.

[0008] In some embodiments, the expansion rule includes at least one of the following: the size of the first file corresponding to the first subtree is greater than a preset first threshold; the remaining storage space of the memory corresponding to the first subtree is less than a preset second threshold.

[0009] In some embodiments, the second subtree is rewritten according to the preset period; after matching the second prefix of the second subtree with the number of the memory and allocating the second subtree to the memory corresponding to the number and the second prefix, the method further includes: for multiple second subtrees obtained by splitting the first subtree, after the rewrite operation of the second subtree in the current period is completed, if at least two second subtrees are detected to meet the preset shrinkage rule, then in the rewrite operation of the at least two second subtrees in the next period, the at least two second subtrees are merged into a third subtree, wherein the third subtree corresponds to a third prefix; the third prefix of the third subtree is matched with the number of the memory and the third subtree is allocated to the memory corresponding to the number and the third prefix.

[0010] In some embodiments, merging the at least two second subtrees into a third subtree includes: merging the second prefixes of the at least two second subtrees into a third prefix; writing the data of the at least two second subtrees into a third file corresponding to the third prefix in the order of the second prefixes to obtain a third subtree; recording the file offset of each second subtree, and recording the file offset of each second subtree in the root node of the third subtree.

[0011] In some embodiments, after matching the first prefix of the first subtree with the preset memory number and allocating the first subtree to the memory corresponding to the number and the first prefix, the method further includes: after the rewrite operation of the first subtree in the current cycle is completed, if it is detected that at least one memory among the plurality of memories satisfies a preset migration rule, then in the rewrite operation of the first subtree in the next cycle, the first subtree corresponding to the at least one memory is allocated to other memories among the plurality of memories according to a preset storage space and load balancing strategy.

[0012] In some embodiments, after matching the second prefix of the second subtree with the number of the memory and allocating the second subtree to the memory corresponding to the number and the second prefix, the method further includes: writing the data to be written to a pre-write log; and, based on a given key, using a preset prefix matching strategy, allocating the data to be written in the pre-write log to the first subtree or the second subtree; wherein the prefix matching strategy includes: when the given key matches both the first prefix and the second prefix, writing the data to be written in the pre-write log to the second subtree corresponding to the second prefix; and when the given key matches only the first prefix, writing the data to be written in the pre-write log to the first subtree corresponding to the first prefix.

[0013] Secondly, embodiments of this application provide a tree structure data storage device, comprising: an acquisition module, configured to acquire tree structure data to be stored, wherein the tree structure data includes multiple first subtrees; a first allocation module, configured to match a first prefix of the first subtree with a preset memory number, and allocate the first subtree to the memory corresponding to the number and the first prefix, wherein the first subtree is rewritten according to a preset period; a splitting module, configured to, after the rewrite operation of the first subtree in the current period is completed, if it is detected that the first subtree meets a preset expansion rule, split the first subtree into multiple second subtrees in the rewrite operation of the first subtree in the next period, wherein each second subtree corresponds to a second prefix; and a second allocation module, configured to match the second prefix of the second subtree with the memory number, and allocate the second subtree to the memory corresponding to the number and the second prefix.

[0014] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the tree structure data storage method of any of the first aspects described above.

[0015] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the tree structure data storage method described in any one of the first aspects to be executed.

[0016] Compared to related technologies, the tree structure data storage method, apparatus, terminal device, and computer program product provided in this application allocate the first subtree to the memory corresponding to the first prefix by matching the first prefix of the first subtree in the tree structure data with the memory number. After the rewrite operation of the first subtree in the current cycle is completed, if the first subtree is detected to meet the expansion rule, the first subtree is split into multiple second subtrees in the rewrite operation of the first subtree in the next cycle. The second prefix of the second subtree is matched with the memory number and the second subtree is allocated to the memory corresponding to the second prefix. In this way, by analyzing the prefix of the subtree, the original large tree structure data is logically split into multiple smaller subtrees. Each subtree can be stored independently in different memories, realizing data sharding and parallel storage, thereby breaking through the storage space limitation of a single memory. In addition, by detecting whether the first subtree meets the preset expansion rule, the first subtree that meets the expansion rule (i.e., the subtree with higher load) can be split a second time, thereby alleviating local hot spots, avoiding storage resource waste and performance bottlenecks, and improving the read and write efficiency of tree structure data. This application solves the problem of low read / write efficiency of tree structure data due to the use of a single memory to store tree structure data in related technologies, and achieves the technical effect of improving the read / write efficiency of tree structure data.

[0017] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a tree structure data storage method according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of a tree structure according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram illustrating the splitting and allocation of tree structure data according to one embodiment of this application;

[0022] Figure 4 This is a schematic diagram illustrating the splitting and allocation of the first subtree according to an embodiment of this application;

[0023] Figure 5This is a schematic diagram illustrating the merging and allocation of a second subtree according to one embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of a tree structure data storage device according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] Tree data structures (also known as tree structures) are a fundamental data structure in computer science. They simulate the structure of trees in nature and are widely used in file systems, database indexes, compiler design, and other fields.

[0033] Current methods for storing tree structure data typically use a single memory to store the tree structure data. When the tree structure data is large, the storage space of a single memory is limited, which reduces the efficiency of reading and writing the stored tree structure data.

[0034] Currently, no effective solution has been proposed to address the issue of low read / write efficiency of tree structure data due to the use of a single memory to store tree structure data in related technologies.

[0035] In view of this, embodiments of this application provide a tree structure data storage method. This method involves matching the first prefix of the first subtree in the tree structure data with the memory number, and allocating the first subtree to the memory corresponding to the first prefix. After the rewrite operation of the first subtree in the current cycle is completed, if the first subtree is detected to meet the expansion rule, then in the rewrite operation of the first subtree in the next cycle, the first subtree is split into multiple second subtrees. The second prefix of the second subtree is matched with the memory number, and the second subtree is allocated to the memory corresponding to the second prefix. In this way, by analyzing the prefix of the subtree, the original large tree structure data is logically split into multiple smaller subtrees. Each subtree can be independently stored in different memories, realizing data sharding and parallel storage, thereby breaking through the storage space limitation of a single memory. Furthermore, by detecting whether the first subtree meets the preset expansion rule, the first subtree that meets the expansion rule (i.e., the subtree with higher load) can be split a second time, thereby alleviating local hotspots, avoiding storage resource waste and performance bottlenecks, and thus improving the read and write efficiency of tree structure data. This application solves the problem of low read / write efficiency of tree structure data due to the use of a single memory to store tree structure data in related technologies, and achieves the technical effect of improving the read / write efficiency of tree structure data.

[0036] The following will combine Figure 1 This application describes a tree structure data storage method according to one embodiment. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart of a tree structure data storage method according to an embodiment of this application, such as... Figure 1 As shown, the method includes:

[0037] Step S101: Obtain the tree structure data to be stored, wherein the tree structure data includes multiple first subtrees.

[0038] In this embodiment, the tree structure can be a variant of a 16-way trie. Each branch of the tree expands layer by layer according to the hexadecimal characters of the keys of the leaf nodes, constructing a hierarchical structure similar to a trie.

[0039] Specifically, each intermediate node in the tree structure can contain 16 child node slots, corresponding to the hexadecimal characters 0 to F. The keys of the leaf nodes in the tree structure consist of hexadecimal characters, which are mapped to the corresponding child node slots layer by layer in character order.

[0040] As an example, you can refer to Figure 2 , Figure 2 This is a schematic diagram of a tree structure according to an embodiment of this application. For example... Figure 2 As shown, Figure 2 The tree structure shown contains 6 leaf nodes, with keys ABCD, ACDE, ACEF, BCDE, BCFE, and BDEF. The keys of the leaf nodes can be mapped to the corresponding child node slots layer by layer according to the character order of the keys.

[0041] Step S102: Match the first prefix of the first subtree with the preset memory number, and assign the first subtree to the memory corresponding to the first prefix. The first subtree is rewritten according to a preset period.

[0042] In this embodiment, the storage device can be a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Multiple disks can be prepared in advance and mounted on the server; for example, 16 disks can be prepared and mounted on the server, resulting in 16 storage devices numbered from 0 to F (hexadecimal representation).

[0043] Taking a memory bank of 16, numbered 0 to F, as an example, for tree-structured data, the key prefix (i.e., the first prefix) of the root node in the tree structure can be used as the basis for splitting and allocating the tree-structured data. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram illustrating the splitting and allocation of tree structure data according to one embodiment of this application, as shown below. Figure 3 As shown, the tree structure data can be split into 16 first subtrees with first prefixes from 0 to F. By analyzing the prefixes of the data keys, the original large tree structure data is logically split into multiple smaller first subtrees. Then, the first prefix of each first subtree can be matched with the memory number, and the first subtree can be assigned to the memory corresponding to its first prefix. For example, the first subtree with a first prefix of 0 is assigned to memory number 0, the first subtree with a first prefix of 1 is assigned to memory number 1, and so on. Each first subtree is stored as an independent first file, and the numbering rule of the first file can be set according to the first prefix of the first subtree to ensure its uniqueness in memory. For example, the first subtree with a first prefix of 0 can be stored in first file number 0 of memory number 0.

[0044] In this way, by storing each first subtree independently in different memories or memory nodes, data sharding and parallel storage can be achieved, thereby breaking through the storage space limitations of a single memory. This can improve the storage space utilization efficiency of the memory and improve the read and write efficiency of the tree structure data stored in the memory.

[0045] It should be noted that the above description only introduces the method for allocating tree structure data when there are 16 memory locations. The tree structure data storage method provided in this application embodiment can also be applied to fewer or more memory locations. For example, when there are 8 memory locations numbered 0 to 7, the first subtree with a first prefix of 0 and a first prefix of 8 can be allocated to the memory location numbered 0, and the first subtree with a first prefix of 1 and a first prefix of 9 can be allocated to the memory location numbered 1, and so on.

[0046] Furthermore, in this embodiment, the first subtree can be rewritten according to a preset period. As data is continuously written, some subtrees may grow rapidly due to continuous data writing, causing their respective memory locations to approach the storage space limit. To avoid uneven memory load, after the subtree rewriting operation is completed, it can be checked whether the subtree meets the preset expansion rules.

[0047] Step S103: After the rewrite operation of the first subtree in the current cycle is completed, if it is detected that the first subtree meets the preset expansion rule, then in the rewrite operation of the first subtree in the next cycle, the first subtree is split into multiple second subtrees, where each second subtree corresponds to a second prefix.

[0048] In this embodiment, the expansion rule may include at least one of the following: the size of the first file corresponding to the first subtree is greater than a preset first threshold; the remaining storage space of the memory corresponding to the first subtree is less than a preset second threshold.

[0049] As an example, after the rewrite operation of the first subtree in the current cycle is completed, the size of the first file corresponding to the first subtree and the remaining storage space of the memory where the first subtree is located can be automatically detected. The first threshold can be set to 20GB, and the second threshold can be set to 10% of the total storage space of the memory. If the size of the first file corresponding to the first subtree is detected to be greater than 20GB or the remaining storage space of the memory is less than 10% of its total storage space, then a re-partitioning operation of the first subtree can be performed. It should be noted that the above-mentioned first and second thresholds can be set according to the actual application scenario and user expectations, and this application does not impose any restrictions on them.

[0050] In this embodiment, the re-partitioning operation of the first subtree may include the following steps:

[0051] Step 1: Set the first prefix of the first subtree to the first value, and set the same number of second values ​​according to the number of second subtrees. Each second value is unique.

[0052] Step 2: Add the second value after the first value in sequence to obtain multiple second prefixes.

[0053] Step 3: Write the data of the first file corresponding to the first subtree into the second file corresponding to each second prefix, thus obtaining multiple second subtrees.

[0054] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the splitting and allocation of the first subtree according to an embodiment of this application, as shown below. Figure 4 As shown, when a certain first subtree is detected ( Figure 1 When the first file corresponding to the first subtree (with the first prefix 1) has reached a set size threshold (e.g., its size exceeds 20GB) or the remaining storage space in the memory is about to be insufficient (e.g., less than 10% of the total storage space), in the rewrite operation of the first subtree in the next cycle, the first prefix of the first subtree can be further split, splitting the first prefix "1" into 16 second prefixes such as "10", "11", ..., "1F". The data of the first file corresponding to the first subtree is written into the second file corresponding to each second prefix, resulting in multiple second subtrees. Each newly generated second subtree will be independently written into a new file, thereby achieving finer-grained data partitioning and storage isolation.

[0055] Step S104: Match the second prefix of the second subtree with the memory number, and assign the second subtree to the memory corresponding to the number and the second prefix.

[0056] like Figure 4 As shown, after splitting the first subtree into 16 second subtrees, the second subtrees with the prefixes "10", "11", ..., "1F" can be sequentially written into the memory numbered "0", "1", ..., "F" in a one-to-one mapping manner.

[0057] Alternatively, storage allocation can be prioritized based on the remaining storage space of all current storage devices, prioritizing those with larger remaining storage spaces. For example, multiple second subtrees can be merged and allocated to storage devices with lower loads or more abundant remaining storage space to achieve balanced utilization of overall storage resources.

[0058] In this way, by detecting whether the first subtree meets the preset expansion rules, the first subtree that meets the expansion rules (i.e. the subtree with higher load) can be split a second time. This ensures that the first subtree can be dynamically split and migrated in a timely manner as the amount of data continues to grow. This effectively avoids hot spot concentration, avoids storage resource waste and storage space bottlenecks, improves the reading and writing efficiency of tree structure data, and ensures the stability and scalability of system operation.

[0059] In one embodiment, since the total amount of data in a subtree may not only increase but also decrease in some cases, a merging operation can be performed on some subtrees when the amount of data in some subtrees is small.

[0060] In this embodiment, the second subtree is rewritten according to a preset period. The merging operation includes the following steps:

[0061] Step 1: For the multiple second subtrees obtained by splitting the first subtree, after the rewrite operation of the second subtree in the current cycle is completed, if at least two second subtrees are detected to meet the preset shrinkage rule, then in the rewrite operation of the at least two second subtrees in the next cycle, the at least two second subtrees are merged into a third subtree, where the third subtree corresponds to the third prefix.

[0062] In this embodiment, the scaling rule may include: the size of the second file corresponding to the second subtree is less than a preset third threshold. This third threshold can be set to 10GB, 5GB, 2GB, etc. To avoid management redundancy and performance waste caused by allocating multiple smaller second subtrees to different storage devices, smaller second subtrees obtained by splitting the same first subtree can be merged.

[0063] Step 2: Match the third prefix of the third subtree with the memory number, and assign the third subtree to the memory corresponding to the number and the third prefix.

[0064] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the merging and allocation of a second subtree according to one embodiment of this application, as shown below. Figure 5 As shown, if at least two subtrees are detected after the rewrite operation of the second subtree in the current cycle is completed ( Figure 5 If the second subtrees shown with the second prefix "11", ..., "1F" satisfy the preset shrinkage rule, then the second prefixes ("11", ..., "1F") of at least two second subtrees can be merged into a third prefix ("1"); according to the order of the second prefixes, the data of at least two second subtrees are written into the third file corresponding to the third prefix in turn to obtain the third subtree; the file offset of each second subtree is recorded, and the file offset of each second subtree is recorded in the root node of the third subtree.

[0065] In this embodiment, if the first subtree with the first prefix "1" is split into 16 second subtrees with second prefixes "10" to "1F", and if it is detected that the size of the second file corresponding to only the second subtree with the second prefix "10" exceeds the third threshold (for example, it can be set to 10GB), then the second subtrees with second prefixes "11" to "1F" can be merged. During the rewriting process of the second subtrees with second prefixes "11" to "1F", the data of the second subtrees with second prefixes "11" to "1F" can be written into the third file corresponding to the third prefix "1". The writing order can be consistent with the arrangement order of the second prefixes. After each second subtree is written, the starting offset position of its root node in the file (i.e., the file offset) is recorded. Finally, these file offset information are uniformly recorded in the root node of the merged third subtree to describe the offset position of each original second subtree in the file, ensuring the accuracy and structural consistency of subsequent access.

[0066] In this way, efficient scaling down and resource reclamation of tree structure data can be achieved without affecting the efficiency and integrity of tree structure data access, thereby further enhancing the system's resilience and adaptability.

[0067] In addition, in one embodiment, memory redundancy may occur. For example, when the total amount of data in all memories is detected to be low, the original number of memories may be reduced (e.g., from 16 memories to 8 memories).

[0068] Specifically, after the rewrite operation of the first subtree in the current cycle is completed, if it is detected that at least one memory among multiple memories meets the preset migration rules, then in the rewrite operation of the first subtree in the next cycle, the first subtree corresponding to at least one memory is allocated to other memories among multiple memories according to the preset storage space and load balancing strategy.

[0069] As an example, if it is detected that 8 out of 16 memories (e.g., memories numbered 8 to F) meet the preset migration rules, or if it is detected that the total remaining storage space of the 16 memories is sufficient (e.g., the total remaining storage space is greater than 50% of the total storage space of the 16 memories), the subtrees on a preset number of memories (e.g., memories numbered 8 to F) can be evenly distributed and migrated to other memories (e.g., memories numbered 0 to 7) according to the storage space and load balancing strategy.

[0070] The storage space and load balancing strategy in the above embodiments may include: after the subtree migration of the memory is completed, the difference in remaining storage space between any two memory locations in the remaining memory is less than a preset fourth threshold (for example, it can be set to 5% of the total storage space of the memory).

[0071] By combining the migration process with the existing rewriting mechanism, the consistency of tree structure data and the atomicity of operations can be ensured.

[0072] In one embodiment, after matching the second prefix of the second subtree with the memory number and assigning the second subtree to the memory corresponding to the second prefix, the method further includes the following steps:

[0073] Step 1: Write the data to be written to the pre-write log.

[0074] In this embodiment, the tree structure is organized and stored in key-value pair format. For data writing scenarios, a write-ahead log (WAL) mechanism can be used to balance the efficiency of concurrent data writing.

[0075] To ensure transaction integrity during write operations and data recovery capabilities in the event of system failures, all data to be written is first written to the Write-Ahead Log (WAL). Compared to distributing the data to be written to multiple subtrees for concurrent writing, writing the data to the WAL first avoids the commit delay problem of the entire batch of data caused by the slow write speed of individual memory locations. In addition, centralized WAL writing also reduces the complexity of transaction management that would otherwise require spanning multiple subtrees.

[0076] Step 2: Based on the given key, use a preset prefix matching strategy to allocate the data to be written in the pre-write log to the first subtree or the second subtree; wherein, the prefix matching strategy includes: if the given key matches both the first prefix and the second prefix, write the data to be written in the pre-write log to the second subtree corresponding to the second prefix; if the given key matches only the first prefix, write the data to be written in the pre-write log to the first subtree corresponding to the first prefix.

[0077] After a successful WAL write, the data to be written can be asynchronously distributed to the corresponding subtree based on the given key, and the actual write operation can be performed. The distribution process adopts a "prefix-first matching" strategy, that is, for a given key, among all existing subtree prefixes, the subtree with the longest prefix is ​​matched first. For example, when there is a first subtree with the first prefix "1" and a second subtree with the second prefix "10", the given key 10XX... will be routed to the second subtree with the second prefix "10" first, rather than the first subtree with the first prefix "1". This ensures that the data to be written falls into the subtree with the finest and most matching structure, thereby improving storage granularity control and subsequent access efficiency.

[0078] In this way, by storing the subtrees corresponding to different prefixes in different memories, data can be concurrently written to multiple memories after distribution, thereby giving full play to the parallel writing capabilities of multiple memories and improving the overall throughput and response performance of the system.

[0079] Furthermore, the data reading operation is similar to the writing operation, which is to match the corresponding subtree according to the given key and read in parallel. This application will not elaborate on this further.

[0080] Through steps S101 to S104, the first subtree in the tree structure data is matched with the memory number, and the first subtree is allocated to the memory corresponding to the first prefix. After the rewrite operation of the first subtree in the current cycle is completed, if the first subtree is detected to meet the expansion rule, the first subtree is split into multiple second subtrees in the next cycle's rewrite operation. The second prefix of the second subtree is matched with the memory number, and the second subtree is allocated to the memory corresponding to the second prefix. In this way, by analyzing the prefix of the subtree, the original large tree structure data is logically split into multiple smaller subtrees. Each subtree can be stored independently in different memories, realizing data sharding and parallel storage, thereby breaking through the storage space limitation of a single memory. In addition, by detecting whether the first subtree meets the preset expansion rule, the first subtree that meets the expansion rule (i.e., the subtree with higher load) can be split a second time, thereby alleviating local hotspots, avoiding storage resource waste and performance bottlenecks, and improving the read and write efficiency of tree structure data. This application solves the problem of low read / write efficiency of tree structure data due to the use of a single memory to store tree structure data in related technologies, and achieves the technical effect of improving the read / write efficiency of tree structure data.

[0081] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0082] Corresponding to the tree structure data storage method described in the above embodiments, Figure 6 A schematic diagram of a tree-structured data storage device according to an embodiment of this application is shown. This device is used to train a continuous-time dynamic graph model. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0083] Please see Figure 6 The tree structure data storage device 6 includes: an acquisition module 60, used to acquire tree structure data to be stored, wherein the tree structure data includes multiple first subtrees; a first allocation module 61, used to match the first prefix of the first subtree with the number of a preset memory, and allocate the first subtree to the memory corresponding to the number and the first prefix, wherein the first subtree is rewritten according to a preset period; a splitting module 62, used to split the first subtree into multiple second subtrees in the next period's rewrite operation if the first subtree is detected to meet a preset expansion rule after the rewrite operation of the first subtree in the current period, wherein each second subtree corresponds to a second prefix; and a second allocation module 63, used to match the second prefix of the second subtree with the number of the memory, and allocate the second subtree to the memory corresponding to the number and the second prefix.

[0084] In one embodiment, the splitting module 62 is further configured to set the first prefix of the first subtree to a first value, set the same number of second values ​​according to the number of second subtrees, wherein each second value is unique; add second values ​​sequentially after the first values ​​to obtain multiple second prefixes; and write the data of the first file corresponding to the first subtree into the second file corresponding to each second prefix to obtain multiple second subtrees.

[0085] In one embodiment, the expansion rule includes at least one of the following: the size of the first file corresponding to the first subtree is greater than a preset first threshold; the remaining storage space of the memory corresponding to the first subtree is less than a preset second threshold.

[0086] In one embodiment, the second subtree is rewritten according to a preset period. The tree structure data storage device 6 further includes a merging module, which is used to merge at least two second subtrees into a third subtree in the next period of the rewriting operation of the tree structure data storage device 6, after the rewriting operation of the first subtree is completed and at least two second subtrees satisfy a preset shrinkage rule. The third subtree corresponds to a third prefix. The tree structure data storage device 6 further includes a third allocation module, which is used to match the third prefix of the third subtree with the memory number and allocate the third subtree to the memory number corresponding to the third prefix.

[0087] In one embodiment, the merging module is further configured to merge the second prefixes of at least two second subtrees into a third prefix; write the data of at least two second subtrees into the third file corresponding to the third prefix in the order of the second prefixes to obtain the third subtree; record the file offset of each second subtree, and record the file offset of each second subtree in the root node of the third subtree.

[0088] In one embodiment, the tree-structured data storage device 6 further includes a migration module, which, after the rewrite operation of the first subtree in the current cycle is completed, if it is detected that at least one memory among the multiple memories satisfies a preset migration rule, then in the rewrite operation of the first subtree in the next cycle, according to a preset storage space and load balancing strategy, allocates the first subtree corresponding to at least one memory to other memories among the multiple memories.

[0089] In one embodiment, the tree-structured data storage device 6 further includes a writing module for writing data to be written to a pre-write log; and for allocating the data to be written in the pre-write log to a first subtree or a second subtree based on a given key and using a preset prefix matching strategy; wherein the prefix matching strategy includes: when the given key matches both the first prefix and the second prefix, writing the data to be written in the pre-write log to the second subtree corresponding to the second prefix; and when the given key matches only the first prefix, writing the data to be written in the pre-write log to the first subtree corresponding to the first prefix.

[0090] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0092] Figure 7 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Figure 7 As shown, the terminal device 7 includes: at least one processor 70 ( Figure 7 (Only one is shown) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on at least one processor 70. When the processor 70 executes the computer program 72, it implements the steps in any of the above-described tree structure data storage method embodiments.

[0093] Terminal device 7 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 7 may include, but is not limited to, processor 70 and memory 71. Those skilled in the art will understand that... Figure 7The example of terminal device 7 is merely an illustration and does not constitute a limitation on terminal device 7. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0094] The processor 70 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0095] In some embodiments, memory 71 may be an internal storage unit of terminal device 7, such as a hard disk or memory of terminal device 7. In other embodiments, memory 71 may be an external storage device of terminal device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on terminal device 7. In other embodiments, memory 71 may include both internal and external storage units of terminal device 7. Memory 71 is used to store operating system, application programs, bootloader, data, and other programs, such as the program code of computer program 72. Memory 71 may also be used to temporarily store data that has been output or will be output.

[0096] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described tree structure data storage method embodiments.

[0097] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the various tree structure data storage method embodiments.

[0098] This application implements all or part of the processes in the methods of the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0099] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0100] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A tree-structured data storage method, characterized in that, include: Obtain tree structure data to be stored, wherein the tree structure data includes multiple first subtrees; The first prefix of the first subtree is matched with the number of a preset memory, and the first subtree is assigned to the memory whose number corresponds to the first prefix. The first subtree is rewritten according to a preset period. After the rewrite operation of the first subtree in the current cycle is completed, if it is detected that the first subtree meets the preset expansion rule, then in the rewrite operation of the first subtree in the next cycle, the first subtree is split into multiple second subtrees, wherein each second subtree corresponds to a second prefix; The second prefix of the second subtree is matched with the number of the memory, and the second subtree is assigned to the memory corresponding to the number and the second prefix.

2. The method according to claim 1, characterized in that, Splitting the first subtree into multiple second subtrees includes: Set the first prefix of the first subtree to a first value, and set the same number of second values ​​according to the number of second subtrees, wherein each second value is unique; By sequentially adding the second value after the first value, multiple second prefixes are obtained; The data of the first file corresponding to the first subtree is written into the second file corresponding to each of the second prefixes to obtain the plurality of second subtrees.

3. The method according to claim 1 or 2, characterized in that, The expansion rules include at least one of the following: the size of the first file corresponding to the first subtree is greater than a preset first threshold; the remaining storage space of the memory corresponding to the first subtree is less than a preset second threshold.

4. The method according to claim 1 or 2, characterized in that, The second subtree is rewritten according to the preset period; after matching the second prefix of the second subtree with the number of the memory and allocating the second subtree to the memory corresponding to the number and the second prefix, the method further includes: For the multiple second subtrees obtained by splitting the first subtree, if at least two second subtrees are detected to meet the preset shrinkage rule after the rewrite operation of the second subtree in the current cycle is completed, the at least two second subtrees are merged into a third subtree in the rewrite operation of the at least two second subtrees in the next cycle, wherein the third subtree corresponds to the third prefix; The third prefix of the third subtree is matched with the number of the memory, and the third subtree is assigned to the memory corresponding to the number and the third prefix.

5. The method according to claim 4, characterized in that, Merging the at least two second subtrees into a third subtree includes: Merge the second prefixes of the at least two second subtrees into a third prefix; According to the arrangement order of the second prefix, the data of at least two second subtrees are sequentially written into the third file corresponding to the third prefix to obtain the third subtree; Record the file offset of each second subtree, and record the file offset of each second subtree in the root node of the third subtree.

6. The method according to claim 1 or 2, characterized in that, After matching the first prefix of the first subtree with the preset memory number and assigning the first subtree to the memory corresponding to the number and the first prefix, the method further includes: After the rewrite operation of the first subtree in the current cycle is completed, if it is detected that at least one memory among the multiple memories meets the preset migration rule, then in the rewrite operation of the first subtree in the next cycle, the first subtree corresponding to the at least one memory is allocated to other memories among the multiple memories according to the preset storage space and load balancing strategy.

7. The method according to claim 1 or 2, characterized in that, After matching the second prefix of the second subtree with the number of the memory, and assigning the second subtree to the memory corresponding to the number and the second prefix, the method further includes: Write the data to be written to the pre-write log; Based on a given key, a preset prefix matching strategy is used to allocate the data to be written in the pre-write log to the first subtree or the second subtree. The prefix matching strategy includes: when the given key matches both the first prefix and the second prefix, writing the data to be written in the pre-write log to the second subtree corresponding to the second prefix; when the given key matches only the first prefix, writing the data to be written in the pre-write log to the first subtree corresponding to the first prefix.

8. A tree-structured data storage device, characterized in that, include: An acquisition module is used to acquire tree structure data to be stored, wherein the tree structure data includes multiple first subtrees; The first allocation module is used to match the first prefix of the first subtree with the number of a preset memory, and allocate the first subtree to the memory corresponding to the number and the first prefix, wherein the first subtree is rewritten according to a preset period. The splitting module is used to split the first subtree into multiple second subtrees in the next cycle of the first subtree if the first subtree is detected to meet the preset expansion rules after the rewrite operation of the first subtree in the current cycle is completed. Each second subtree corresponds to a second prefix. The second allocation module is used to match the second prefix of the second subtree with the number of the memory, and allocate the second subtree to the memory corresponding to the number and the second prefix.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the tree structure data storage method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the tree structure data storage method as described in any one of claims 1 to 7 to be executed.

Images