A method for organizing distributed terminal files based on MPT tree

Abstract

The application discloses a kind of distributed terminal file organization methods based on MPT tree.The method first constructs multiple distributed storage nodes, forms the storage resource of terminal equipment file;Based on the unique identification of each terminal equipment, using MPT tree (prefix-based tree index structure) index file storage location, MPT tree includes extension node, branch node and leaf node.When storing file, according to the remaining space of storage node and the number of stored files, the available volume rate is calculated, and the target node is selected according to the volume rate from high to low;If node space is insufficient, the next one is excluded and selected in order.The file search, insertion, update or deletion operation is performed by traversing MPT tree.Periodically dynamic inspection process traverses MPT tree to obtain storage state, and dynamically adjusts the available volume rate calculation parameter, to optimize storage efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of digital storage, and particularly relates to a distributed terminal file organization method based on MPT tree. BACKGROUND

[0002] The management platform needs to access terminal devices such as cameras, power meters, and collectors, so the management platform sometimes needs to store the information of these devices. A common method is to use one or more memories to save the terminal device information. The currently common method is to calculate the hash value of the terminal file, then take the remainder of the hash, and determine the file to be placed in which memory according to the remainder. In order to improve the efficiency of searching terminal devices from the storage, an index needs to be added for the information of these devices. The conventional index organization and allocation method is to calculate the hash value of the terminal number, determine the location of the terminal file in the storage according to the hash value, and if the hash values of two or more terminals are the same, link them together using a linked list. If a query is needed, the linked list is traversed to find the node where the real terminal is located.

[0003] This terminal device storage design has two deficiencies: first, the terminal number is estimated according to the terminal number, which may cause too much memory allocation, and if the terminal number needs to be expanded, the terminal number in the configuration file also needs to be modified; second, if the hash calculation function is not properly selected, or because of the terminal number, more input terminal numbers have the same hash value calculated, which will increase the probability of hash collision, and thus reduce the index function; third, the same hash value will also cause the storage of the file in the distributed terminal file system to be uneven, some memories have a high usage rate, and other memories have a low usage rate, which will reduce the usage rate of the entire distributed terminal file system, and also cause the fragmentation of files in some memories to be serious, which is not conducive to subsequent maintenance and retrieval.

[0004] When the number of terminals is in the order of millions, the management of the terminal files in the order of millions requires higher requirements for the design of the index, hash function, and distributed terminal file system storage. Therefore, finding a suitable distributed terminal file storage method is of great benefit to the efficiency of the system. SUMMARY

[0005] Based on the above problems, the application provides a distributed terminal file organization method based on MPT tree (Merkle Patricia Trie), that is, based on MPT tree to organize and manage device terminals, and the terminal search and maintenance are no longer dependent on the organization mode of hash index and no longer dependent on the selection of hash function. In the distributed storage process, the remaining capacity of the storage and the number of saved files are fully considered to select the optimal storage, and the corresponding terminal is managed in this way, so that the time complexity of the query, update and deletion operation is controlled within O(1), and the remaining capacity of each storage and the number of saved files are fully considered to improve the storage effect of the distributed terminal file system, reduce the centralized storage of fragmented files, and optimize the storage of the distributed terminal file system.

[0006] The method specifically comprises the following steps:

[0007] S1: a plurality of distributed storage nodes are constructed to constitute a distributed terminal file resource pool for storing terminal device files;

[0008] S2: based on the unique device identifier of each terminal device, a prefix-based tree index structure is used to construct an MPT tree, and the MPT tree indexes the storage position of the terminal device file in the distributed storage node through the extension node, branch node and leaf node contained in the MPT tree;

[0009] S3: when a terminal file needs to be stored, the available volume rate of each storage node is calculated according to the remaining storage space and the number of stored files, and the target storage node is selected in descending order of the available volume rate; when the remaining space of the currently selected storage node is insufficient, the node is excluded and the next storage node is sequentially selected; when all available storage nodes cannot meet the storage requirement, a storage failure prompt is returned or a system expansion process is triggered;

[0010] The calculation formula of the available volume rate V is: k>0, k is a proportion coefficient for adjusting the balance between the remaining storage space and the number of files, and the initial value is set to 1, S i represents the remaining storage space of the i-th storage node, F i represents the number of files currently stored in the i-th storage node, 1≤i≤m, and m represents the total number of distributed storage nodes;

[0011] S4: the MPT tree structure is traversed to perform the search, insertion, update or deletion operation on the terminal device file; wherein,

[0012] For the insertion operation, the target storage node is found based on S3, the file is inserted, the available volume rate of the storage node is updated after the insertion, and the MPT tree is updated to reflect the storage position of the new file.

[0013] For the delete operation, after deleting the file, the available volume rate of the corresponding storage node is updated, and the MPT tree is updated to remove the index of the file;

[0014] For the update operation, first, it is judged whether the file size after the update exceeds the remaining capacity of the original storage node; when it does not exceed, the file is updated in the original node, and the MPT tree is updated to maintain the metadata change; when it exceeds, the S3 process is re-executed for the file to select a new target storage node, perform file migration and update, and synchronously update the available volume rate of the related storage node and the MPT tree;

[0015] For the find operation, after locating the file by traversing the MPT tree, the result is directly returned, without any update to the MPT tree;

[0016] S5: periodically performing a dynamic patrol process, traversing the MPT tree to obtain the system storage state, and dynamically adjusting the parameters for calculating the available volume rate in S3 based on the obtained storage state; the dynamic adjustment of the parameters for calculating the available volume rate in S3 is specifically the dynamic adjustment of the proportion coefficient k, including the following sub-steps:

[0017] Calculate the average available volume rate of all m storage nodes

[0018] According to the current available volume rate V i of each storage node, update its own proportion coefficient k according to the ratio of the average available volume rate :

[0019] Where i represents the ith storage node, 1≤i≤m, and m represents the total number of distributed storage nodes.

[0020] Preferably, step S2 includes: constructing an MPT tree based on the SN code of each terminal device, wherein the SN code is taken as the key and the position of the corresponding device information in the distributed storage is taken as the value.

[0021] And in the MPT tree:

[0022] The extension node is configured to merge device identifiers with the same prefix and point to the next node. The branch node contains a fixed number of sub-node pointers to navigate according to the current character of the device identifier.

[0023] The leaf node is configured to store the specific location information of the terminal device file in the distributed storage node.

[0024] Preferably, the searching operation in step S4 further comprises: starting from the root node of the MPT tree, and recursively searching with the complete device identifier as the initial key.

[0025] In the recursive process, corresponding operations are performed according to the current node type:

[0026] When the current node is a branch node, a corresponding child node path is selected according to the current character of the key remaining part, and the recursive search is continued.

[0027] When the current node is an extension node, the key remaining part is compared with the prefix of the extension node.

[0028] When the key remaining part starts with the prefix of the extension node, the prefix is removed from the key remaining part, and the recursive search of the remaining key part is continued in the child node pointed to by the extension node.

[0029] When the key remaining part does not start with the prefix of the extension node, the search is terminated and a search failure is returned.

[0030] When the current node is a leaf node, the key remaining part is compared with the path of the leaf node.

[0031] When the complete match is found, the terminal file location information stored in the leaf node is successfully obtained.

[0032] When the complete match is not found, a search failure is returned.

[0033] Through the above process, the target leaf node is finally located or it is confirmed that the search fails.

[0034] Preferably, the inserting operation in step S4 further comprises:

[0035] Starting from the root node of the MPT tree, and recursively searching for the insertion position with the complete device identifier as the initial key.

[0036] In the recursive process, corresponding operations are performed according to the current node type:

[0037] When the current node is empty, a new leaf node is created, and the location information of the terminal file is stored as the value of the node.

[0038] When the current node is an extension node and the key remaining part completely matches the prefix of the extension node, the matched prefix is removed from the key remaining part, and the recursive call of the insertion operation is continued to process the child node.

[0039] When the current node is an extended node and the remaining part of the key does not exactly match the prefix of the extended node, a new extended node is created based on their common prefix, and it points to a newly created branch node. Subsequently, the remaining part of the original extended node after splitting and the remaining part of the new key are mounted as different child nodes of the branch node.

[0040] When the current node is a branch node, the corresponding child node path is selected based on the current character of the remaining part of the key, and the child node corresponding to the insertion operation is processed recursively.

[0041] When the current node is a leaf node, compare the remaining key with the path of that leaf node: if they are exactly the same, update the terminal file location information of that leaf node. If they are not exactly the same, create a new branch node, and mount the original leaf node and the new leaf node as different child nodes of that branch node.

[0042] The insertion operation includes an upgrade process, which involves converting a leaf node or an extended node into a branch node and mounting the remaining part of the original node's path as a child node of the branch node. When multiple child nodes of a branch node have a common prefix, these child nodes with the common prefix are merged to create an extended node, and the common prefix is ​​used as the path of the extended node to dynamically optimize the tree structure.

[0043] After successfully inserting a node, the location information of the terminal device file in the distributed storage is stored in that node, and the available volume ratio of the storage corresponding to that location is updated.

[0044] Preferably, the deletion operation in step S4 further includes:

[0045] Starting from the root node of the MPT tree, recursively search for the node to be deleted, using the complete device identifier as the initial key.

[0046] During the recursion, perform the corresponding operation based on the current node type:

[0047] When the current node is a branch node, the corresponding child node path is selected based on the current character of the remaining key and recursively deleted. After recursion, if the branch node has only one child node left, a downgrading operation is performed to convert it into an expanded node or a leaf node.

[0048] If the current node is an extended node, then determine whether the remaining part of the key begins with the prefix of that extended node:

[0049] If so, remove the prefix of the extended node from the remaining key and recursively delete its child nodes.

[0050] If not, the deletion process will terminate.

[0051] When the current node is a leaf node, compare the remaining part of the key with the path of that leaf node: if there is a complete match, delete the leaf node.

[0052] The deletion process is terminated if there is no complete match.

[0053] The deletion operation includes a downgrading process, which refers to converting a branch node with only one child node into an extended node or a leaf node, and performing a merge operation on consecutive mergeable extended nodes.

[0054] After successfully deleting a node, the corresponding terminal file in the distributed storage is deleted based on the location information stored in that node, and the available capacity of the corresponding storage is updated.

[0055] Preferably, after determining the target storage node for the file, the file is transferred to that node via SSH or FTP protocol.

[0056] Preferably, the distributed storage node includes at least one of a disk array, a network-attached storage device, or a cloud storage service.

[0057] Compared to existing technologies, this organizational method supports insertion, deletion, modification, and query operations on terminal device information. Using the MPT tree, the corresponding device information can be located quickly and accurately. Another significant advantage of the MPT tree structure is its ability to effectively reduce memory usage. The design of the MPT tree allows for dynamic adjustment of memory allocation based on actual needs, thus avoiding the tree structure bloat caused by an excessive number of estimated terminals. Furthermore, another important feature of the MPT tree structure is that it does not rely on a hash function. This means that compared to traditional hash-based indexing methods, the MPT tree structure can effectively avoid the indexing performance degradation caused by hash collisions. This design allows the MPT tree to maintain high efficiency and stable performance even when processing large amounts of device information. Meanwhile, this organizational method fully considers the capacity of each storage device and the number of files already stored. By calculating the available volume ratio V of each storage device in the distributed terminal file system, which measures the storage status of each distributed storage device, and by adjusting the corresponding k value of each storage device, distributed storage can be implemented more intelligently. This allows subsequent insertion or modification operations to locate distributed storage files in storage devices with better available volume ratios, avoiding storing too many files in one storage device, avoiding fragmentation and waste of distributed storage space, and improving the utilization rate of distributed storage space.

[0058] The above overview and the following embodiments are provided to further illustrate the technical means and effects of the present invention. However, the described embodiments and accompanying drawings are for illustrative purposes only and are not intended to limit the present invention. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of key-value pairs and an MPT tree formed according to an embodiment of the present invention. 57399123573, 57399761534, 57399761689, and 57399958573 all represent example keys (i.e., combinations of all node keys traversed on the path from the root node to the leaf node), and their values ​​represent the corresponding location information in the distributed memory.

[0061] Figure 3 This is a schematic diagram of the MPT tree search operation process according to an embodiment of the present invention.

[0062] Figure 4 This is a schematic diagram of an MPT tree insertion operation according to an embodiment of the present invention, where n represents the device SN code of the inserted device.

[0063] Figure 5 This is a schematic diagram of an MPT tree deletion operation process according to an embodiment of the present invention, where n represents the device SN code of the device to be deleted.

[0064] Figure 6 This is a schematic diagram of a dynamic inspection process according to an embodiment of the present invention. Detailed Implementation

[0065] The management system constructed in this embodiment sets up terminal devices such as cameras, electricity meters, and data collectors. At the same time, the system integrates multiple storage devices into intelligent organization and access storage. The specific operation steps include: Step 1: Construct multiple distributed storage nodes to form a distributed terminal file system for storing terminal device files.

[0066] Distributed endpoint file systems can store files across multiple local or remote storage devices, such as disk arrays, network attached storage (NAS) devices, or cloud storage services, to achieve high availability and redundant backups. To ensure the operation of the distributed endpoint file system, the system needs to pre-configure multiple distributed storage devices as storage nodes for subsequent files. The capacity specifications of these storage devices can vary and do not need to be completely identical. Because in step 3, when performing the actual file storage operation, the system intelligently considers the key factor of the capacity difference of each storage node and calculates the available capacity ratio of each storage device. Specifically, the system uses a dynamic allocation algorithm to optimize the distributed storage strategy of files based on parameters such as the remaining available space of each storage device, the number of stored files, and performance indicators, thereby ensuring the most rational utilization of storage resources. This design approach improves the flexibility of the storage system while ensuring optimal storage efficiency.

[0067] To obtain the unique identifier or device code corresponding to each terminal device, it is necessary to ensure that each device has a unique identification code defined according to certain rules, such as the device serial number (SN) given at the factory. To build a distributed terminal file system, several storage devices are needed to form the distributed terminal file system; these storage devices do not need to have the same storage space size. Step 2: Construct an MPT tree based on the unique device identifier of each terminal device.

[0068] Based on the device serial number (SN) of each device, an MPT (Multi-Level Tree) structure is constructed, using the device code of each device as the key and the location of that device information in the distributed storage as the value. The MPT tree structure mainly has three types of nodes: extension nodes, branch nodes, and leaf nodes.

[0069] The primary function of an extended node is to merge key-value pairs with the same prefix, i.e., device codes with the same prefix. These are the keys of the extended node. The value of an extended node points to the next child node, and the value only has a pointer to the next node's location.

[0070] Since device codes are generally 10 digits from 0 to 9, branch nodes always contain 11 elements. The first 10 elements correspond to the numbers 0 to 9 and are used to index different child nodes, while the 11th element is used to store the node content at the end of the current node.

[0071] Leaf nodes play the role of storing the actual terminal memory data. They do not contain any child nodes and are the very end of the entire structure. Each device terminal will have a corresponding child node at the end. The value of the child node is the remaining part of the last few bits of the terminal device code that is different from all other terminal device codes. The value of the leaf node is the storage location of the terminal device in memory.

[0072] The construction of the MPT tree is also a gradual process. Based on the functions of the nodes mentioned above, device addition requests initiated through the distributed terminal file system are used to add devices to each terminal one by one, thus constructing a complete MPT tree. The MPT tree is a device code index tree for each device, used to quickly locate the device file of a specific device, and it is located in the storage location of the distributed terminal file system.

[0073] This structure can efficiently maintain and manage the information corresponding to each device, ensuring data integrity and consistency.

[0074] Next, the upper-layer platform will operate this distributed terminal file system platform to achieve comprehensive management of device file information, including operations such as inserting, deleting, modifying, and querying device information. This will not only ensure the accuracy and real-time nature of device information but also provide efficient and reliable data support for upper-layer applications.

[0075] The overall flowchart of this embodiment is shown below. Figure 1 As shown.

[0076] Step 3: Calculate the available capacity of the distributed terminal file system.

[0077] For each terminal file that needs to be stored, it is necessary to determine which storage device the file should be stored in.

[0078] Before storing a terminal file, the distributed terminal file system first needs to determine which storage device to store it in based on the current available capacity of each storage device. The available capacity V is calculated as follows:

[0079]

[0080] Where: k > 0, k represents a coefficient, a proportional coefficient used to balance the remaining storage space and the number of files, S i F represents the remaining storage space of the i-th memory. i This represents the number of files stored in the i-th storage unit, where 1 ≤ i ≤ m, and m represents the total number of distributed storage units. A larger available capacity ratio (V) indicates a better state of remaining available capacity for that storage unit. If storage space is prioritized, k is a number greater than 1; if file fragmentation is prioritized, k is a number less than 1. Considering both factors, k should ideally be 1, and it is generally set to 1 initially.

[0081] This calculation method can determine the optimal memory for storing terminal files each time, preparing for subsequent insertion operations.

[0082] If the calculated optimal storage space is insufficient to accommodate the new file, then that storage space is excluded, and the search continues to find the next optimal storage space based on the available capacity.

[0083] Furthermore, by combining the initial configuration in this step with the dynamic inspection and self-optimization process in subsequent step 9, the available capacity ratio coefficient k can be dynamically adjusted to achieve different storage optimization goals (prioritizing space, prioritizing fragmentation reduction, or balancing considerations).

[0084] The advantage of this design is that it can fully consider the remaining space of each storage device, preventing uneven use of each storage device. At the same time, it can consider the number of files already stored in each storage device, preventing too many fragmented files from being stored in one storage device, which would be detrimental to the later work of searching in the distributed terminal file system.

[0085] If all available storage devices are unable to meet the storage demand, a storage failure message will be returned or the system expansion process will be triggered, prompting the system to handle the situation.

[0086] Step 4: Traversing the distributed terminal file system.

[0087] The traversal of the distributed terminal file system mainly involves traversing the device encoding index MPT tree. The traversal mainly involves processing leaf nodes, extended nodes, and branch nodes. The traversal process starts from the root node and uses a recursive algorithm to expand downwards according to the node type.

[0088] Step 5: Searching the distributed terminal file system.

[0089] When a distributed terminal file system initiates a search request for a device terminal file, it first needs to determine the location of the device terminal in the distributed storage using the maintained MPT tree. In the MPT tree, each node stores a partial device code. These device codes, combined from the root to the final node, serve as the key. The final node stores the corresponding location of the terminal file in the storage as the value. Together, they form a key-value pair, determining the file's exact location. During the recursive search process, the complete device code is used as the initial key, starting from the root (layer). When recursively entering the next layer, the common prefix of the current layer is removed, and the remaining key is used as input to recursively search the next layer. Therefore, in this embodiment, the remaining key refers to the part remaining after removing the common prefix of the upper layer. Initially, starting from the root (layer), the remaining key refers to the entire key.

[0090] The search process starts from the root node and uses the device code as the initial key to perform a recursive search.

[0091] When a branch node is encountered, the corresponding child node path is selected based on the current character of the remaining part of the key. After removing the current character, the remaining key part is recursively searched in the child node path to ensure the correctness of the search path.

[0092] When an extended node is encountered, the remaining part of the key is compared with the prefix of that extended node to determine the matching relationship between the two:

[0093] If the remaining key starts with the prefix of the extended node, the matching prefix is ​​removed from the remaining key, and the remaining key is recursively searched in the child nodes pointed to by the extended node.

[0094] If the remaining part of the key does not begin with the prefix of the extended node, the search process is terminated and a search failure is returned.

[0095] When a leaf node is encountered, compare the remaining part of the key with the path of that leaf node:

[0096] If the remaining part of the key completely matches the path of the leaf node, then the location information of the terminal file stored in that leaf node is obtained.

[0097] If the remaining part of the key does not completely match the path of the leaf node, the search process is terminated and a search failure is returned.

[0098] Through the above recursive search process, the target leaf node (i.e., the storage location) is eventually located or the search failure is confirmed.

[0099] A diagram illustrating the MPT tree search operation is shown below. Figure 3 As shown.

[0100] The search process in the MPT tree has a time complexity of only O(1) (O(1) indicates constant time, meaning it can be completed in constant time, with a fixed execution time that is independent of the amount of input data. All instances of O(1) mentioned in this invention represent this meaning). After finding the value of the corresponding terminal node, the terminal file can be located in the corresponding position in the distributed storage. Step 6: Updating the distributed terminal file system.

[0101] When a distributed terminal file system initiates an update request for a device terminal file, it first determines whether the size of the updated file exceeds the remaining capacity of the original storage node.

[0102] If the limit is not exceeded, the file is updated on the original node, and the MPT tree is updated to maintain metadata changes. First, the location P of the device terminal in the distributed storage is determined based on the device SN code of the device through the maintained MPT tree, i.e., the device SN and storage location P. The method for finding the device terminal file is as shown in step 5. After successfully finding the accurate location P corresponding to the terminal device number SN key-value pair to be updated, the content information of the terminal file stored in the distributed storage location P can be modified.

[0103] When the limit is exceeded, the S3 process is re-executed for the file to select a new target storage node, perform file migration (i.e., deletion at the original location and insertion at the new location) and update, and synchronously update the available volume ratio and MPT tree of the relevant storage.

[0104] The time complexity of the update operation is O(1).

[0105] Step 7: Add a distributed terminal file system.

[0106] After the distributed terminal file system initiates an insertion request for a device terminal file, it determines which storage in the distributed terminal file system the corresponding device file should be stored in. This can be done by calculating the available volume ratio V of each storage in the distributed terminal file system using Formula 1. Based on the maximum value of V and whether the remaining space of the storage can accommodate the device file, the storage location P of the terminal file is determined. The next step is to insert the device SN and storage location P of the device file into the MPT tree for subsequent indexing.

[0107] The insertion process is as follows:

[0108] The system will use the device serial number (SN) of the newly added device (a unique label that comes with the device at the factory) as the initial key to recursively search and insert the device, starting from the root node of the MPT tree. First, it will determine the node type:

[0109] When the current node is an empty node, a new leaf node is created to store the terminal file location information.

[0110] When the current node is a branch node, select the corresponding child node path based on the current character of the remaining part of the key, and recursively call the insertion operation to process the child node.

[0111] When the current node is an extended node, compare the remaining part of the key with the prefix of that extended node:

[0112] When the remaining key matches the prefix of the extended node exactly, remove the matching prefix from the remaining key and recursively call the insertion operation to process the child node pointed to by the extended node.

[0113] When the remaining part of the key does not exactly match the prefix of the extended node, calculate the common prefix of the two, create a new extended node to store the common prefix, and make it point to the newly created branch node. The remaining part of the original extended node after splitting and the remaining part of the new key are respectively mounted as different child nodes of the branch node.

[0114] When the current node is a leaf node, compare the remaining part of the key with the path of that leaf node:

[0115] When the remaining part of the key is exactly the same as the path of the leaf node, update the terminal file location information stored in that leaf node.

[0116] When the remaining part of the key is not exactly the same as the path of the leaf node, a new branch node is created, and the original leaf node and the new leaf node are mounted as different child nodes of the branch node.

[0117] The time complexity of the insertion operation is also O(1). A flowchart of the MPT tree insertion operation is shown below. Figure 4 As shown.

[0118] During insertion operations in a distributed terminal file system, the structure of the MPT tree is inevitably affected. Specifically, the level of some nodes changes; some nodes may be promoted in the tree due to the insertion of a new node. This phenomenon is called "promotion." In detail, the promotion operation involves converting a leaf node or extended node into a branch node, mounting the remaining path of the original node as a child node of the branch node, and merging multiple child nodes with a common prefix to create an extended node, thereby optimizing the tree structure. Through the MPT tree's promotion operation, the system can dynamically adjust the overall tree structure.

[0119] After inserting the newly added terminal device into the MPT tree position using the above method, the position information P is filled into the value position of the corresponding inserted device node in the MPT tree. At the same time, the available volume ratio of the memory corresponding to position P is updated. For each terminal file that needs to be stored, the terminal file name is inserted into the MPT tree as a key, and the value corresponding to each key in the MPT tree is the memory number and position, which is used to indicate which memory and position the terminal file is stored in. By indexing the distributed storage location using the MPT tree, the corresponding information content in the specified terminal file can be queried in O(1) time. Moreover, this search efficiency is even more efficient when the number of terminal files reaches millions. Because the terminal serial number is unique, the hash function of the terminal serial number is no longer performed. The advantage of doing this is that it does not need to consider the collision caused by the hash function and the resulting linked list storage, which can reduce storage space and improve the efficiency of system maintenance.

[0120] Step 8: Deletion of the distributed terminal file system.

[0121] When the distributed terminal file system initiates a deletion request for a device terminal file, it recursively searches for the node to be deleted, starting from the root node of the MPT tree and using the complete device SN code as the initial key.

[0122] First, determine the node type:

[0123] When the current node is a branch node, select the corresponding child node path based on the current character of the remaining key and recursively delete them. After recursion, check the number of child nodes of the branch node:

[0124] When a branch node has only one child node left, perform a downgrading operation to convert it into an extended node or a leaf node.

[0125] When a branch node has two or more child nodes remaining, keep the branch node state unchanged.

[0126] When the current node is an extended node, determine whether the remaining part of the key begins with the prefix of that extended node:

[0127] When the remaining key begins with a prefix of the extended node, remove the matching prefix from the remaining key and recursively delete its child nodes.

[0128] The deletion process terminates when the remaining part of the key does not begin with the prefix of the extended node.

[0129] When the current node is a leaf node, compare the remaining part of the key with the path of that leaf node:

[0130] Delete a leaf node when the remaining part of the key completely matches the path of the leaf node.

[0131] The deletion process terminates when the remaining part of the key does not completely match the path of the leaf node.

[0132] The time complexity of the deletion operation is also O(1). A flowchart of the MPT tree deletion operation is shown below. Figure 5 As shown.

[0133] During deletion operations using an MPT tree, the tree structure is inevitably affected. Specifically, the level of some nodes changes; certain nodes may have their level reduced due to the removal of existing nodes, a phenomenon known as "demotion." The demotion process involves converting a branch node with only one child node into an expanded node or a leaf node, and performing a merge operation on consecutive mergeable expanded nodes. Through the MPT tree's demotion operation, the system can dynamically adjust the overall tree structure.

[0134] After a node is deleted from the MPT tree, the file at the corresponding memory location P indicated by that node's value is also deleted. Simultaneously, the available volume ratio of the memory at location P is updated.

[0135] Step 9: Dynamic inspection of the storage status of the distributed terminal file system.

[0136] Patching is a periodic operation performed by the distributed terminal file system to periodically optimize the various storage devices within it. Following step 4, after traversing all nodes of the MPT tree and all device terminal nodes, information such as used space capacity, remaining space capacity, number of files, and available capacity ratio for each storage device can be calculated. Using the obtained available capacity ratio information for each storage device, if the available capacity ratio of a certain storage device is too high, the available capacity ratio coefficient k can be increased. If the available capacity ratio of a certain storage device is too low, the available capacity ratio coefficient k can be decreased, thus achieving dynamic adjustment of the available capacity ratio coefficient k.

[0137] The dynamic adjustment method is as follows:

[0138] First, calculate the average available capacity of all m memory units.

[0139]

[0140] Update the available volume ratio k for each memory:

[0141]

[0142] Here, i represents the i-th memory, 1≤i≤m, and m represents the total number of distributed memory units. This method dynamically adjusts the available capacity of each memory unit in the distributed terminal file system, allowing subsequently inserted device files to be stored in memory units with higher capacity, thus improving the overall performance of the distributed memory.

[0143] A flowchart of the dynamic inspection process is shown below. Figure 6 As shown.

[0144] The dynamic inspection time complexity of the distributed terminal file system is O(m), where O(m) represents linear time complexity, and the execution time is proportional to the amount of input data. m represents the number of terminals in the distributed terminal file system.

[0145] m represents the number of terminals in the distributed terminal file system.

[0146] Step 10: Evolution of the MPT tree. All terminals can maintain the structure of the entire MPT tree by performing the insertion and deletion operations described above.

[0147] During the evolution of the MPT tree, as new terminal devices are added or existing devices are deleted, the MPT tree structure is dynamically adjusted in steps 7 and 8. The MPT tree maintains its compactness and efficiency, ensuring that query, update, and delete operations are completed in O(1) time. This performance advantage is particularly evident when processing information from large-scale terminal devices, significantly improving operational efficiency and overall system performance. By dynamically adjusting the available capacity in step 9, fragmented files in the distributed memory terminals can be optimized, improving the storage efficiency of the distributed terminal file system and balancing the performance of various distributed memories.

[0148] After following the steps above, a complete MPT tree can be generated, as shown in the following example of key-value pairs and their corresponding MPT tree structure diagram. Figure 2 As shown.

[0149] 57399123573 is a key-value pair where 57399123573 is the key (a key is a combination of the keys of all nodes traversed on the path from the root node to the leaf node) and 57399123573 is the value in the distributed memory.

[0150] 57399761534 is a key-value pair where 57399761534 is the key and 57399761534 is the value in the distributed memory.

[0151] 57399761689 is a key-value pair where 57399761689 is the key and 57399761689 is the value in the distributed memory.

[0152] 57399958573 is a key-value pair where 57399958573 is the key and 57399958573 is the value in the distributed memory.

[0153] Once the location of the file in the distributed file storage is determined, the file can be transferred to the specified storage via protocols such as SSH / FTP.

[0154] The above description and accompanying drawings are specific embodiments of the present invention. All rights and obligations of the present invention shall be determined by the appended claims. Any variations or modifications that can be easily conceived by those skilled in the art are covered within the scope of the claims defined in the present invention.

Claims

1. A distributed terminal file organization method based on MPT tree, characterized in that, Includes the following steps: S1: Construct multiple distributed storage nodes to form a distributed terminal file resource pool for storing terminal device files; S2: Based on the unique device identifier of each terminal device, an MPT tree is constructed using a prefix-based tree index structure. The MPT tree indexes the storage location of the terminal device file in the distributed storage node through its extended nodes, branch nodes, and leaf nodes. S3: When a terminal file needs to be stored, calculate the available capacity ratio of each storage node based on its remaining storage space and the number of files already stored, and select the target storage node in descending order of the available capacity ratio; if the currently selected storage node has insufficient remaining space, exclude the node and select the next storage node in sequence; if all available storage nodes cannot meet the storage requirements, return a storage failure prompt or trigger the system expansion process; The formula for calculating the available floor area ratio V is as follows: k is an adjustable ratio coefficient that balances remaining storage space and the number of files, and its initial value is set to 1. i F represents the remaining storage space of the i-th storage node. i This represents the number of files currently stored on the i-th storage node, 1≤i≤m, where m represents the total number of distributed storage nodes; S4: By traversing the MPT tree structure, perform search, insertion, update, or deletion operations on the terminal device files; wherein... For the insertion operation, the target storage node is found based on S3, the file is inserted, the available volume ratio of the storage node is updated after insertion, and the MPT tree is updated to reflect the storage location of the new file. For deletion operations, after deleting a file, the available volume ratio of the corresponding storage node is updated, and the MPT tree is updated to remove the index of the file; For update operations, first determine whether the size of the updated file exceeds the remaining capacity of the original storage node; if it does not exceed the limit, update the file on the original node and update the MPT tree to maintain the metadata changes; if it exceeds the limit, re-execute the S3 process for the file to select a new target storage node, perform file migration and update, and synchronously update the available capacity and MPT tree of the relevant storage nodes. For search operations, after traversing the MPT tree to locate the file, the result is returned directly without any updates to the MPT tree. S5: Periodically perform a dynamic inspection process, traverse the MPT tree to obtain the system storage status, and dynamically adjust the parameters used to calculate the available volume ratio in S3 based on the obtained storage status; the dynamic adjustment of the parameters used to calculate the available volume ratio in S3 specifically involves dynamically adjusting the proportional coefficient k, including the following sub-steps: Calculate the average available volume ratio of all m storage nodes. Based on the current available capacity V of each storage node i With the average available floor area ratio The ratio is used to update its own proportional coefficient k: Where i represents the i-th storage node, 1≤i≤m, and m represents the total number of distributed storage nodes.

2. The method according to claim 1, characterized in that, Step S2 includes: constructing an MPT tree based on the SN encoding of each terminal device, wherein the SN encoding is used as the key and the position of the corresponding device information in the distributed memory is used as the value; Furthermore, in the MPT tree: The extension node is configured to merge device identifiers with the same prefix and point to the next node; the branch node contains a fixed number of child node pointers to perform branch navigation based on the current character of the device identifier; The leaf nodes are configured to store the specific location information of the terminal device files in the distributed storage nodes.

3. The method according to claim 1, characterized in that, The search operation in step S4 further includes: traversing from the root node of the MPT tree, using the complete device identifier as the initial key, and recursively searching; During the recursion, perform the corresponding operation based on the current node type: If the current node is a branch node, select the corresponding child node path based on the current character of the remaining part of the key and continue the recursive search; If the current node is an extended node, compare the remaining part of the key with the prefix of that extended node: If the remaining key part begins with the prefix of the extended node, then the prefix is ​​removed from the remaining key part, and the remaining key part is recursively searched in the child nodes pointed to by the extended node. If the remaining part of the key does not begin with the prefix of the extended node, the search is terminated and a search failure is returned. When the current node is a leaf node, compare the remaining part of the key with the path of that leaf node: if there is a complete match, the location information of the terminal file stored in that leaf node is successfully obtained. If the match is not complete, the search fails. Through the above process, the target leaf node is eventually located or the search failure is confirmed.

4. The method according to claim 1, characterized in that, The insertion operation in step S4 further includes: Starting from the root node of the MPT tree, using the complete device identifier as the initial key, recursively search for the insertion position; During the recursion, perform the corresponding operation based on the current node type: If the current node is empty, a new leaf node is created, and the location information of the terminal file is stored as the value of the node. When the current node is an extended node and the remaining key part completely matches the prefix of the extended node, remove the matching prefix from the remaining key part and continue to recursively call the insertion operation to process child nodes; When the current node is an extended node and the remaining part of the key does not completely match the prefix of the extended node, a new extended node is created based on the common prefix of the two and it points to a newly created branch node; then, the remaining part of the original extended node after splitting and the remaining part of the new key are mounted as different child nodes of the branch node respectively. When the current node is a branch node, the corresponding child node path is selected based on the current character of the remaining part of the key, and the child node corresponding to the insertion operation is processed recursively. When the current node is a leaf node, compare the remaining part of the key with the path of that leaf node: if they are exactly the same, update the terminal file location information of that leaf node; if they are not exactly the same, create a new branch node and mount the original leaf node and the new leaf node as different child nodes of that branch node. The insertion operation includes a promotion process, which involves converting a leaf node or an extended node into a branch node and mounting the remaining part of the original node's path as a child node of the branch node. When multiple child nodes of a branch node have a common prefix, these child nodes with the common prefix are merged to create an extended node, and the common prefix is ​​used as the path of the extended node to dynamically optimize the tree structure. After successfully inserting a node, the location information of the terminal device file in the distributed storage is stored in that node, and the available volume ratio of the storage corresponding to that location is updated.

5. The method according to claim 1, characterized in that, The deletion operation described in step S4 also includes: Starting from the root node of the MPT tree, recursively search for the node to be deleted, using the complete device identifier as the initial key. During the recursion, perform the corresponding operation based on the current node type: When the current node is a branch node, the corresponding child node path is selected based on the current character of the remaining part of the key and deleted recursively. After the recursion is completed, when the branch node has only one child node left, a downgrading operation is performed to convert it into an extended node or a leaf node. If the current node is an extended node, then determine whether the remaining part of the key begins with the prefix of that extended node: If so, remove the prefix of the extended node from the remaining part of the key and recursively delete its child nodes; If not, the deletion process will terminate. When the current node is a leaf node, compare the remaining part of the key with the path of that leaf node: if there is a complete match, delete the leaf node; The deletion process is terminated when there is no complete match. The deletion operation includes a downgrading process, which refers to converting a branch node with only one child node into an extended node or a leaf node, and performing a merge operation on consecutive mergeable extended nodes. After successfully deleting a node, the corresponding terminal file in the distributed storage is deleted based on the location information stored in that node, and the available capacity of the corresponding storage is updated.

6. The method according to claim 1, characterized in that, After determining the target storage node for the file, transfer the file to that node using SSH or FTP protocols.

7. The method according to claim 1, characterized in that, The distributed storage node includes at least one of a disk array, a network-attached storage device, or a cloud storage service.

Images