A distributed data query method and system for ubiquitous storage

Abstract

This invention discloses a distributed data query method and system for ubiquitous storage. The method comprises the following steps: Step 1, constructing a hierarchical index network model, which includes a metadata management network and a data block index network: a decentralized metadata management network is constructed using a fully connected networking approach; a data block index network is constructed based on a distributed hash table; Step 2, defining a hierarchical data index interaction paradigm to achieve collaborative management and fast querying of metadata and data blocks. This invention, through the hierarchical index network model, decouples metadata from location information, reducing the management and maintenance costs of metadata.

Description

Technical Field

[0001] This invention belongs to the field of distributed data storage technology, specifically relating to a distributed data indexing method and system for ubiquitous storage application scenarios. Background Technology

[0002] Currently, traditional distributed storage solutions, exemplified by data centers, largely rely on servers to provide reliable data storage and stable access services. Their centralized design limits system efficiency and scalability due to server performance constraints. Ubiquitous storage, on the other hand, follows a completely decentralized design philosophy. Users contribute or rent out their idle terminal device resources (storage, computing, or bandwidth) to become working entities within the ubiquitous storage system. This continuous aggregation of end-node resources allows for balanced deployment of business loads across different nodes, resolving system expansion and single-point performance bottlenecks. However, in this model, the large number and highly dispersed nature of storage nodes makes locating data difficult, and this loose node organization cannot guarantee node stability, further exacerbating the difficulty of data location. Therefore, an efficient and robust lookup algorithm resistant to node jitter (offline or location migration) is essential for quickly obtaining data location information.

[0003] In a data center storage model, metadata is used to record the location information of data on the storage medium. High-performance server nodes are configured to manage and maintain this metadata information, thereby handling massive data query requests. When the data volume or data access volume increases, it is necessary to upgrade or replace servers to meet the data query demands. This method is often extremely costly to implement and highly dependent on the server's data processing capabilities, which violates the decentralized design principle and is therefore unsuitable for ubiquitous storage applications.

[0004] Another way to improve data query efficiency is through indexing mechanisms to accelerate data retrieval, with typical applications including IPFS and Storj. However, existing indexing algorithms are mainly suitable for situations where storage nodes are relatively stable, while the stability of nodes in ubiquitous storage systems is highly susceptible to network fluctuations and user behavior. In such environments, the efficiency and accuracy of indexing algorithms will drop sharply.

[0005] Based on the above problems, it is necessary to design an efficient data indexing and management method for ubiquitous storage systems, which can effectively reduce the impact of node jitter on data query while achieving efficient data query and maintaining data query efficiency and accuracy. Summary of the Invention

[0006] This invention provides a distributed data indexing and management model and method for ubiquitous storage applications. By layering the data index network structure, it improves the efficiency and accuracy of data retrieval. Furthermore, based on existing indexing algorithms, it proposes an adaptive node jitter adjustment method to reduce the impact of node jitter on the data index.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A distributed data query method for ubiquitous storage, comprising the following steps:

[0009] Step 1: Construct a hierarchical index network model, which includes a metadata management network and a data block index network.

[0010] A decentralized metadata management network is built using a fully connected networking approach;

[0011] Based on the design pattern of Distributed Hash Table (DHT), a data block index network is built to achieve fast location of data blocks.

[0012] Step 2: Construct a hierarchical data indexing interaction paradigm to achieve collaborative management and fast querying of metadata and data blocks.

[0013] Preferably, after step 2, step 3 is also included, in which a node availability probability prediction and index network adjustment method is adopted to establish a node jitter prediction model based on the heavy-tailed Pareto distribution, and the index network is adjusted according to the severity of the jitter.

[0014] Preferably, in step 1, the query network is logically divided into: a metadata management network and a data block index network;

[0015] The metadata management network is constructed in the following way: the metadata management nodes are networked in a fully connected manner, and their core task is to handle a large number of client query requests and respond quickly; at the same time, they update and maintain metadata information according to data and changes in storage nodes.

[0016] The data block index network is constructed in the following way: the data block index nodes are designed with reference to the distributed hash table pattern and are connected point-to-point to build the index network topology. Their main task is to record and maintain the location information of the data block storage nodes; at the same time, they assist in managing the stored data and monitoring and updating the status of the storage nodes.

[0017] The purpose of introducing a data block index network in this invention is to decouple metadata from storage node location information, thereby reducing the frequent modification and writing of metadata due to changes in node location.

[0018] Preferably, in step 1, the construction steps of the metadata management network are as follows:

[0019] 1) The metadata management node consists of nodes with high stability, high bandwidth, and strong data processing capabilities, networked using a fully connected architecture. Each metadata management node records and maintains only a portion of the metadata information, and the average data load of the node is:

[0020] Where E is the average data load, w i The amount of data recorded by the i-th node at a certain time point, where n is the number of nodes.

[0021] 2) The metadata management node obtains its own identity using the hash algorithm h1(*), and simultaneously maintains a metadata management node registry to locate nodes and their metadata. The table records the node's identity (NodeID) and routing information in ascending order of their XOR distance from other nodes. The calculation formula can be expressed as:

[0022] dist(node ​​x,node y)=[log2 XOR(id(x),id(y))]+1.

[0023] Where dist is the relative distance (XOR distance) between nodes, x and y represent two different nodes, and id(x) and id(y) represent the node identity identifiers obtained after calculation, which are generally large random numbers.

[0024] 3) Similar to the identity identifier of the computing nodes, the storage files are uniformly addressed using the hash algorithm h1(*). Following the proximity principle, the file's metadata information is stored at the XOR distance (i.e., ...). The nearest nodes are selected, where x and y represent the node identifier and metadata identifier, respectively. Based on this storage model, the file's metadata information will be randomly stored on different nodes, thus achieving data load balancing.

[0025] Metadata information<fid,metadata> It is stored in key-value pair format. The fid serves as a globally unique file identifier; the metadata records specific summary information of the file, including filename, author, creation time, and data block information. In this invention, the data block information only includes the data block identifier (bid), and is no longer associated with the actual storage location of the data, thereby reducing node management and maintenance costs and improving query efficiency.

[0026] As a preferred embodiment, the data block index network construction steps in step 1 are as follows:

[0027] 1) In this invention, the data block index network is networked with reference to the data structure of the Kademlia algorithm, ultimately forming a tree-shaped network topology. The index nodes and data block identifiers (bid) are uniformly addressed using the same hash algorithm h2(*). The node and data block location information act as the leaves of the tree. The location of each node is uniquely determined by the hash value of its identifier, and neighbor relationships are established between nodes based on their relative distance.

[0028] In actual operation, based on the decentralized design principle, each node only stores a portion of the index data, and the location index of the data is stored in the nearest node according to the proximity principle. Therefore, the essence of index building can be abstracted as a process of continuously approaching and eventually converging to the index node with the smallest relative distance. During the convergence process, data redundancy backup and load balancing can be achieved.

[0029] 2) In this invention, the index table adopts a non-clustered indexing pattern, meaning that nodes no longer store specific data, but rather record the data's storage location information. Therefore, the key-value pair data structure is as follows:<id,(ip,port)> , where id represents the identification information of the node or data block, and (ip, port) corresponds to the network location and interaction interface of the index node or data storage node.

[0030] The id, as an identifier, corresponds to two different types of objects: one is bid, which corresponds to the data block identifier; the other is nid, which corresponds to the inode identifier. Both are stored in the k-bucket data structure of the inode. Therefore, from the perspective of indexing, the inode and the data block are equivalent. (ip, port) corresponds to the network location and service identifier of the node. The port is used to distinguish whether the object is a storage node or an inode.

[0031] Preferably, in step 2, a complete file query process is divided into: metadata query and data block index location.

[0032] During the metadata query process, the user first establishes a communication connection with the nearest metadata management node and passes the file identifier (fid) of the target file to that node. Then, the node performs an XOR operation between the file identifier and the node identifier to obtain the relative distance (dist). Finally, the node finds the metadata management node with the smallest relative distance by querying the metadata management node registry, forwards the request message to that node, and finally returns the response result to the user through a proxy.

[0033] During the data block indexing process, the metadata is first unpacked to obtain the data block identifier (bid). Then, the user establishes a connection with the nearest index node and sends a data block query request. Upon receiving the request, the index node determines whether the target data block location information is stored in its own index table. If it is, the location information of the data block is returned directly. Otherwise, the node queries the node that is closer to the bid through a proxy and redirects the user's index request to that node. Through continuous iteration, the process continues until the node closest to the bid is found and the data is obtained, or until the target data index information cannot be found, and a query failure message is returned.

[0034] As a preferred option, step 3 is as follows:

[0035] 1) Node jitter prediction, which includes two parts: the establishment of the node jitter prediction model and the correction of the model;

[0036] Establishment of node jitter prediction model: Based on the behavioral characteristics and statistical laws of mobile nodes, a jitter prediction algorithm is used to model the access loss caused by node offline or network fluctuations in the storage network, and at the same time, a quantitative standard for the degree of node jitter is given.

[0037] Model revision: The model is revised based on historical network state data and node behavior patterns.

[0038] 2) Based on the node jitter prediction model and actual measurement results, dynamically adjust the data redundancy and access concurrency of the data block index network under different jitter conditions, thereby improving indexing efficiency and hit rate and accelerating data updates;

[0039] 3) In order to repair invalid data caused by nodes that have been offline for a long time or have been permanently disconnected from the storage network, it is necessary to adjust the update frequency of index entries according to the node failure probability. By adjusting the data access frequency and query concurrency, the update of invalid index entries can be accelerated.

[0040] This invention also discloses a system based on the above-mentioned distributed data query method for ubiquitous storage, which includes the following modules:

[0041] Hierarchical Index Network Model Construction Module: The hierarchical index network model includes a metadata management network and a data block index network. It constructs a decentralized metadata management network through a fully connected networking approach and builds a data block index network based on a distributed hash table.

[0042] Metadata and data block query module: Constructs a hierarchical data index interaction paradigm to achieve collaborative management and fast querying of metadata and data blocks.

[0043] Compared with existing indexing technologies, the significant advantages of this invention are:

[0044] This invention achieves decoupling of metadata and location information through a hierarchical index network model, reducing the management and maintenance costs of metadata. Furthermore, the fully connected networking of metadata nodes improves the query efficiency of metadata. A data block index layer based on the DHT algorithm fully utilizes end-node resources while minimizing the impact of node instability on index efficiency and accuracy, thus improving system scalability. The index adjustment method proposed in this invention effectively alleviates the performance degradation of the index network caused by node jitter, and can promptly update failed nodes and data, improving system availability. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the index network structure;

[0046] Figure 2 This is a diagram of the metadata management node registry;

[0047] Figure 3 This is a diagram illustrating data interaction;

[0048] Figure 4 This is a flowchart of the file search process;

[0049] Figure 5 This is a flowchart of node jitter adjustment;

[0050] Figure 6 This is a block diagram of the distributed data query system for ubiquitous storage according to the present invention. Detailed Implementation

[0051] The technical solution of the present invention will be further described below with reference to the accompanying drawings and preferred embodiments.

[0052] To better understand the methods proposed in this embodiment, a complete file query and the index adjustment and repair process under different node jitter conditions are selected for description. The specific steps of this embodiment are as follows:

[0053] A distributed data query method for ubiquitous storage, comprising the following steps:

[0054] S1. Hierarchical Index Network Construction

[0055] like Figure 1 As shown in this embodiment, the node members in the model can be divided into three roles according to their functions: Metadata Management Node (MN), Data Index Node (IN), and Data Storage Node (DN). Figure 1 The text illustrates the location and function of different functional nodes in the index network.

[0056] S11. Based on the scale of stored data, select several nodes to act as MN nodes to form a metadata management network, and select several IN nodes to form a data block index network.

[0057] S12. In this embodiment, nodes with good stability and high performance need to be selected as MN nodes to ensure the stability of connections between nodes and efficient processing of access requests. The MN nodes form a metadata management network through a fully connected networking method. Each node maintains some metadata information, and together they form the metadata management network. See S1-12 for detailed construction steps.

[0058] S13. Further, nodes with better stability and larger communication bandwidth are selected as IN nodes to ensure the efficiency of data query. IN nodes establish neighbor relationships with other IN nodes that are relatively close through point-to-point connections. Each node stores a portion of the index data, which together form the data block index network. See S1-13 for detailed construction steps.

[0059] S1-12. Construction of Metadata Management Network Model:

[0060] 1) such as Figure 1 As shown, the metadata management nodes construct the management network in a fully connected manner. Nodes maintain neighbor relationships through data interaction. Each node records the network identifiers and routing information of other nodes. Therefore, any metadata query operation can be modeled as a 1-hop reachable communication process. In this mode, the query service can complete the query by accessing at most two nodes, effectively reducing communication latency and achieving fast response for metadata queries.

[0061] 2) During the system initialization phase, the metadata management node generates its own identity identifier NodeID = h1(node) using the hash function h1(*) based on the node's attribute characteristics (e.g., MAC address, network access license number, etc.). Subsequently, each node calculates the relative distance from its neighboring nodes to itself. The nodes are then sorted from nearest to farthest based on relative distance, thereby constructing a metadata management node registry, such as... Figure 2 As shown. The ultimate goal is to accelerate node localization.

[0062] 3) When storing file metadata, firstly, a hash algorithm h1(*) is used to uniformly address the nodes and stored files, obtaining the node identifier NodeID and the file identifier fid respectively. Then, to achieve fast indexing, the metadata management node compares the fid with the NodeID in the metadata management node's registry, calculating the relative distance between fid and NodeID, i.e.: Ultimately, the metadata will be stored on the management node that is closest to it.

[0063] It should be noted that since the values ​​of the metadata identifiers are completely random, the metadata is essentially stored in a random form on the nearest node, thus ensuring load balancing among system nodes.

[0064] S1-13. Construction of the data block index network model:

[0065] 1) such as Figure 1 As shown, the logical structure of the data block index network follows the Kademlia algorithm, forming a binary tree. Node identifiers (nid) and data block identifiers (bid) are uniformly addressed using the hash function h2(*), forming the leaf nodes of the binary tree. Nodes are located using nid and establish neighbor relationships with some nearby nodes based on proximity. Instead of periodically monitoring the status of neighboring nodes through heartbeat messages, nodes establish weak neighbor relationships and update node states through "identity declaration" and "access interaction." The specific implementation steps are as follows:

[0066] When the system initializes or a new node is added, it broadcasts its node identifier (nid) along with network identifier information (IP, port) to other nodes through "identity declaration". The node that receives the broadcast message first calculates the distance with the sender node's nid, adds it to the index list of the corresponding k-bucket according to the relative distance between the nodes, and sends its own nid back to the sending node.

[0067] Furthermore, when nodes are running, neighbor relationships between nodes are maintained and updated through "access interaction." That is, each time a complete message exchange occurs between nodes, both nodes synchronously update the status information of their neighbor lists and advance the identifier of the interacting node from its corresponding k-bucket. Therefore, nodes with higher interaction frequencies require less time to be located. This mechanism allows hot data to be accessed more quickly and gradually eliminates cold data or information from failed nodes, freeing up storage space.

[0068] After the above steps, the index node will fill the network location information of the neighboring nodes into the corresponding k-bucket structure according to the relative distance, thereby constructing a complete node routing information table.

[0069] 2) Index information is adopted<key,value> Stored in key-value pair format, the key-value pair data structure is as follows<id,(ip,port)> Since any node in the index network only stores partial index information, new index data needs to locate its own storage node before it can be stored.

[0070] The location process can originate from any node in the index network. It involves continuously searching for the index node with the smallest relative distance to the ID, based on the neighbor node identifiers and routing information recorded by the node, until the node with the smallest relative distance is located. During this continuous interaction with nodes and eventual convergence, the ID needs to be constantly compared with the identifier information (nid) of the interacting node, and requests need to be made to add the index information to the corresponding k-bucket table entry. This achieves redundancy in the index message, thereby ensuring its availability.

[0071] S2 File Data Query Process

[0072] S21 Metadata Acquisition

[0073] like Figure 3 and 4 As shown, assuming a user needs to retrieve a file with fid=δ, they first need to establish a connection with any neighboring metadata node (MN) and send a Get(fid) metadata request message. Then, the metadata index locates the data storage node through fid, queries through a proxy to ultimately retrieve the metadata information, and sends back a response result of (fid, Metadata). Taking the query operation on node NodeID=n as an example, the specific steps are as follows:

[0074] First, calculate the XOR distance between δ and n. If d is less than the distance from δ to any neighboring node, then it is the metadata storage node being searched. The database is queried and the result is returned. Otherwise, the nearest management node to δ is found by querying the node registry.

[0075] Locate the target node in the node registry and forward the lookup request to that node. If the node contains metadata identified as δ, return the metadata information; otherwise, return a message indicating that the target data is missing to the client.

[0076] S22 Data Block Index

[0077] In this embodiment, the data block query adopts a concurrent access method, sending a Get_DataNode_Route(bid) request to obtain the data block location information: for each query, the query status is either... This indicates that the query failed; alternatively, it retrieves feedback information from α neighboring nodes simultaneously. Therefore, the query status can be represented as:

[0078]

[0079] The indexing process is further refined as follows:

[0080] The first step, after the client obtains the metadata, is to unpack and analyze the data packets to obtain the data block identification information of the target file, namely: (bid1, bid2, ..., bidn).

[0081] The second step involves the client accessing a nearby index node, which acts as a proxy node to help the client obtain the location information of the data block storage node, further refining the process:

[0082] First, the proxy node compares the bid with its own nid to obtain the relative distance. Based on the logical distance, it queries the data item in the corresponding interval of its own index table. If a corresponding item is found, it returns the location information of the storage node; otherwise, it needs to request data from a node that is logically closer to the bid.

[0083] When a proxy node needs to request data from neighboring nodes, it first requests the data using the FIND_VALUE instruction. If none of the neighboring nodes have the location information of the target data block, it then requests information about a node closer to the target data using the FIND_NODE instruction and redirects the user connection to the queried node.

[0084] The communication complexity of the entire query process is O(n). Where N is the number of index nodes. According to theoretical analysis and experimental verification, the larger the network size, the more significant the effect of accelerating data query through indexing methods.

[0085] Subsequently, after the proxy node obtains the data block location information, it returns the result (bid, (ip, port)) to the client, which then sends a Get_DataBlock(bid, (ip, port)) request message to the storage node to obtain the data block information (bid, data block).

[0086] Finally, once the minimum number of data blocks required to restore the file is determined, the entire query process is complete. For data blocks whose location information cannot be obtained, the default assumption is that they are lost or the storage node is offline. The index node needs to report the results to the metadata management node to assist in updating the metadata information.

[0087] S3. Node Jitter Prediction and Processing

[0088] like Figure 5 As shown, to address the impact of node jitter on data indexing, improvements are made to the indexing network layer. The improvement steps include:

[0089] 1) Model the "offline" state of nodes by predicting node jitter, and assist the index network in adaptive adjustment to ensure system stability;

[0090] 2) Based on the node jitter model, an adaptive index optimization method is proposed to adapt to it. By adjusting the index entry update rate, entry size, index node trust assessment and aging time, etc., the index efficiency and accuracy are stabilized under different node jitter conditions.

[0091] S31. Node jitter prediction model

[0092] The system's determination of node shake rate (NSR) relies on statistical modeling of terminal device mobility patterns. This embodiment selects the mobility patterns of P2P nodes within a typical city over time. Therefore, assuming that the accessibility probability of nodes in the system follows a heavy-tailed Pareto distribution with parameters γ and β, the accessibility probability of any node is:

[0093]

[0094] Among them, t α Let T be a random variable representing the node's accessibility time, where T is a certain time length, and γ and β represent the shape and scale parameters of the distribution, respectively. The expression represents the probability that the node's accessibility time is less than T. Simultaneously, T is introduced during the modeling process. last T represents the time elapsed since the last interaction between a neighboring node and the observation. online This represents the time elapsed since a neighboring node came online until its most recent interaction. The current availability probability of any node, p, can be calculated and expressed as:

[0095]

[0096] By transforming the above equation, we get:

[0097]

[0098] In the formula T online With t online They all follow a Pareto distribution with heavy tails, where the value is When T online When determined, T last It is only related to the probability p of a node being online. Therefore, in Within a given timeframe, the probability that 50% of the neighboring nodes will be online is less than P. min That is, the probability that the index entries stored in the system's inodes are available is less than P. min .

[0099] Since node jitter in ubiquitous storage network scenarios is affected by a variety of factors, the availability probability of various nodes (excluding malicious nodes with attack intentions) often has relatively clear time characteristics and behavioral patterns. The most typical example is the "tidal pattern" of network links. In order to more accurately reflect the jitter status of nodes, it is necessary to measure and correct the accessibility probability of nodes in real time.

[0100] S32. Index Adjustment Method

[0101] like Figure 5 As shown, the index adjustment method in this embodiment operates on the data block index network. Based on the k-bucket mechanism of the Kademlia algorithm and the concurrent query form, it adaptively adjusts the original parameters according to the node jitter prediction results and optimizes some mechanisms in the original algorithm. The specific steps are as follows:

[0102] First, in order to ensure the availability of indexed data, i.e., to maintain P min For stability, nodes must continuously update neighbor information, backing up and updating those with an availability probability less than P. min The index entries. To achieve the desired goal, the node needs to... last Within a given timeframe, 50% of the entries in the index table are repaired. A parameter C is introduced, representing the number of new neighbor nodes a node needs to interact with per unit of time, reflecting the frequency of node interactions in the system. Substitution We can obtain:

[0103]

[0104] Setting the parameter γ = 1 indicates that the node availability probability changes linearly, and we can deduce that the average availability probability of the index entries is:

[0105]

[0106] At this point, the availability probability of a node is determined by β and K, while parameter C represents the node's ability to update routing data. The Kademlia algorithm uses an explicit update method to verify the node's liveness status, and the node's data update capability is directly related to the access concurrency α per unit time.

[0107] The relationship between the availability probability of data items in the index table and the k-bucket capacity K is shown. It can be seen that when βC is constant, the availability probability of data items in the index table decreases continuously as K increases. This indicates that, under the condition of constant node jitter and data updates, the larger the routing table, the more difficult it is to maintain. When K is constant, an increase in βC indicates a decrease in node jitter or an improvement in data update capability, and the availability probability of data in the index table increases accordingly.

[0108] Therefore, when the probability of node availability decreases, increasing the k-bucket capacity can increase the redundancy of index data, thereby improving index accuracy. Conversely, increasing the access concurrency α of index nodes can increase the hit rate of data queries, thereby improving the overall index efficiency.

[0109] S33. Index Entry Update

[0110] Based on node availability probability, jitter is categorized into three levels: slight jitter, moderate jitter, and severe jitter. The update frequency is adjusted according to the different jitter levels, and the update results are fed back to the auxiliary metadata management node for metadata updates.

[0111] 1) When the node jitter level is "slight jitter", the system does not take any action and updates the data itself based on the Kademlia algorithm's own update and repair mechanism.

[0112] 2) When the node jitter level reaches "normal jitter", gradually increase the single access concurrency α and access frequency t, with an update cycle of T. Before each adjustment cycle begins, the system node jitter situation needs to be reassessed: if node jitter continues, follow the principle of gradient descent to accelerate the update frequency; if node jitter eases, gradually slow down the update frequency and reduce access concurrency and access frequency until a stable level is restored.

[0113] 3) If the node jitter level reaches “severe jitter”, the adjustment mode is similar to that when the jitter level is “normal”. However, in order to avoid drastic system fluctuations, the update of failed nodes and data should be accelerated. At this time, the adjustment cycle will be continuously reduced at a fixed rate, and the access concurrency α and access frequency will be continuously increased according to the principle of gradient increase until the node is out of the state of “severe jitter”.

[0114] Maintaining high concurrency and frequency of access consumes significant link bandwidth and increases computational overhead. Therefore, as node jitter decreases, update frequency and concurrency are gradually reduced to conserve system resources.

[0115] The core of the index tuning method is based on the continuous monitoring and adjustment of the node jitter prediction model, namely "prediction and monitoring - adjustment and repair - monitoring". At the same time, in order to balance system performance and overhead, the relevant parameters of the indexing algorithm can be gradually backtracked when the node state tends to stabilize, avoiding additional overhead.

[0116] like Figure 6 As shown, this embodiment of the invention also provides a system for a distributed data query method for ubiquitous storage based on the above embodiments, which includes the following modules:

[0117] Hierarchical Index Network Model Construction Module: The hierarchical index network model includes a metadata management network and a data block index network. It constructs a decentralized metadata management network through a fully connected networking approach and builds a data block index network based on a distributed hash table.

[0118] Metadata and data block query module: Constructs a hierarchical data index interaction paradigm to achieve collaborative management and fast querying of metadata and data blocks;

[0119] Node jitter prediction and processing module: It adopts the node availability probability prediction and index network adjustment method, establishes a node jitter prediction model based on the heavy-tailed Pareto distribution, and adjusts the index network according to the severity of jitter.

[0120] The above embodiments are based on the technical solution of the present invention and provide detailed implementation steps. The accompanying drawings are only one embodiment of the present invention and can be used as a reference for related application scenarios. The scope of protection of the present invention is not limited to the above embodiments.

Claims

1. A distributed data query method for ubiquitous storage, characterized by: Follow these steps: Step 1: Construct a hierarchical index network model, which includes a metadata management network and a data block index network: a decentralized metadata management network is constructed through a fully connected networking approach. A data block index network is constructed based on a distributed hash table; Step 2: Construct a hierarchical data indexing interaction paradigm to achieve collaborative management and fast querying of metadata and data blocks; In step 1: the metadata management network is constructed in the following way: the metadata management nodes are networked in a fully connected manner to process client query requests; at the same time, the metadata information is updated and maintained according to the status changes of data and storage nodes; In step 1: the data block index network is constructed in the following way: the data block index nodes refer to the distributed hash table and construct the index network topology based on point-to-point connections; at the same time, it assists in managing and storing data, and monitors and updates the status of storage nodes; The specific steps for building the data block index network in step 1 are as follows: 1) The data block index network is constructed using the data structure of the Kademlia algorithm, ultimately forming a tree-like network topology; where index nodes and data block identifiers... Using hash algorithm Unified addressing is used, with node and data block location information acting as leaves of a tree. The location of each node is uniquely determined by the identifier hash value, and nodes establish neighbor relationships based on their relative distance to each other. 2) The index table uses a non-clustered indexing pattern, meaning that nodes do not store actual data, but rather record the data's storage location information; therefore, the key-value pair data structure is as follows: ,in Identification information representing nodes or data blocks. The network location and interaction interface corresponding to the index node or data storage node; In step 2, the metadata query process, the user first establishes a communication connection with the nearest metadata management node and sets the file identifier of the target file. The file identifier is passed to the node; subsequently, the node performs an XOR operation between the file identifier and the node identifier to obtain the relative distance. Finally, the node finds the metadata management node with the smallest relative distance by querying the metadata management node registry, forwards the request message to that node, and finally returns the response result to the user through a proxy. During the data block indexing process, the metadata is first unpacked to obtain the data block identifier. The user then establishes a connection with a nearby index node and sends a data block query request. Upon receiving the request, the index node determines whether the target data block location information is stored in its own index table. If it is, the node directly returns the data block location information; otherwise, it queries the index node through a proxy. The system iterates through nodes that are relatively closer to the user and redirects the user's indexing request to that node; it continues this process until a node that is relatively closer to the user is found. If the nearest node is found, the data is retrieved; otherwise, a query failure message is returned. Following step 2, step 3 is also included: using the node availability probability prediction and index network adjustment method, a node jitter prediction model is established based on the heavy-tailed Pareto distribution, and adjustments are made accordingly based on the severity of the jitter.

2. The distributed data query method for ubiquitous storage according to claim 1, characterized in that, In step 1, the specific steps for building the metadata management network are as follows: 1) The metadata management node consists of multiple nodes networked together in a fully connected manner; each metadata management node records and maintains a portion of metadata information, and the average data load of the node is: ; in, For average data load, For a certain point in time, the first The amount of data recorded by each node The number of nodes; 2) The metadata management node uses a hash algorithm. The system obtains its own identity identifier and simultaneously locates nodes and their metadata by maintaining a metadata management node registry. The table records node identity identifiers sequentially from closest to furthest from other nodes, based on their XOR distance from the current node. And routing information, the calculation formula is expressed as: in, This represents the relative distance between nodes, i.e., the XOR distance. , They represent two different nodes. , This represents the node identity identifier obtained after calculation. 3) Same as the identity identifier of the computing node, using a hash algorithm. Files are uniformly addressed; according to the principle of proximity, file metadata information is stored at the XOR distance. At the nearest nodes, among them , These represent the node identifier value and the metadata identifier value, respectively.

3. The distributed data query method for ubiquitous storage according to claim 1, characterized in that, Step 3 is as follows: 1) Node jitter prediction, including: Establishment of node jitter prediction model: Based on the behavioral characteristics and statistical laws of mobile nodes, a jitter prediction algorithm is used to model the access loss caused by node offline or network fluctuations in the storage network, and at the same time, a quantitative standard for the degree of node jitter is given. Model refinement: The model is refined based on historical network state data and node behavior patterns; 2) Based on the node jitter prediction model and actual measurement results, dynamically adjust the data redundancy and access concurrency of the data block index network under different jitter conditions; 3) Adjust the update frequency of index entries based on the node failure probability. By adjusting the data access frequency and query concurrency, the update of failed index entries can be accelerated.

4. A system based on the distributed data query method for ubiquitous storage as described in any one of claims 1-3, characterized in that: Includes the following modules: Hierarchical Index Network Model Construction Module: The hierarchical index network model includes a metadata management network and a data block index network. It constructs a decentralized metadata management network through a fully connected networking approach and builds a data block index network based on a distributed hash table. Metadata and data block query module: Constructs a hierarchical data index interaction paradigm to achieve collaborative management and fast querying of metadata and data blocks.

Images