A blockchain-based distributed file management and control method and system

Abstract

The application relates to a kind of distributed file management and control method and system based on blockchain, belong to the technical field of distributed file storage, solve the low security and unreasonable storage of the existing distributed system file.Problems. Including using clustering algorithm to divide each node of the distributed file system into multiple local areas;When requesting file operation, locate the target area from multiple local areas according to the hash value of the file content, the master node of the target area constructs a request message, and submits it to the blockchain through the smart contract;The master node and the copy node of the blockchain perform multi-level verification on the request message, authorize the master node of the target area to execute the file operation through the smart contract;The master node and the slave node of the target area reach a consensus on the confirmation operation, and execute the file operation in the target area, construct the operation completion message, and submit it to the blockchain through the smart contract;The blockchain writes the file operation into the file operation log. Realize the safe and reasonable management and control of distributed files.

Description

Technical Field

[0001] This invention relates to the field of distributed file storage technology, and in particular to a distributed file management method and system based on blockchain. Background Technology

[0002] The widespread adoption of the internet and the application of emerging technologies such as the Internet of Things, big data, and artificial intelligence have led to an exponential growth in data. Distributed file storage, by distributing data across multiple nodes, can fully utilize the storage resources of each node in the cluster, achieving elastic expansion of storage capacity and meeting the demand for massive data storage in the era of big data.

[0003] Traditional distributed file systems centralize file operation permission verification on a single server or node, which presents security and reliability issues and makes it difficult to record and audit the entire operation process. Once a file is tampered with or deleted, the system lacks an effective recording and authentication mechanism, making it difficult to trace the responsible party or verify the legitimacy of the operation.

[0004] Moreover, existing distributed file systems lack effective mechanisms to prevent data from being illegally manipulated, and the risk of file data being maliciously erased or tampered with is high; since data storage nodes are distributed in different regions, data consistency verification and cross-regional data synchronization face significant challenges. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a blockchain-based distributed file management method and system to solve the problems of low security and unreasonable storage of files in existing distributed systems.

[0006] On one hand, embodiments of the present invention provide a blockchain-based distributed file management method, comprising the following steps:

[0007] Clustering algorithms are used to divide the nodes of a distributed file system into multiple local regions;

[0008] When requesting a file operation, the target region is located from multiple local regions based on the hash value of the file content. The master node of the target region constructs a request message based on the file operation and submits it to the blockchain through a smart contract.

[0009] The blockchain's master node and replica nodes perform multi-level verification of request messages. When a consensus is reached that the verification is successful, the master node in the target area is authorized to execute file operations through a smart contract.

[0010] After the master node and slave node in the target area reach a consensus on the confirmation operation, they execute the file operation in the target area and construct an operation completion message to submit to the blockchain through a smart contract; the blockchain writes the file operation into the file operation log.

[0011] Based on the further improvement of the above method, the nodes of the non-target region periodically check whether the file operation time in the log stored by itself is consistent with the corresponding time in the file operation log of the blockchain. If not, the data operation is performed according to the inconsistent file operation type and the log is updated, or a data synchronization request is sent to the master node of the target region recorded in the file operation log, and the data synchronization operation is completed according to the received latest file copy and the log is updated.

[0012] Based on the further improvement of the above method, the nodes of the distributed file system are divided into multiple local regions by using a clustering algorithm, including:

[0013] The network delay of each node to the preset multiple reference nodes is calculated and standardized to obtain a network delay vector; the CIDR network segment identifier of each node is obtained according to the IP address and is one-hot encoded to obtain a CIDR vector; and the feature vector of each node is obtained according to the delay vector and the CIDR vector of each node.

[0014] The multiple initial centroids are selected according to the network load, local density and relative distance of each node; and the multiple local regions are clustered by using the K-Means clustering algorithm according to the multiple initial centroids and the feature vector of each node.

[0015] Based on the further improvement of the above method, the multiple initial centroids are selected according to the network load, local density and relative distance of each node, including:

[0016] The network load includes CPU utilization, memory utilization and bandwidth utilization; the three utilization rates of the network load of each node are calculated, and the nodes with all the three utilization rates less than the corresponding threshold are put into a candidate node set.

[0017] The local density of each candidate node in the candidate node set is calculated, the distance between it and all the candidate nodes with higher local density than it is calculated, and the maximum distance is taken as the relative distance of the candidate node.

[0018] The local density and the relative distance are sorted in descending order respectively, the product of the two sorting values of each candidate node is calculated to obtain a comprehensive value, and then the candidate nodes are sorted in descending order according to the comprehensive value, and the first K candidate nodes are taken as the initial centroids.

[0019] Based on the further improvement of the above method, the target region is located from the multiple local regions according to the hash value of the file content, including:

[0020] The weight of each local region is obtained according to the number of nodes in each local region and the storage total capacity proportion; and the virtual node number of each local region is obtained according to the weight of each local region.

[0021] According to the identification of each virtual node in each local area, the hash value of the virtual node is obtained and sorted to form a hash ring;

[0022] On the hash ring, a first virtual node greater than the hash value of the file content is found clockwise, and the local area to which the virtual node belongs is taken as the target area.

[0023] Based on the further improvement of the above method, the master node of the target area is the node with the highest bandwidth in the target area; the master node of the blockchain is obtained by calculating the credit value of each node in the blockchain and normalizing it as a weight, using a roulette algorithm, and selecting based on the weight of each node and the random number generated by VRF; the credit value of each node is obtained by calculating the online rate, task completion rate and resource availability rate of each node and weighted sum.

[0024] Based on the further improvement of the above method, the master node and the replica node of the blockchain perform multi-level verification on the request message, including:

[0025] The master node of the blockchain performs primary verification on the request message, and if the primary verification is passed, the master node of the blockchain broadcasts a pre-preparation message to all replica nodes, and the pre-preparation message includes the request message and a mark indicating that the primary verification is passed;

[0026] After each replica node receives the pre-preparation message, it performs secondary verification on the request message, and if the secondary verification is passed, it returns a preparation message to the master node; when the number of preparation messages received by the master node exceeds a first threshold, it enters a submission stage and broadcasts a submission message to all replica nodes;

[0027] After each replica node receives the submission message, if it agrees to execute the request message, it broadcasts a submission message; when the number of nodes broadcasting the submission message exceeds the first threshold, a consensus of passing the verification is reached.

[0028] Based on the further improvement of the above method, the primary verification is to obtain the target file and the master node of the target area from the request message, and to verify whether the master node of the target area has the permission to operate the target file; the secondary verification includes verifying whether the master node of the target area has the permission to operate the target file, and reviewing whether the historical operation record of the target file is normal.

[0029] Based on the further improvement of the above method, the master node of the target area and the slave node reach a consensus on the confirmation operation, and then execute the file operation in the target area, including:

[0030] The master node of the target area writes the file operation into a log, and synchronizes the new log entry to all slave nodes in the target area; all slave nodes compare their own historical log records with the file operation log of the blockchain, and if they are consistent, they write the new log entry and return a confirmation message to the master node of the target area;

[0031] When the number of the confirmation messages received by the master node of the target region exceeds the second threshold value, the master node and the slave node of the target region reach a consensus on the confirmation operation, and the master node and the slave node of the target region perform the file operation.

[0032] In another aspect, the embodiment of the present application provides a distributed file management and control system based on a blockchain, comprising:

[0033] A local region division module is configured to divide nodes of a distributed file system into a plurality of local regions by using a clustering algorithm.

[0034] A file operation request module is configured to locate a target region from the plurality of local regions according to a hash value of file content when a file operation is requested, and to submit a request message constructed by the master node of the target region according to the file operation to the blockchain through a smart contract.

[0035] A file operation approval module is configured to perform multi-level verification on the request message by the master node and the replica node of the blockchain, and to authorize the master node of the target region to perform the file operation through the smart contract when a consensus on the verification is reached.

[0036] A file operation execution module is configured to perform the file operation in the target region after the master node and the slave node of the target region reach a consensus on the confirmation operation, and to submit an operation completion message to the blockchain through the smart contract, and the blockchain writes the file operation into a file operation log.

[0037] Compared with the prior art, the present application can achieve at least one of the following beneficial effects:

[0038] 1. The file operation is recorded and protected by the dual mechanism of the blockchain and the smart contract, the legality and consistency of the operation request are ensured by the consensus mechanism, the malicious erasure protection is realized, and the anti-risk ability when responding to malicious operations or unauthorized behaviors is improved.

[0039] 2. The feature vector of each node is constructed according to the physical location and network delay of each node in the distributed system, the clustering algorithm is improved, a plurality of initial centroids are selected according to the network load, local density and relative distance of each node, different regions of the entire node space are covered, the performance and accuracy of the clustering algorithm are improved, the distributed file storage and management are optimized by dynamically dividing the nodes into a plurality of local regions.

[0040] 3. The weight is dynamically calculated based on the actual storage capacity and the number of nodes in the local region, and the weight is integrated into the file storage decision, the problem of uneven distribution of files among the distributed storage regions is effectively solved, and the load balancing, resource utilization and overall stability of the distributed file system are significantly improved.

[0041] 4、Through the timing calculation of the credit value of each node in the blockchain, the rotation mechanism of the main node of the blockchain is realized by using the roulette algorithm, and in the consensus process of the file operation request, the safety and reliability of the file operation are improved through multi-level verification; through the automatic operation authorization and state feedback of the smart contract, the efficiency is improved, and the distributed environment of large-scale nodes is adapted;

[0042] 5、Considering the joining and exiting of nodes in the distributed file system, a cross-region data synchronization mechanism is provided, and when inconsistent data is found, fast response and automatic synchronization are provided to ensure the consistency of data stored in different regions.

[0043] In the present application, the above-mentioned technical solutions can be combined with each other to realize more preferred combination schemes. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification, or will be understood by implementing the present application. The purpose and other advantages of the present application can be achieved and obtained from the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings are included to provide a better understanding of the embodiments of the present application, and should not be considered as limiting the present application, and throughout the drawings, the same reference signs represent the same components;

[0045] Figure 1 A flow chart of a distributed file management and control method based on a blockchain in embodiment 1 of the present application;

[0046] Figure 2 A structure schematic diagram of a distributed file management and control system based on a blockchain in embodiment 2 of the present application. DETAILED DESCRIPTION

[0047] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, wherein the drawings constitute a part of the present application, and are used to illustrate the principles of the embodiments of the present application, and are not used to limit the scope of the present application.

[0048] Embodiment 1

[0049] In one specific embodiment of the present application, a distributed file management and control method based on a blockchain is disclosed, as shown in Figure 1 The method comprises the following steps:

[0050] S1, using a clustering algorithm to divide each node of the distributed file system into a plurality of local areas;

[0051] S2, when requesting a file operation, locating a target area from the plurality of local areas according to the hash value of the file content, and the main node of the target area constructs a request message according to the file operation and submits it to the blockchain through the smart contract;

[0052] S3, the main node and the copy node of the blockchain perform multi-level verification on the request message, and when a consensus of passing verification is reached, the main node of the target area is authorized to perform the file operation through the smart contract;

[0053] S4, after the main node and the slave node of the target area reach a consensus on the operation, the file operation is performed in the target area, and an operation completion message is constructed and submitted to the blockchain through the smart contract; the blockchain writes the file operation into a file operation log.

[0054] The embodiment combines the tamper-proofing feature of the blockchain with the distributed file system, records all operation logs of the IPFS distributed file system by using the blockchain, and ensures the tamper-proofing of the operation logs.

[0055] In implementation, the smart contract is customized and deployed to the blockchain, and the file operation is recorded and protected through the double mechanism of the blockchain and the smart contract, the legality and consistency of the operation request are ensured by using the consensus mechanism, the malicious erasure protection is realized, and the anti-risk ability when responding to malicious operations or unauthorized behaviors is improved.

[0056] It should be noted that the smart contract of the embodiment includes a file operation request contract, which includes a plurality of methods: recording a file operation request, verifying a file operation request, and returning a verification result. The distributed file system and the blockchain complete the corresponding operation by calling the corresponding method in the interaction process.

[0057] In step S1, the nodes are dynamically divided into a plurality of local areas by using a clustering algorithm according to the physical positions and network delays of the nodes in the distributed system, and the storage and management of the distributed file are optimized, which specifically includes:

[0058] ①The network delay vector is obtained by calculating the network delay of each node to a plurality of preset reference nodes and performing standardization processing; the CIDR vector is obtained by obtaining the CIDR network segment identifier of each node according to the IP address and performing one-hot encoding; and the feature vector of each node is obtained according to the delay vector and the CIDR vector of each node.

[0059] It should be noted that a plurality of nodes are selected as reference nodes in the network topology of the distributed file system, and the network delay of other nodes to each reference node is calculated. Exemplarily, the ping command of the ICMP protocol is used to obtain the network delay time between nodes.

[0060] Preferably, the network delay is measured multiple times to take the median as the final network delay, so as to avoid the instantaneous jitter of the network.

[0061] Further, the network delays of each node to multiple reference nodes are normalized to obtain a network delay vector, such as Z-score normalization or Min-Max scaling, to ensure comparability of different dimensional features.

[0062] Considering that an IP address itself is a discrete decimal value, whether two IP addresses belong to the same subnet, the same physical cabinet, or the same data center cannot be directly reflected from the original IP value, and it is difficult to understand its physical meaning when directly used for clustering. For example, the numerical difference between 10.0.1.5 and 192.168.1.101 is large, and they can be in the same machine room. However, operation and maintenance personnel usually plan IP network segments according to physical locations.

[0063] Therefore, the embodiment obtains a CIDR (Classless Inter-Domain Routing) network segment identifier according to the IP address of each node, which is represented in the form of “network address / prefix length”. For example, when the CIDR network segment identifier is “192.168.10.0 / 24”, it represents IP addresses from “192.168.10.0” to “192.168.10.255”.

[0064] It should be noted that when calculating the CIDR network segment identifier according to the IP address, the IP address version is first determined. If it is IPv4, the IP address and subnet mask are converted into binary form, and then the network address is obtained by performing an “and” operation on the bits. Then, the prefix length is obtained according to the length of the continuous 1 in the binary form of the subnet mask. The network address and the prefix length are spliced with “ / ” to obtain the CIDR network segment identifier. If it is IPv6, the leftmost address of the IP address is kept unchanged according to the known prefix length, and the remaining bits are set to 0. Then, the network address is obtained according to the IPv6 compression rule, and the CIDR network segment identifier is obtained by splicing the network address and the prefix length with “ / ”.

[0065] Further, the CIDR network segment identifier of each node is one-hot encoded to obtain a CIDR vector.

[0066] The delay vector and the CIDR vector of each node are spliced to obtain a feature vector of each node.

[0067] ② Select an initial centroid according to the network load, local density, and relative distance of each node; use the K-Means clustering algorithm to cluster multiple local areas according to the initial centroid and the feature vector of each node.

[0068] It should be noted that the network load includes CPU utilization, memory utilization, and bandwidth utilization. The three utilization rates of each node are calculated, and the nodes with all three utilization rates less than the corresponding threshold are put into a candidate node set.

[0069] The local density of each candidate node in the candidate node set is calculated, the distance between it and all candidate nodes with higher local density than it is calculated, and the maximum distance is taken as the relative distance of the candidate node; wherein the local density of each candidate node is obtained according to the distance threshold to obtain the neighbor nodes of each candidate node, and then the local density is calculated according to the distance between the candidate node and its neighbor nodes using the Gaussian kernel density method.

[0070] The local density and the relative distance are sorted in descending order respectively, the product of the two ranking values of each candidate node is calculated to obtain a comprehensive value, and then the K candidate nodes with the highest comprehensive values are taken as the initial centroids.

[0071] The embodiment comprehensively considers the network load, local density and relative distance of the nodes, so that the selection of the initial centroids not only excludes the overloaded nodes, but also considers the density of the region where the node is located and the relative position of the node in the overall data distribution, and selects the nodes with good network status, high local density and large relative distance as the initial centroids. Such initial centroids are more likely to be located at the center positions of different clustering clusters and can cover different regions of the entire node space, thereby improving the performance and accuracy of the clustering algorithm.

[0072] It should be noted that when a new node is added to the distributed file system, the distance from the new node to the center of each local region is calculated, the new node is allocated to the local region closest to it, and a data synchronization request is sent to the master node in the local region to synchronize the files.

[0073] When the number of node changes in the distributed file system exceeds the change threshold, re-clustering is triggered and each local region is updated.

[0074] In step S2, when a file operation is requested, such as uploading a file, modifying a file or deleting a file, the target region of the file operation is first located from the multiple local regions according to the hash value of the file content, which specifically includes:

[0075] ①According to the number of nodes and the storage capacity proportion in each local region, the weight of each local region is obtained; and according to the weight of each local region, the number of virtual nodes in each local region is obtained.

[0076] It should be noted that the weight of each local region is obtained according to the number of nodes and the storage capacity proportion in each local region, which is obtained by taking the normalized storage capacity proportion of each local region as the basic weight, calculating the weight adjustment factor according to the number of nodes in each local region, and multiplying the basic weight by the weight adjustment factor. This method fine-tunes the node number under the premise of ensuring the capacity of the local region, avoiding the situation that a region with too few nodes bears too much load due to too large capacity.

[0077] Specifically, the storage capacity proportion of each local area is the ratio of the total storage available in each node in the local area to the total available storage of all nodes. The weight adjustment factor is to compare the number of nodes in each local area with the average number of nodes. If it is less than half of the average number of nodes, the weight adjustment factor is a value less than 1; if it is greater than 1.5 times the average number of nodes, the weight adjustment factor is a value greater than 1 but less than 1.5; if it does not satisfy the above conditions, the weight adjustment factor is 1; wherein the average number of nodes is the total number of nodes in the distributed file system divided by the number of local areas.

[0078] Further, according to the preset total number of virtual nodes and the normalized weight of each local area, the number of virtual nodes in each local area is obtained.

[0079] ②According to the identification of each virtual node in each local area, the hash value of the virtual node is obtained, and is sorted to form a hash ring.

[0080] It should be noted that the identification of each virtual node is composed of area code and index; according to a hash function such as MD5, SHA-256, the hash value of the identification of each virtual node is calculated; these hash values are sorted to form a hash ring. The head and tail of the hash ring are connected, and the virtual node with the smallest hash value follows the largest hash value, thereby forming a closed loop.

[0081] ③Clockwise search for the first virtual node greater than the hash value of the file content on the hash ring, and the local area to which the virtual node belongs is taken as the target area.

[0082] It should be noted that the hash value of the file content and the hash value of the virtual node are calculated by using the same hash function.

[0083] Compared with the prior art, the embodiment dynamically calculates the weight based on the actual storage capacity and the number of nodes in the local area, and integrates these weights into the file storage decision, effectively solving the problem of uneven distribution of files among distributed storage areas, and significantly improving the load balancing, resource utilization and overall stability of the system.

[0084] The master node of the target area is the node with the highest bandwidth in the target area. Preferably, a mark of the number of selected times is added to each node. If the bandwidths of multiple nodes are the same, the node with the smaller number of selected times is selected, and the number of selected times of the selected node is increased, so that the nodes with the same bandwidth take turns to act as the master node.

[0085] It should be noted that the embodiment finds the target area and its master node through the virtual node, and establishes the mapping relationship of the file and the target area and the master node. When the local area weight is updated regularly, the number of virtual nodes changes, and the new file will be located in the target area according to the changed virtual node, which does not affect the previously established mapping relationship.

[0086] Further, the master node of the target area constructs a request message according to the file operation, including: file identifier, request node (i.e. master node of the target area), timestamp, user identifier and operation type; then submits to the blockchain through the smart contract, executes step S3, and the master node and replica node of the blockchain perform multi-level verification on the request message.

[0087] It should be noted that the master node of the blockchain is obtained by calculating the credit value of each node in the blockchain and normalizing it as a weight, using a roulette algorithm, and selecting based on the weight of each node and the random number generated by VRF; the credit value of each node is obtained by calculating the online rate, task completion rate and resource availability rate of each node and weighted sum.

[0088] It should be noted that VRF (Verifiable Random Function) is a cryptographic tool, and the process of generating a random number using VRF is publicly verifiable, ensuring the unpredictability and fairness of the random number. Each generated random number is unique and unpredictable. When determining the master node of the blockchain, the weights of all nodes are accumulated to form a weight roulette, and each node corresponds to a roulette interval. The random number falls within the roulette interval of which node, and the node is selected. That is, the node with high credit value and weight occupies a larger area on the roulette, and the probability of being selected is also higher.

[0089] It should be noted that in order to reduce the load on the chain, the credit value of each node in the blockchain is calculated by the smart contract and the off-chain component at regular intervals, and the data calculated by the off-chain component is transmitted to the smart contract by the oracle. Among them, the off-chain components include heartbeat detection module, task management module and resource monitoring module; the smart contract further includes: credit contract, task contract and election contract.

[0090] Specifically, the heartbeat detection module in the off-chain component is used to send a heartbeat request to each node at regular intervals, record the response result, and calculate the online rate of each node within a preset time range; the task management module is used to calculate the task completion rate of each node according to the recorded task information of each node, and the task information includes allocation time, task status and task completion time; the resource monitoring module is used to calculate the resource availability rate of each node according to the regularly collected remaining storage, free bandwidth, CPU usage and memory usage of each node.

[0091] The credit contract comprises a credit value calculation, update and query method of each node; the task contract comprises a task allocation and verification method; and the election contract comprises a master node election method.

[0092] In implementation, the heartbeat detection module divides the preset time range into multiple non-overlapping time windows, calculates the online rate of each node in each time window according to the heartbeat request sent in each time window and the response result thereof, weights the online rate of each node in each time window according to the weight of each time window to obtain the online rate of each node, and submits the online rate to the credit contract by using the oracle machine. The sum of the weights of each time window is 1, and the closer to the current time, the greater the weight of the time window.

[0093] Illustratively, the preset time range is 72 hours, and each time window is 24 hours, so the weight of the time window closest to the current time t (t-0 to t-24) is 0.6, the weight of the time window in the middle (t-24 to t-48) is 0.3, and the weight of the time window farthest away (t-48 to t-72) is 0.1.

[0094] The task management module calculates the task completion rate of each node according to the task allocated by the task contract and the result of verifying the task state, and the proportion of the number of tasks completed by each node in the total number of tasks allocated in the preset period, and submits the task completion rate to the credit contract by using the oracle machine. Preferably, the preset period has the same length as the time window for calculating the online rate.

[0095] The resource monitoring module calculates the resource availability rate of each node according to the resource availability rate = 0.5×(remaining storage amount / total storage amount) + 0.3×(idle bandwidth / maximum bandwidth) + 0.2×(1 - (CPU usage rate+memory usage rate) / 2), and submits the resource availability rate to the credit contract by using the oracle machine.

[0096] Finally, the credit contract calculates the credit value of each node by weighted summation according to the online rate, the task completion rate and the resource availability rate of each node and the respective weights. Preferably, the weight of the online rate is 0.5, the weight of the task completion rate is 0.3, and the weight of the resource availability rate is 0.2.

[0097] After the master node of the blockchain is determined, the nodes in the blockchain that are not the master node are called replica nodes. The master node and the replica nodes of the blockchain perform multi-level verification on the request message, including:

[0098] ① The master node of the blockchain performs primary verification on the request message, and if the primary verification passes, the master node of the blockchain broadcasts a pre-preparation message to all replica nodes, the pre-preparation message comprising the request message and a mark indicating that the primary verification passes.

[0099] It should be noted that the primary verification is to obtain the target file and the target area of the master node from the request message, and verify whether the master node of the target area has the permission to operate the target file by calling the smart contract. The white list and the permission mapping table are included in the smart contract, which is convenient for checking the permission.

[0100] If the primary verification fails, it is considered that the verification fails, and the subsequent steps do not need to be performed.

[0101] ②After each replica node receives the pre-preparation message, the request message is subjected to secondary verification, and if the secondary verification passes, a preparation message is returned to the master node; when the number of preparation messages received by the master node exceeds the first threshold, the submission stage is entered, and a submission message is broadcast to all replica nodes.

[0102] It should be noted that the secondary verification includes verifying whether the master node of the target area has the permission to operate the target file, and checking whether the historical operation record of the target file is normal.

[0103] Specifically, checking whether the historical operation record of the target file is normal is to check whether the sequence of file operations conforms to the normal process of the file life cycle, for example, whether there is a file creation operation before the file deletion operation, and whether there is a file modification operation after the file deletion operation.

[0104] Each replica node performs double verification by calling the smart contract, and returns a preparation message to the master node when both verifications pass.

[0105] In order to avoid the existence of malicious nodes tampering with local permission data, the present embodiment does not rely on a single node in the blockchain, but checks through the white list and the permission mapping table commonly used in the smart contract, and when the number of preparation messages received by the master node exceeds the first threshold, such as more than two-thirds of the number of nodes in the network, the submission stage is entered, and a submission message is broadcast to all replica nodes, otherwise, it is considered that the verification fails, and the subsequent steps do not need to be performed.

[0106] ③After each replica node receives the submission message, if it agrees to execute the request message, it broadcasts the submission message; when the number of nodes broadcasting the submission message exceeds the first threshold, a consensus of passing the verification is reached.

[0107] It should be noted that in the submission stage, all replica nodes broadcast the submission message to the entire network, indicating that they agree to execute the request message, and when the number of nodes broadcasting the submission message exceeds two-thirds of the number of nodes in the network, it is considered that the verification passes, that is, the master node and the replica nodes reach the final consensus of passing the verification of the request message. Otherwise, it is considered that the verification fails, that is, the blockchain rejects this request message.

[0108] After the verification, the verification result is fed back to the master node of the target area through the smart contract, and if the verification result is that the verification is passed, it means that the master node of the target area is authorized to perform the file operation.

[0109] Step S3 verifies the authority of the request and the legality of the historical operation through the rotation of the master node of the blockchain, the preliminary verification and the consensus of the slave node, and ensures that the file operation request is authorized to execute the file operation of the distributed file system after being unanimously approved by the whole network.

[0110] Step S4 is the data synchronization operation in the target area. Specifically, after the master node of the target area is authorized to perform the file operation, it first reaches a consensus on the operation with the slave nodes in the same target area, including:

[0111] The master node of the target area writes the file operation into the log and synchronizes the new log entry to all slave nodes in the target area; all slave nodes compare their historical log records with the file operation log of the blockchain, and if they are consistent, write the new log entry and return a confirmation message to the master node of the target area;

[0112] When the number of confirmation messages received by the master node of the target area exceeds the second threshold, a consensus on the operation is reached with the slave nodes.

[0113] It should be noted that the target area is the direct impact range of the file operation, and the master node synchronizes the log entry to the slave node, which needs to be guaranteed by the distributed confirmation of the slave node that the versions of all files of all nodes in the target area are consistent, and the number of confirmation messages received by the master node of the target area exceeds the second threshold to improve fault tolerance. Preferably, the second threshold is two-thirds of the number of nodes in the target area.

[0114] If the historical log record of the slave node is consistent with the file operation log of the blockchain, the new log entry is written and a confirmation message is returned to the master node of the target area; otherwise, a data synchronization request is sent to the master node of the target area.

[0115] Further, when the master node and the slave node reach a consensus on the operation, the master node of the target area and the slave node that returns the confirmation message respectively perform the file operation, and the master node of the target area constructs an operation completion message and submits it to the blockchain through the smart contract. The master node sends the latest data copy to the inconsistent slave node according to the received data synchronization request.

[0116] Each node in the blockchain writes the file operation into the file operation log, and records the unique identifier of the file, the region code of the latest operation, the node identifier of the latest operation, the upload time, the latest modification time and the deletion time according to the type of the file operation.

[0117] It should be noted that due to the joining and exiting mechanism of the nodes in the distributed file system, different nodes may participate in the storage and management of the file at different times because they belong to a certain target area. Therefore, the nodes in the non-target area will regularly check whether the copy stored by themselves is consistent with the operation record on the blockchain, and if not, through the cross-area data synchronization mechanism, the consistency of the data is ensured.

[0118] Specifically, the cross-area data synchronization mechanism of the embodiment includes that the nodes in the non-target area regularly check whether the operation time of each file in the log stored by themselves is consistent with the corresponding time in the file operation log of the blockchain, if not, according to the operation type of the inconsistent file, execute the data operation and update the log, or send a data synchronization request to the master node of the target area recorded in the file operation log, complete the data synchronization operation according to the received latest file copy and update the log.

[0119] It should be noted that if the operation type of the inconsistent file is deletion, the node itself performs the deletion operation on the file and updates the log; if the operation type of the inconsistent file is upload or modification, a data synchronization request is sent to the master node of the corresponding target area, and the data synchronization operation is completed according to the received latest data copy and the log is updated.

[0120] It can be understood that when the node accessed by the user does not have the required file, the cross-area data synchronization mechanism is also used to send a data synchronization request to the master node of the target area where the required file was last operated, and the user's required operation is completed according to the received latest file copy.

[0121] Preferably, before sending the latest file copy, the master node of the target area checks whether the node in the non-target area has the permission to operate the file through the smart contract, and if not, the synchronization is refused.

[0122] Compared with the prior art, the distributed file management method based on the blockchain provided by the embodiment records and protects file operations through the dual mechanism of the blockchain and the smart contract, uses the consensus mechanism to ensure the legality and consistency of the operation request, realizes malicious erasure protection, and improves the anti-risk ability when responding to malicious operations or unauthorized behaviors. According to the physical location and network delay of each node in the distributed system, a feature vector of each node is constructed, a clustering algorithm is improved, a plurality of initial centroids are selected according to the network load, local density and relative distance of each node, different areas of the entire node space are covered, and the performance and accuracy of the clustering algorithm are improved; by dynamically dividing the nodes into a plurality of local areas, the storage and management of the distributed file are optimized. Based on the actual storage capacity and the number of nodes in the local area, the weight is dynamically calculated, and the weight is integrated into the file storage decision, effectively solving the problem of uneven distribution of files among the distributed storage areas, and significantly improving the load balancing, resource utilization and overall stability of the distributed file system. By calculating the credit value of each node in the blockchain at regular intervals, a roulette algorithm is used to realize the rotation mechanism of the main node of the blockchain, and through multi-level verification in the consensus process of the file operation request, the security and reliability of the file operation are improved; through the automatic operation authorization and state feedback of the smart contract, the efficiency is improved, and the distributed environment of a large number of nodes is adapted. Considering the joining and exiting of the nodes in the distributed file system, a cross-area data synchronization mechanism is provided, which quickly responds and automatically synchronizes when data inconsistency is found, and ensures the consistency of data stored in different areas.

[0123] Embodiment 2

[0124] Another embodiment of the application discloses a distributed file management system based on a blockchain, so as to realize the distributed file management method based on the blockchain in embodiment 1. The specific implementation of each module is referred to the corresponding description in embodiment 1. The management system deploys a smart contract on the blockchain for interaction between the distributed file system and the blockchain; as shown in the figure, the management system comprises: Figure 2

[0125] The local area division module 101 is used for dividing the nodes of the distributed file system into a plurality of local areas by using a clustering algorithm;

[0126] The file operation request module 102 is used for positioning a target area from the plurality of local areas according to the hash value of the file content when requesting a file operation, and the main node of the target area constructs a request message according to the file operation and submits it to the blockchain through the smart contract;

[0127] The file operation approval module 103 is used for the main node and the copy node of the blockchain to perform multi-level verification on the request message, and when a consensus of passing the verification is reached, the main node of the target area is authorized to perform the file operation through the smart contract; ​

[0128] The file operation execution module 104 is configured to execute the file operation in the target region after the master node and the slave node of the target region reach consensus on the confirmation operation, and construct an operation completion message and submit the operation completion message to the blockchain through the smart contract; the blockchain writes the file operation into the file operation log.

[0129] Preferably, the management system further comprises a file operation synchronization module; the file operation synchronization module is configured to periodically check whether the time of each file operation in the log stored by the node of a non-target region is consistent with the time of each corresponding file operation in the file operation log of the blockchain, if not, perform a data operation according to the type of the inconsistent file operation and update the log, or send a data synchronization request to the master node of the target region recorded in the file operation log, complete the data synchronization operation according to the latest file copy received and update the log.

[0130] Since the embodiment of the present application is related to the foregoing method of managing and controlling a distributed file based on a blockchain, the two can be mutually referred to, and thus, the description is repeated here. Since the system embodiment and the method embodiment have the same principles, the system embodiment also has the corresponding technical effects of the method embodiment.

[0131] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0132] The above description is only a preferred embodiment of the present application, and the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the present application.

Claims

1.A blockchain-based distributed file management method, characterized in that, The method comprises the following steps: dividing nodes of a distributed file system into multiple local areas by using a clustering algorithm; when a file operation is requested, locating a target area from the multiple local areas according to a hash value of file content, and constructing a request message by a master node of the target area according to the file operation and submitting the request message to a blockchain through a smart contract; performing multi-level verification on the request message by master nodes and copy nodes of the blockchain, and when a consensus of passing the verification is reached, authorizing the master node of the target area to execute the file operation through the smart contract; after a consensus of confirming the operation is reached by the master node and the slave node of the target area, executing the file operation in the target area and constructing an operation completion message and submitting the operation completion message to the blockchain through the smart contract; writing the file operation into a file operation log by the blockchain; the locating of the target area from the multiple local areas according to the hash value of the file content comprises: obtaining a weight of each local area according to a number of nodes in each local area and a storage total capacity proportion; obtaining a number of virtual nodes of each local area according to the weight of each local area; obtaining a hash value of each virtual node according to an identification of each virtual node in each local area, and sorting to form a hash ring; finding a first virtual node greater than the hash value of the file content on the hash ring in a clockwise direction, and taking a local area to which the virtual node belongs as the target area; when the weight of the local area is updated periodically, the number of virtual nodes changes, and a new file is located to the target area according to the changed virtual nodes. 2.The blockchain-based distributed file management method of claim 1, wherein, the method further comprises: periodically checking, by a node of a non-target area, whether times of each file operation in a log stored by the node are consistent with corresponding times in a file operation log of the blockchain, and if not, performing a data operation according to an operation type of an inconsistent file and updating the log, or sending a data synchronization request to a master node of the target area recorded in the file operation log, completing a data synchronization operation according to a latest file copy received and updating the log. 3.The blockchain-based distributed file management method of claim 1, wherein, the dividing of the nodes of the distributed file system into the multiple local areas by using the clustering algorithm comprises: calculating network delays of the nodes to multiple preset reference nodes and performing standardization processing to obtain network delay vectors; obtaining CIDR vector by obtaining a CIDR network segment identifier of each node according to an IP address of the node and performing one-hot encoding; and obtaining a feature vector of each node according to the delay vector and the CIDR vector of the node; selecting multiple initial centroids according to network loads, local densities and relative distances of the nodes; and clustering the multiple local areas according to the multiple initial centroids and the feature vector of each node by using a K-Means clustering algorithm. 4.The blockchain-based distributed file management method of claim 3, wherein, the selecting of the multiple initial centroids according to the network loads, the local densities and the relative distances of the nodes comprises: the network load comprises CPU utilization, memory utilization and bandwidth utilization; three utilization rates of the network load of each node are calculated, and nodes with all the three utilization rates less than corresponding thresholds are put into a candidate node set; The local density of each candidate node in the candidate node set is calculated, the distance between it and all candidate nodes with higher local density than it is calculated, and the maximum distance is taken as the relative distance of the candidate node; The local density and the relative distance are sorted in descending order respectively, the product of the two ranking values of each candidate node is calculated to obtain a comprehensive value, and the top K candidate nodes are taken as the initial centroids in descending order of the comprehensive value of each candidate node. 5.The blockchain-based distributed file management method of claim 1, wherein, The master node of the target region is the node with the highest bandwidth in the target region; the master node of the blockchain is obtained by calculating the credit value of each node in the blockchain and normalizing it as a weight, using a roulette algorithm, and selecting based on the weight of each node and a random number generated by VRF; the credit value of each node is obtained by calculating the online rate, task completion rate and resource availability rate of each node and weighted sum. 6.The blockchain-based distributed file management method according to claim 1 or 5, wherein, The master node and the replica node of the blockchain perform multi-level verification on the request message, including: The master node of the blockchain performs primary verification on the request message, and if the primary verification is passed, the master node of the blockchain broadcasts a pre-preparation message to all replica nodes, the pre-preparation message including the request message and a mark indicating that the primary verification is passed; After each replica node receives the pre-preparation message, it performs secondary verification on the request message, and if the secondary verification is passed, it returns a preparation message to the master node; when the number of preparation messages received by the master node exceeds a first threshold, it enters a submission stage and broadcasts a submission message to all replica nodes; After each replica node receives the submission message, if it agrees to execute the request message, it broadcasts a submission message; when the number of nodes broadcasting the submission message exceeds the first threshold, a consensus of passing the verification is reached. 7.The blockchain-based distributed file management method of claim 6, wherein, The primary verification is to obtain the target file and the master node of the target region from the request message, and to verify whether the master node of the target region has the permission to operate the target file; the secondary verification includes verifying whether the master node of the target region has the permission to operate the target file, and checking whether the historical operation record of the target file is normal. 8.The blockchain-based distributed file management method of claim 1, wherein, The master node of the target region and the slave node reach a consensus on the operation confirmation, and then perform the file operation in the target region, including: The master node of the target region writes the file operation into a log and synchronizes the new log entry to all slave nodes in the target region; all slave nodes compare their own historical log records with the file operation log of the blockchain, and if they are consistent, they write the new log entry and return a confirmation message to the master node of the target region; When the number of confirmation messages received by the master node of the target region exceeds a second threshold, a consensus on the operation confirmation is reached with the slave node, and the master node and the slave node of the target region perform the file operation. 9.A blockchain-based distributed file management system, characterized in that, It includes: A local region division module for dividing each node of a distributed file system into multiple local regions using a clustering algorithm; A file operation request module for requesting a file operation, locating a target region from the multiple local regions according to the hash value of the file content, and constructing a request message according to the file operation by the master node of the target region and submitting it to the blockchain through a smart contract; The target region is located from the multiple local regions according to the hash value of the file content, including: obtaining the weight of each local region according to the number of nodes and the storage total capacity proportion in each local region; obtaining the virtual node number of each local region according to the weight of each local region; obtaining the hash value of the virtual node according to the identification of each virtual node in each local region, and sorting to form a hash ring; clockwise searching the first virtual node greater than the hash value of the file content on the hash ring, and the local region to which the virtual node belongs is taken as the target region; when the local region weight is updated regularly, the virtual node number changes accordingly, and the new file is located to the target region according to the changed virtual node; The file operation approval module is used for the master node and the copy node of the blockchain to perform multi-level verification on the request message, and when a consensus of passing the verification is reached, the target region master node is authorized to execute the file operation through the smart contract; The file operation execution module is used for the target region master node and the slave node to reach a consensus of confirming the operation, and then execute the file operation in the target region, and construct an operation completion message and submit it to the blockchain through the smart contract; the blockchain writes the file operation into the file operation log.

Images