Data processing method, device and readable storage medium

Abstract

The application provides a data processing method and device and a readable storage medium, and relates to the technical field of communication. The method comprises the following steps: determining a feature value of each data shard of target data; determining a feature value of each storage node; mapping each data shard and each storage node to a hash space according to the feature value of each data shard and the feature value of each storage node; and transmitting each data shard to a corresponding storage node respectively; wherein, the storage node is a distributed storage node, and the data shard and the corresponding storage node are mapped to the same position in the hash space.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and particularly relates to a data processing method and device and a readable storage medium. BACKGROUND

[0002] In the era of big data, the scale of data grows exponentially, and traditional centralized storage systems cannot meet the efficient, secure and scalable storage requirements. Distributed storage stores data on multiple nodes, which not only improves the fault tolerance and scalability of the system, but also effectively deals with the challenges of high concurrency access and massive data processing.

[0003] Currently, distributed storage systems usually adopt a centralized architecture, relying on metadata servers to manage data distribution and consistency. However, the centralized architecture has poor scalability, and new nodes need to frequently communicate with metadata servers, increasing network overhead and delay. SUMMARY

[0004] The purpose of the present application is to provide a data processing method, device and readable storage medium to avoid the need for new nodes in existing separate storage systems to frequently communicate with metadata servers, thereby increasing network overhead and delay.

[0005] To achieve the above purpose, an embodiment of the present application provides a data processing method, comprising:

[0006] determining the characteristic value of each data shard of the target data;

[0007] determining the characteristic value of each storage node;

[0008] mapping each data shard and each storage node to a hash space according to the characteristic value of each data shard and the characteristic value of each storage node;

[0009] transmitting each data shard to the corresponding storage node respectively;

[0010] wherein the storage node is a distributed storage node, and the data shard and the corresponding storage node are mapped to the same location in the hash space.

[0011] Optionally, the method further comprises:

[0012] determining a plurality of data shards of the target data according to the target data and storage node information.

[0013] Optionally, the method further comprises:

[0014] generating and storing verification information according to the characteristic value of each data shard, wherein the verification information is used to verify the integrity of the data.

[0015] Optionally, the generating and storing the verification information according to the feature value of each data shard comprises:

[0016] constructing a binary tree according to the feature value of each data shard; wherein the root feature value of the binary tree is determined by parallel computing;

[0017] storing the root feature value as the verification information to a block chain.

[0018] Optionally, the method further comprises:

[0019] in the case that a user applies to obtain a first target data shard, obtaining the feature value and path information of the first target data shard; the path information comprises the feature value of the sibling node of the first target data shard in the binary tree;

[0020] verifying the integrity of the first target data shard according to the verification information, the feature value of the first target data shard and the path information;

[0021] in the case that the verification of the first target data shard is passed, transmitting the first target data shard to the user.

[0022] Optionally, the method further comprises:

[0023] in the case that the user completes the download of the first target data shard, caching the path information; or,

[0024] in the case that the access frequency of the path information is greater than a preset threshold, updating the path information to a path feature value, the path feature value being obtained by compressing the path information.

[0025] Optionally, the method further comprises:

[0026] determining the number of copies of each data shard according to the storage node information;

[0027] generating copy data according to the number of copies of each data shard;

[0028] transmitting the copy data of each data shard to the corresponding storage node respectively.

[0029] Optionally, the method further comprises:

[0030] monitoring the running state of all storage nodes;

[0031] In the case of monitoring a fault storage node, a first target copy data on a non-fault storage node is queried according to a second target data shard on the fault storage node, the first target copy data being a copy data of the second target data shard;

[0032] If all copy data of the second target data shard is queried, the first target copy data is copied and stored to a target storage node;

[0033] If all copy data of the second target data shard is not queried, the second target data shard is recovered through erasure coding according to the first target copy data queried, and is stored to a target storage node.

[0034] Optionally, the method further comprises:

[0035] Obtaining index information of a recovery index corresponding to each storage node;

[0036] According to the index information, a storage node meeting a preset condition is determined as the target storage node.

[0037] Optionally, the method further comprises:

[0038] Judging whether version numbers of copy data of a third target data shard on different storage nodes are consistent;

[0039] In the case of inconsistent version numbers, the copy data with the highest version number is synchronized on each storage node according to the version numbers of the copy data of the third target data shard.

[0040] To achieve the above object, an embodiment of the present application provides a data processing device, comprising:

[0041] A first processing module is configured to determine a characteristic value of each data shard of target data;

[0042] A second processing module is configured to determine a characteristic value of each storage node;

[0043] A third processing module is configured to map each data shard and each storage node to a hash space according to the characteristic value of each data shard and the characteristic value of each storage node;

[0044] A transmission module is configured to transmit each data shard to a corresponding storage node respectively;

[0045] The storage node is a distributed storage node, and the data shard and the corresponding storage node are mapped to the same position in the hash space.

[0046] Optionally, the device further comprises:

[0047] a fourth processing module, configured to determine a plurality of data shards of the target data according to the target data and storage node information.

[0048] Optionally, the apparatus further includes:

[0049] a fifth processing module, configured to generate and store verification information according to a feature value of each data shard, the verification information being used to verify integrity of the data.

[0050] Optionally, the fifth processing module is further configured to:

[0051] construct a binary tree according to the feature value of each data shard; wherein a root feature value of the binary tree is determined through parallel calculation;

[0052] store the root feature value as the verification information to a blockchain.

[0053] Optionally, the apparatus further includes:

[0054] a sixth processing module, configured to

[0055] in a case where a user applies to obtain a first target data shard, obtain a feature value and path information of the first target data shard; the path information including a feature value of a sibling node of the first target data shard in the binary tree;

[0056] verify integrity of the first target data shard according to the verification information, the feature value of the first target data shard and the path information;

[0057] in a case where the first target data shard passes the verification, transmit the first target data shard to the user.

[0058] Optionally, the apparatus further includes:

[0059] a seventh processing module, configured to, in a case where a user completes downloading of the first target data shard, cache the path information;

[0060] an eighth processing module, configured to, in a case where an access frequency of the path information is greater than a preset threshold, update the path information to a path feature value, the path feature value being obtained based on compression of the path information.

[0061] Optionally, the apparatus further includes:

[0062] a ninth processing module, configured to

[0063] determine a number of copies of each data shard according to storage node information;

[0064] According to the number of copies of each data slice, copy data is generated;

[0065] The copy data of each data slice is respectively transmitted to the corresponding storage node.

[0066] Optionally, the apparatus further comprises:

[0067] The tenth processing module is configured to

[0068] Monitor the running state of all storage nodes;

[0069] In the case of monitoring a faulty storage node, according to a second target data slice on the faulty storage node, a first target copy data on a non-faulty storage node is queried, the first target copy data being copy data of the second target data slice;

[0070] If all copy data of the second target data slice is queried, the first target copy data is copied and stored to a target storage node;

[0071] If all copy data of the second target data slice is not queried, the second target data slice is recovered through erasure coding according to the first target copy data that is queried, and is stored to a target storage node.

[0072] Optionally, the apparatus further comprises:

[0073] The eleventh processing module is configured to

[0074] Obtain index information of a recovery index corresponding to each storage node;

[0075] According to the index information, a storage node that meets a preset condition is determined as the target storage node.

[0076] Optionally, the apparatus further comprises:

[0077] The twelfth processing module is configured to

[0078] Determine whether the version numbers of copy data of a third target data slice on different storage nodes are consistent;

[0079] In the case of inconsistent version numbers, according to the version numbers of the copy data of the third target data slice, the copy data with the highest version number is synchronized on each storage node.

[0080] To achieve the above purpose, an embodiment of the present application provides a data processing device, comprising a processor and a transceiver, the processor being configured to:

[0081] Determine a feature value of each data slice of target data;

[0082] determining eigenvalues of each storage node;

[0083] mapping each data shard and each storage node to a hash space according to the eigenvalue of each data shard and the eigenvalue of each storage node;

[0084] the transceiver is configured to transmit each data shard to a respective corresponding storage node;

[0085] wherein the storage nodes are distributed storage nodes, and the data shards and the corresponding storage nodes are mapped to the same location in the hash space.

[0086] Optionally, the processor is further configured to:

[0087] determining a plurality of data shards of the target data according to the target data and the storage node information.

[0088] Optionally, the processor is further configured to

[0089] generating and storing verification information according to the eigenvalue of each data shard, the verification information being used to verify the integrity of data.

[0090] Optionally, the processor is further configured to

[0091] constructing a binary tree according to the eigenvalue of each data shard, wherein a root eigenvalue of the binary tree is determined by parallel computing;

[0092] storing the root eigenvalue as the verification information to a blockchain.

[0093] Optionally, the processor is further configured to

[0094] in a case where a user applies to obtain a first target data shard, obtaining an eigenvalue and path information of the first target data shard, the path information including an eigenvalue of a sibling node of the first target data shard in the binary tree;

[0095] verifying the integrity of the first target data shard according to the verification information, the eigenvalue of the first target data shard, and the path information;

[0096] the transceiver is further configured to transmit the first target data shard to the user in a case where the first target data shard passes the verification.

[0097] Optionally, the processor is further configured to

[0098] in a case where the user completes downloading of the first target data shard, caching the path information; or,

[0099] In a case where the access frequency of the path information is greater than a preset threshold, the path information is updated as a path feature value, the path feature value being obtained based on compression of the path information.

[0100] Optionally, the processor is further configured to

[0101] According to the storage node information, determine the number of copies of each data shard;

[0102] According to the number of copies of each data shard, generate copy data;

[0103] Transmit the copy data of each data shard to the corresponding storage node respectively.

[0104] Optionally, the processor is further configured to

[0105] Monitor the running state of all storage nodes;

[0106] In a case where a fault storage node is monitored, according to a second target data shard on the fault storage node, query a first target copy data on a non-fault storage node, the first target copy data being copy data of the second target data shard;

[0107] If all copy data of the second target data shard is queried, copy and store the first target copy data to a target storage node;

[0108] If all copy data of the second target data shard is not queried, according to the queried first target copy data, recover the second target data shard through erasure coding, and store to a target storage node.

[0109] Optionally, the processor is further configured to

[0110] Obtain index information of a recovery index corresponding to each storage node;

[0111] According to the index information, determine a storage node satisfying a preset condition as the target storage node.

[0112] Optionally, the processor is further configured to

[0113] Determine whether the version numbers of copy data of a third target data shard on different storage nodes are consistent;

[0114] In a case where the version numbers are not consistent, according to the version numbers of the copy data of the third target data shard, synchronize the copy data with the highest version number on each storage node.

[0115] To achieve the above object, the embodiment of the present application provides a data processing device, comprising a transceiver, a processor, a memory and a program or instruction stored in the memory and executable on the processor; the processor implements the data processing method as described above when executing the program or instruction.

[0116] To achieve the above object, the embodiment of the present application provides a readable storage medium, which stores a program or instruction, and the program or instruction implements the steps of the data processing method as described above when executed by a processor.

[0117] To achieve the above object, the embodiment of the present application provides a computer program product, which comprises computer instructions, and the computer instructions implement the steps of the data processing method as described above when executed by a processor.

[0118] The beneficial effects of the above technical solutions of the present application are as follows:

[0119] The method of the embodiment of the present application can map the data shards and the storage nodes to the hash space through the characteristic values of the data shards and the characteristic values of the storage nodes, establish the corresponding relationship between the data shards and the storage nodes, and thus store the data shards in the corresponding storage nodes. In this way, only part of the data needs to be migrated when the nodes are added or exited, thereby reducing the network overhead and delay. BRIEF DESCRIPTION OF DRAWINGS

[0120] Figure 1 The flow chart of the data processing method of the embodiment of the present application;

[0121] Figure 2 The structural schematic diagram of the data processing device of the embodiment of the present application;

[0122] Figure 3 The structural schematic diagram of the data processing device of the embodiment of the present application;

[0123] Figure 4 The schematic diagram of the distributed storage system in the embodiment of the present application;

[0124] Figure 5 The structural schematic diagram of the data processing device of another embodiment of the present application. DETAILED DESCRIPTION

[0125] To make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0126] It should be understood that the reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described is included in at least one embodiment of the application. Therefore, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0127] In various embodiments of the present application, it should be understood that the magnitude of the serial number of the following processes does not mean the order of execution, the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0128] In addition, the terms "system" and "network" are often used interchangeably herein.

[0129] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that the determination of B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0130] For the convenience of understanding, some contents related to the embodiments of the present application are described as follows:

[0131] The research of distributed storage technology has important practical significance, especially in the era of big data, the data scale grows exponentially, and the traditional centralized storage system has been unable to meet the efficient, secure, and scalable storage needs. Distributed storage stores data on multiple nodes, not only improves the fault tolerance and scalability of the system, but also effectively deals with the challenges of high concurrency access and massive data processing. However, with the increasing demand for public data sharing and openness, distributed storage systems face many challenges in data security and integrity verification.

[0132] Traditional distributed storage systems typically employ a centralized architecture, relying on metadata servers to manage data distribution and consistency. The metadata server stores file metadata (such as file paths and shard locations) and simplifies data location and access through centralized management. When a client accesses data, it first requests the file's shard location information from the metadata server and then communicates directly with the storage nodes to retrieve the data. This architecture improves data access efficiency through centralized metadata management, performing particularly well in small-scale data scenarios. The Hadoop Distributed File System (HDFS) uses a master-slave architecture, with a single master node (NameNode) managing metadata and multiple slave nodes (DataNodes) storing the actual data. Ceph, on the other hand, dynamically calculates data distribution using the CRUSH algorithm, reducing reliance on the metadata server, but still requires a centralized monitor node to maintain the cluster state. This approach offers high efficiency in data management and access control, making it suitable for scenarios requiring strong consistency and centralized management.

[0133] Blockchain storage systems employ a decentralized peer-to-peer network structure, completely eliminating the design of centralized nodes. Data is sharded and distributed across multiple nodes in the network, with each node acting as both a data provider and consumer. The InterPlanetary File System (IPFS) uses content addressing technology to shard files and generate unique hash values ​​as identifiers. Users can retrieve data directly from the network using these hash values, without relying on centralized servers. The Storj system utilizes blockchain technology to record the distribution information of data shards and storage contracts, ensuring data traceability and security. Decentralized storage systems achieve data location through distributed hash tables, and nodes exchange data directly through peer-to-peer communication, avoiding single points of failure. This technological approach offers high resistance to attacks and data redundancy, making it suitable for scenarios requiring high reliability and decentralized management.

[0134] like Figure 1 As shown in the figure, a data processing method according to an embodiment of this application includes:

[0135] Step 101: Determine the feature values ​​of each data segment of the target data;

[0136] Step 102: Determine the characteristic values ​​of each storage node;

[0137] Step 103: Based on the feature value of each data shard and the feature value of each storage node, map each data shard and each storage node to a hash space;

[0138] Step 104: Transmit each of the data fragments to its corresponding storage node;

[0139] The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0140] Following the steps described above, the method of this application embodiment can map both data shards and storage nodes to a hash space using the feature values ​​of the data shards and the feature values ​​of the storage nodes, establishing a correspondence between data shards and storage nodes, thereby enabling the storage of data shards to the corresponding storage nodes. In this way, only partial data migration is required when a node joins or leaves, reducing network overhead and latency.

[0141] Moreover, the mapping implemented using feature values ​​in this application can quickly verify whether data fragments have been tampered with, avoiding the need to verify the entire data fragment content byte by byte.

[0142] Optionally, in this embodiment, the hash space is a ring structure with the first and last ends connected.

[0143] Optionally, in this embodiment, the target data is public data, in order to improve the performance, reliability, and scalability of the public data distributed storage system.

[0144] Optionally, in this embodiment, the characteristic value of the data shard is a hash value, such as data shard D. i The eigenvalues ​​can be denoted as H(D) i ) or H i H ( The hash function can be SHA-256 or other algorithms. The characteristic value of a data fragment can serve as its unique identifier.

[0145] Optionally, in this embodiment, the characteristic value of the storage node is a hash value, which can be obtained by hashing the name or identifier (ID) of the storage node.

[0146] Optionally, the method further includes:

[0147] Based on the target data and storage node information, multiple data fragments of the target data are determined.

[0148] Here, storage node information is used to indicate the scale of storage nodes, such as the number of storage nodes and network topology. Thus, by combining the target data and the storage node information, the target data is divided into multiple data fragments.

[0149] In one optional implementation, the target data is evenly divided into n data shards of equal size based on the number of storage nodes, such as:

[0150] .in, The total amount of target data, The size of each data shard. This represents the number of storage nodes.

[0151] Distributing the target data evenly avoids frequent adjustments to data shard sizes, dynamically balances storage pressure, and resolves storage performance bottlenecks caused by excessively large or small data blocks. The choice of 'n' can also be dynamically adjusted based on the total data volume and node scale, ensuring that the size of each shard is within a reasonable range to optimize storage and access performance.

[0152] After the target data is evenly divided, the data is divided into D pieces. i Mapping to storage nodes, since data sharding and storage nodes are mapped to hash space, not only does it support the dynamic addition and removal of nodes, reducing the data migration overhead caused by node changes, but it also optimizes the uniformity of data distribution and avoids data skew problems.

[0153] In this process, data sharding is mapped to a hash space, which can employ consistent hashing, such as... , where H ( H(D) is a hash function, which can be an algorithm such as SHA-256. i ) is data sharding D i The characteristic value is N, where N is the number of storage nodes. It is data sharding D i Mapping information.

[0154] Optionally, if the target data needs to be stored in shards, the target data can be sharded. After sharding, it can be checked whether each data shard was successfully sharded, such as whether there are any data shards that are lost or duplicated.

[0155] Optionally, in this embodiment, each data shard is transmitted to the corresponding storage node via a secure transmission protocol. The secure transmission protocol can be HTTPS or an encrypted tunnel.

[0156] Optionally, in this embodiment, after each data shard is transmitted to its corresponding storage node, distribution information can be recorded on the blockchain. This distribution information can be the nodes where each data shard is stored.

[0157] Optionally, in this embodiment, the method further includes:

[0158] Verification information is generated and stored based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0159] In this way, for the target data, verification information can be generated from the feature values ​​of each data fragment to ensure that the data fragments downloaded by the user are complete and tamper-proof original data, thus avoiding access to all data fragments.

[0160] Optionally, after each data fragment of the target data is stored in its corresponding storage node, these data fragments are extracted from each storage node, their feature values ​​are further determined, and then verification information is generated. The determined feature values ​​can be hash values, obtained using a hash function such as the SHA-256 algorithm.

[0161] Optionally, in this embodiment, generating and storing verification information based on the feature values ​​of each data segment includes:

[0162] A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation.

[0163] The root feature value is stored in the blockchain as the verification information.

[0164] That is, the feature values ​​of the parent nodes are calculated layer by layer according to the binary tree structure until the root feature value is generated. The root feature value is then used as verification information to check the integrity of the data. In this process, the binary tree can compress the amount of information in the verification information, and the parallel calculation of the root feature value can effectively reduce the latency of large-scale data verification.

[0165] For example, if the feature value is a hash value, a binary tree node only has two child nodes, left and right. H left H is the left child node. right The right child node is represented by ||, which is the string concatenation operation. root This is the root hash value.

[0166] and T Merkle Let P be the time for parallel computation, Depth be the tree depth, and P be the number of parallel processing units. Thus, the more parallel processing units there are, the shorter the computation time for the root eigenvalue. For a binary tree, P=2.

[0167] In one alternative implementation, the binary tree is a Merkle tree.

[0168] Optionally, in this embodiment, the method further includes:

[0169] When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree;

[0170] The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information.

[0171] If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

[0172] In other words, when a user requests to obtain the first target data fragment (which can also be understood as the first data fragment), the integrity of the first target data fragment will be verified based on the verification information, the feature value of the first target data fragment, and the path information. Only after the verification of the first target data fragment passes will the first target data fragment be transmitted to the user.

[0173] Optionally, verifying the integrity of the first target data fragment based on the verification information, the feature value of the first target data fragment, and the path information includes:

[0174] Based on the feature values ​​of the first target data segment and the path information, the information to be verified is determined;

[0175] Verify whether the verification information and the information to be verified are consistent. If they are consistent, the verification passes; otherwise, the verification fails.

[0176] In one alternative implementation, taking the feature value as a hash value as an example, the process of determining the sibling nodes s1 and s2 based on the path information can be represented as follows:

[0177] , where H i H is the hash value of the first target data shard. s1 H s2 H is the hash value of the sibling node. root This is the root hash value.

[0178] Of course, the device that applies the method of the embodiments of this application can also send the first target data fragment and the corresponding path information to the user when the user requests the download of the first target data fragment, so that the user can verify the integrity of the data, wherein the user obtains the verification information from the blockchain.

[0179] Optionally, in this embodiment, the method further includes:

[0180] If the user completes the download of the first target data segment, the path information will be cached; or,

[0181] If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

[0182] Thus, in this embodiment of the application, on the one hand, the most recently verified path information P will be cached. iThis allows for direct reuse of the cache during repeated verification, reducing the overhead of redundant calculations. Furthermore, for path information accessed more frequently than a preset threshold, it can be updated with compressed path feature values, improving the efficiency of path access.

[0183] In one alternative implementation, Huffman coding is used to compress the path and reduce its length, as shown in the formula:

[0184] , where p i L represents the probability of accessing path information. path This represents the compressed path length.

[0185] Optionally, in this embodiment, the method further includes:

[0186] Based on the storage node information, determine the number of replicas for each data shard;

[0187] Based on the number of replicas for each data shard, generate replica data;

[0188] The copy data of each data shard is transmitted to the corresponding storage node.

[0189] Thus, after determining the number of replicas for each data shard based on the storage node information, a corresponding number of replica data can be generated and transmitted to the storage nodes to ensure normal operation even in the event of node failure. The number of replicas can be dynamically adjusted based on the storage node information to balance redundant storage costs and fault tolerance.

[0190] In one alternative implementation, the number of replicas for each data shard can be determined using the following formula:

[0191] r is the number of replicas of a data shard, and n is the number of storage nodes, ensuring that the system can still operate normally even with the failure of a fault-tolerant node.

[0192] For data fragment D i After determining the number of replicas r, data shards D can be generated. i The data is processed into r replicas, and then the r replicas are transferred to other storage nodes.

[0193] Optionally, in this embodiment, the characteristic values ​​of the replica data are also used to map each replica data to a hash space, and the storage nodes are also mapped to the hash space, so that the replica data can be transmitted to the storage nodes mapped to the same hash space.

[0194] In one optional implementation, consistent hashing is used to distribute replica data:

[0195] j is the replica data sequence number, with a value range of j=0,1,…,r-1. Δ is the fixed step size. It is data sharding D i The location of the j-th sub-data that is mapped to the hash space.

[0196] Consistent hashing distributes replicas evenly, preventing them from being concentrated on a few nodes. Multi-hop hashing is achieved by dynamically optimizing the replica distribution through adjusting the step size Δ.

[0197] Optionally, in this embodiment, the replica data and the corresponding storage node are recorded in the blockchain.

[0198] Optionally, in this embodiment, the method further includes:

[0199] Monitor the operational status of all storage nodes;

[0200] If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard.

[0201] If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node;

[0202] If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0203] In this way, by monitoring the operating status of all storage nodes, data recovery can be carried out in a timely manner when a faulty storage node is detected, ensuring that the system can quickly return to normal when a node fails.

[0204] The monitoring method involves determining whether a heartbeat signal is received within a specified time. If not, the storage node is marked as "failed"; if so, it is marked as "healthy". The formula is as follows:

[0205] T now T represents the current time. hearthbeat Δ is the time since the last monitoring. t Allowed monitoring latency. NodeID is the node identifier.

[0206] Heartbeat signals can be monitored based on a set heartbeat interval. Of course, the heartbeat interval can be dynamically adjusted to change the monitoring frequency, reducing unnecessary detection overhead.

[0207] In this application, data recovery can be performed using replicas or erasure coding. Specifically, when a faulty storage node is detected, based on the second target data shard on the faulty storage node, the first target replica data on the non-faulty storage node is queried. The first target replica data is a replica of the second target data shard. If all replica data of the second target data shard is found (i.e., the queried first target replica data includes all replica data of the second target data shard), this first target replica data is copied and stored on the target storage node so that the second target data shard can be recovered using this replica data when needed later. Alternatively, the second target data shard can be recovered first using this first target replica data, and then the recovered second target data shard can be stored on the target storage node. The formula for recovering a faulty shard using replicas is as follows:

[0208] , This is a copy of the data for the faulty shard.

[0209] Of course, if not all replicas of the second target data shard are found, meaning the found first target replicas include only a portion of the second target data shard, then the second target data shard needs to be recovered using erasure coding, and the recovered second target data shard stored on the target storage node. The formula for recovery using erasure coding is:

[0210] G is the encoding matrix, and V is a vector containing the partially retrieved copy data.

[0211] In addition, in this embodiment, the target storage node is the optimal storage node dynamically selected during the recovery process. Optionally, in this embodiment, the method further includes:

[0212] Obtain the indicator information of the recovery indicators for each storage node;

[0213] Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0214] In this way, by obtaining the metric information of the recovery metrics corresponding to the storage nodes, the optimal storage node can be selected in real time to perform recovery, thereby achieving the purpose of load balancing.

[0215] Recovery metrics include load, network latency, and version number.

[0216] In this embodiment, the Raft protocol is used to synchronize the recovery process, ensuring the consistency and efficiency of distributed storage.

[0217] Optionally, in this embodiment, the method further includes:

[0218] Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes;

[0219] In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0220] Therefore, version numbers can be used to resolve distributed storage node conflicts, ensuring data consistency of replicas of the same data shard across different storage nodes. The version number is set based on the data shard, using the following formula:

[0221] T update Data sharding D i The update time, NodeID is the data shard D i Storage node identifier, Data sharding D i The version number. The set version number can be stored on the storage node along with the replica data.

[0222] According to the version number setting formula, the version numbers of all replica data of the same data shard should be the same. When the version numbers of the replica data of the third target data shard on different storage nodes are inconsistent, the update time of the third target data shard may change, and some replica data may not be updated in time. Therefore, it is necessary to synchronize the replica data with the highest version number among these replica data on each storage node. That is, replace the replica data with the lower version number stored in the storage node with the replica data with the highest version number.

[0223] In one optional implementation, determining whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes includes: scanning all storage nodes and checking whether the version numbers of the replica data of the same data shard are consistent, where the third target data shard is any data shard stored in the distributed storage system. If a version number inconsistency is detected, a "version conflict" flag is set.

[0224] For data shards with "version conflicts," the version numbers of all their replicas are compared, and the replica with the highest version number is selected as the valid replica. A distributed consensus protocol (such as Paxos or Raft) is then used to synchronize the valid replica to all relevant storage nodes, ensuring that the version numbers of the replicas are consistent across all storage nodes. At this point, the updated replica data and the corresponding storage node can be recorded on the blockchain.

[0225] In summary, the method in this application uses consistent hashing to achieve dynamic data mapping, supports the dynamic addition and removal of nodes, and reduces data migration overhead; it optimizes the verification process through Merkle trees and parallel computing, significantly reducing verification latency and resource consumption; and it employs dynamic replica management and fault monitoring to ensure that the system can recover quickly in the event of node failure.

[0226] like Figure 2 As shown, a data processing apparatus 200 according to an embodiment of this application includes:

[0227] The first processing module 210 is used to determine the feature value of each data segment of the target data;

[0228] The second processing module 220 is used to determine the characteristic value of each storage node;

[0229] The third processing module 230 is used to map each data shard and each storage node to a hash space based on the feature value of each data shard and the feature value of each storage node;

[0230] Transmission module 240 is used to transmit each of the data fragments to their respective corresponding storage nodes;

[0231] The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0232] This device can map both data shards and storage nodes to a hash space using their characteristic values, establishing a correspondence between them and enabling the storage of data shards to the corresponding storage nodes. This way, only partial data migration is required when a node joins or leaves, reducing network overhead and latency.

[0233] Optionally, the device further includes:

[0234] The fourth processing module is used to determine multiple data fragments of the target data based on the target data and storage node information.

[0235] Optionally, the device further includes:

[0236] The fifth processing module is used to generate and store verification information based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0237] Optionally, the fifth processing module is further configured to:

[0238] A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation.

[0239] The root feature value is stored in the blockchain as the verification information.

[0240] Optionally, the device further includes:

[0241] The sixth processing module is used for

[0242] When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree;

[0243] The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information.

[0244] If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

[0245] Optionally, the device further includes:

[0246] The seventh processing module is used to cache the path information after the user has completed downloading the first target data segment;

[0247] The eighth processing module is used to update the path information to path feature values ​​when the access frequency of the path information is greater than a preset threshold. The path feature values ​​are obtained by compressing the path information.

[0248] Optionally, the device further includes:

[0249] The ninth processing module is used for

[0250] Based on the storage node information, determine the number of replicas for each data shard;

[0251] Based on the number of replicas for each data shard, generate replica data;

[0252] The copy data of each data shard is transmitted to the corresponding storage node.

[0253] Optionally, the device further includes:

[0254] The tenth processing module is used for

[0255] Monitor the operational status of all storage nodes;

[0256] If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard.

[0257] If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node;

[0258] If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0259] Optionally, the device further includes:

[0260] The eleventh processing module is used for

[0261] Obtain the indicator information of the recovery indicators for each storage node;

[0262] Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0263] Optionally, the device further includes:

[0264] The twelfth processing module is used for

[0265] Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes;

[0266] In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0267] The apparatus of this application embodiment is an apparatus that applies the above-described data processing method. The implementation of the above-described method embodiment is applicable to this apparatus and can achieve the same technical effect.

[0268] like Figure 3 As shown, a data processing device 300 according to an embodiment of this application includes a processor 310 and a transceiver 320, wherein the processor is used for:

[0269] Determine the feature values ​​of each data slice of the target data;

[0270] Determine the characteristic values ​​of each storage node;

[0271] Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space;

[0272] The transceiver is used to transmit each data fragment to its corresponding storage node.

[0273] The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0274] This device can map data shards and storage nodes to a hash space using their characteristic values, establishing a correspondence between them and enabling the storage of data shards to the corresponding storage nodes. This way, only partial data migration is required when a node joins or leaves, reducing network overhead and latency.

[0275] Optionally, the processor 310 is further configured to:

[0276] Based on the target data and storage node information, multiple data fragments of the target data are determined.

[0277] Optionally, the processor 310 is further used for

[0278] Verification information is generated and stored based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0279] Optionally, the processor 310 is further used for

[0280] A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation.

[0281] The root feature value is stored in the blockchain as the verification information.

[0282] Optionally, the processor 310 is further used for

[0283] When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree;

[0284] The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information.

[0285] The transceiver 320 is also used to transmit the first target data fragment to the user if the verification of the first target data fragment passes.

[0286] Optionally, the processor 310 is further used for

[0287] If the user completes the download of the first target data segment, the path information will be cached; or,

[0288] If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

[0289] Optionally, the processor 310 is further used for

[0290] Based on the storage node information, determine the number of replicas for each data shard;

[0291] Based on the number of replicas for each data shard, generate replica data;

[0292] The copy data of each data shard is transmitted to the corresponding storage node.

[0293] Optionally, the processor 310 is further used for

[0294] Monitor the operational status of all storage nodes;

[0295] If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard.

[0296] If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node;

[0297] If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0298] Optionally, the processor 310 is further used for

[0299] Obtain the indicator information of the recovery indicators for each storage node;

[0300] Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0301] Optionally, the processor 310 is further used for

[0302] Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes;

[0303] In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0304] It should be noted that, as Figure 4As shown, the data processing device in this embodiment can be applied to a distributed storage system, which further includes multiple storage nodes and a blockchain. The storage nodes are distributed devices used to store data shards and replicas.

[0305] Another embodiment of the data processing device of this application, such as Figure 5 As shown, it includes a transceiver 510, a processor 500, a memory 520, and a program or instructions stored in the memory 520 and executable on the processor 500; when the processor 500 executes the program or instructions, it implements the above-described data processing method.

[0306] The transceiver 510 is used to receive and send data under the control of the processor 500.

[0307] Among them, Figure 5 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 during operation.

[0308] An embodiment of this application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the data processing method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0309] The processor is the processor in the data processing device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0310] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0311] It should be further noted that many of the functional components described in this specification are referred to as modules in order to emphasize the independence of their implementation.

[0312] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0313] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0314] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0315] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or components. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0316] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Based on the public data and storage node information, the public data is divided into multiple data fragments; Determine the feature value for each data segment, where the feature value is a unique identifier for the data segment; Determine the characteristic values ​​of each storage node; Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space; Each of the data fragments is transmitted to its corresponding storage node. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space. The method further includes: When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in a binary tree; the binary tree is constructed based on the feature value of each data shard, and the root feature value of the binary tree is used as verification information to verify the integrity of the data; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

2. The method according to claim 1, characterized in that, The root eigenvalues ​​of the binary tree are determined through parallel computation. The verification information is stored on the blockchain.

3. The method according to claim 1, characterized in that, Also includes: Once the user has completed downloading the first target data segment, the path information will be cached. or, If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

4. The method according to claim 1, characterized in that, Also includes: Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

5. The method according to claim 4, characterized in that, Also includes: Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

6. The method according to claim 5, characterized in that, Also includes: Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

7. The method according to claim 4, characterized in that, Also includes: Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

8. A data processing apparatus, characterized in that, include: The first processing module is used to divide the public data into multiple data fragments based on the public data and storage node information; In addition, a feature value is determined for each of the data shards, the feature value being a unique identifier for the data shard; The second processing module is used to determine the characteristic values ​​of each storage node; The third processing module is used to map each data shard and each storage node to a hash space based on the feature value of each data shard and the feature value of each storage node. The transmission module is used to transmit each of the data fragments to their respective corresponding storage nodes. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space. The device further includes: When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in a binary tree; the binary tree is constructed based on the feature value of each data shard, and the root feature value of the binary tree is used as verification information to verify the integrity of the data; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

9. A data processing device, characterized in that, Includes a processor and a transceiver, the processor being used for: Based on the public data and storage node information, the public data is divided into multiple data fragments; Determine the feature value for each data segment, where the feature value is a unique identifier for the data segment; Determine the characteristic values ​​of each storage node; Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space; The transceiver is used to transmit each data fragment to its corresponding storage node. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space. When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in a binary tree; the binary tree is constructed based on the feature value of each data shard, and the root feature value of the binary tree is used as verification information to verify the integrity of the data; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

10. A data processing device, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the data processing method as described in any one of claims 1-7.

11. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the data processing method as described in any one of claims 1-7.

Images