Data processing method and device and readable storage medium

Abstract

The invention provides a data processing method and device and a readable storage medium, and relates to the technical field of communication. The method comprises the following steps: determining a characteristic value of each data fragment of target data; determining a characteristic value of each storage node; according to the characteristic value of each data fragment and the characteristic value of each storage node, mapping each data fragment and each storage node to a hash space; respectively transmitting each data fragment to a respective corresponding storage node; wherein the storage nodes are distributed storage nodes, and the data fragments and the corresponding storage nodes are mapped to the same position of the hash space.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data processing method, apparatus and readable storage medium. Background Technology

[0002] In the era of big data, the scale of data is growing exponentially, and traditional centralized storage systems can no longer meet the needs for efficient, secure, and scalable storage. Distributed storage, by distributing data across multiple nodes, not only improves the system's fault tolerance and scalability but also effectively addresses the challenges of high-concurrency access and massive data processing.

[0003] Currently, distributed storage systems typically employ a centralized architecture, relying on metadata servers to manage data distribution and consistency. However, centralized architectures have poor scalability; adding new nodes requires frequent communication with the metadata server, increasing network overhead and latency. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method, apparatus, and readable storage medium to avoid the need for frequent communication between new nodes and metadata servers in existing discrete storage systems, which increases network overhead and latency.

[0005] To achieve the above objectives, embodiments of this application provide a data processing method, including: Determine the feature values ​​of each data slice of the target data; Determine the characteristic values ​​of each storage node; Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space; Each of the data fragments is transmitted to its corresponding storage node. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0006] Optionally, the method further includes: Based on the target data and storage node information, multiple data fragments of the target data are determined.

[0007] Optionally, the method further includes: Verification information is generated and stored based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0008] Optionally, generating and storing verification information based on the feature values ​​of each data segment includes: A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation. The root feature value is stored in the blockchain as the verification information.

[0009] Optionally, the method further includes: When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

[0010] Optionally, the method further includes: If the user completes the download of the first target data segment, the path information will be cached; or, If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

[0011] Optionally, the method further includes: Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

[0012] Optionally, the method further includes: Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0013] Optionally, the method further includes: Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0014] Optionally, the method further includes: Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0015] To achieve the above objectives, embodiments of this application provide a data processing apparatus, including: The first processing module is used to determine the feature values ​​of each data segment of the target data; The second processing module is used to determine the characteristic values ​​of each storage node; The third processing module is used to map each data shard and each storage node to a hash space based on the feature value of each data shard and the feature value of each storage node. The transmission module is used to transmit each of the data fragments to their respective corresponding storage nodes. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0016] Optionally, the device further includes: The fourth processing module is used to determine multiple data fragments of the target data based on the target data and storage node information.

[0017] Optionally, the device further includes: The fifth processing module is used to generate and store verification information based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0018] Optionally, the fifth processing module is further configured to: A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation. The root feature value is stored in the blockchain as the verification information.

[0019] Optionally, the device further includes: The sixth processing module is used for When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

[0020] Optionally, the device further includes: The seventh processing module is used to cache the path information after the user has completed downloading the first target data segment; The eighth processing module is used to update the path information to path feature values ​​when the access frequency of the path information is greater than a preset threshold. The path feature values ​​are obtained by compressing the path information.

[0021] Optionally, the device further includes: The ninth processing module is used for Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

[0022] Optionally, the device further includes: The tenth processing module is used for Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0023] Optionally, the device further includes: The eleventh processing module is used for Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0024] Optionally, the device further includes: The twelfth processing module is used for Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0025] To achieve the above objectives, embodiments of this application provide a data processing device, including a processor and a transceiver, wherein the processor is used for: Determine the feature values ​​of each data slice of the target data; Determine the characteristic values ​​of each storage node; Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space; The transceiver is used to transmit each data fragment to its corresponding storage node. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0026] Optionally, the processor is further configured to: Based on the target data and storage node information, multiple data fragments of the target data are determined.

[0027] Optionally, the processor is also used for Verification information is generated and stored based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0028] Optionally, the processor is also used for A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation. The root feature value is stored in the blockchain as the verification information.

[0029] Optionally, the processor is also used for When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. The transceiver is also used to transmit the first target data fragment to the user if the verification of the first target data fragment passes.

[0030] Optionally, the processor is also used for If the user completes the download of the first target data segment, the path information will be cached; or, If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

[0031] Optionally, the processor is also used for Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

[0032] Optionally, the processor is also used for Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0033] Optionally, the processor is also used for Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0034] Optionally, the processor is also used for Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0035] To achieve the above objectives, embodiments of this application provide a data processing device, including: a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the data processing method described above.

[0036] To achieve the above objectives, embodiments of this application provide a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implement the steps in the data processing method described above.

[0037] To achieve the above objectives, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0038] The beneficial effects of the above technical solution in this application are as follows: The method in this application can map both data shards and storage nodes to a hash space using the feature values ​​of the data shards and the feature values ​​of the storage nodes, establishing a correspondence between data shards and storage nodes, thereby enabling the storage of data shards on the corresponding storage nodes. This way, only partial data migration is required when a node joins or leaves, reducing network overhead and latency. Attached Figure Description

[0039] Figure 1 This is a flowchart of a data processing method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the data processing apparatus according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the data processing device according to an embodiment of this application; Figure 4 This is a schematic diagram of a distributed storage system in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a data processing device according to another embodiment of this application. Detailed Implementation

[0040] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0041] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0042] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0043] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0044] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A, but can also be determined based on A and / or other information.

[0045] For ease of understanding, the following describes some aspects of the embodiments of this application: Research on distributed storage technology has significant practical implications, especially in the era of big data, where data volumes are growing exponentially. Traditional centralized storage systems can no longer meet the demands for efficient, secure, and scalable storage. Distributed storage, by distributing data across multiple nodes, not only improves the system's fault tolerance and scalability but also effectively addresses the challenges of high-concurrency access and massive data processing. However, with the increasing demand for public data sharing and openness, distributed storage systems face numerous challenges in terms of data security and integrity verification.

[0046] Traditional distributed storage systems typically employ a centralized architecture, relying on metadata servers to manage data distribution and consistency. The metadata server stores file metadata (such as file paths and shard locations) and simplifies data location and access through centralized management. When a client accesses data, it first requests the file's shard location information from the metadata server and then communicates directly with the storage nodes to retrieve the data. This architecture improves data access efficiency through centralized metadata management, performing particularly well in small-scale data scenarios. The Hadoop Distributed File System (HDFS) uses a master-slave architecture, with a single master node (NameNode) managing metadata and multiple slave nodes (DataNodes) storing the actual data. Ceph, on the other hand, dynamically calculates data distribution using the CRUSH algorithm, reducing reliance on the metadata server, but still requires a centralized monitor node to maintain the cluster state. This approach offers high efficiency in data management and access control, making it suitable for scenarios requiring strong consistency and centralized management.

[0047] Blockchain storage systems employ a decentralized peer-to-peer network structure, completely eliminating the design of centralized nodes. Data is sharded and distributed across multiple nodes in the network, with each node acting as both a data provider and consumer. The InterPlanetary File System (IPFS) uses content addressing technology to shard files and generate unique hash values ​​as identifiers. Users can retrieve data directly from the network using these hash values, without relying on centralized servers. The Storj system utilizes blockchain technology to record the distribution information of data shards and storage contracts, ensuring data traceability and security. Decentralized storage systems achieve data location through distributed hash tables, and nodes exchange data directly through peer-to-peer communication, avoiding single points of failure. This technological approach offers high resistance to attacks and data redundancy, making it suitable for scenarios requiring high reliability and decentralized management.

[0048] like Figure 1 As shown in the figure, a data processing method according to an embodiment of this application includes: Step 101: Determine the feature values ​​of each data segment of the target data; Step 102: Determine the characteristic values ​​of each storage node; Step 103: Based on the feature value of each data shard and the feature value of each storage node, map each data shard and each storage node to a hash space; Step 104: Transmit each of the data fragments to its corresponding storage node; The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0049] Following the steps described above, the method of this application embodiment can map both data shards and storage nodes to a hash space using the feature values ​​of the data shards and the feature values ​​of the storage nodes, establishing a correspondence between data shards and storage nodes, thereby enabling the storage of data shards to the corresponding storage nodes. In this way, only partial data migration is required when a node joins or leaves, reducing network overhead and latency.

[0050] Moreover, the mapping implemented using feature values ​​in this application can quickly verify whether data fragments have been tampered with, avoiding the need to verify the entire data fragment content byte by byte.

[0051] Optionally, in this embodiment, the hash space is a ring structure with the first and last ends connected.

[0052] Optionally, in this embodiment, the target data is public data, in order to improve the performance, reliability, and scalability of the public data distributed storage system.

[0053] Optionally, in this embodiment, the characteristic value of the data shard is a hash value, such as data shard D. i The eigenvalues ​​can be denoted as H(D) i ) or H i H ( The hash function can be SHA-256 or other algorithms. The characteristic value of a data fragment can serve as its unique identifier.

[0054] Optionally, in this embodiment, the characteristic value of the storage node is a hash value, which can be obtained by hashing the name or identifier (ID) of the storage node.

[0055] Optionally, the method further includes: Based on the target data and storage node information, multiple data fragments of the target data are determined.

[0056] Here, storage node information is used to indicate the scale of storage nodes, such as the number of storage nodes and network topology. Thus, by combining the target data and the storage node information, the target data is divided into multiple data fragments.

[0057] In one optional implementation, the target data is evenly divided into n data shards of equal size based on the number of storage nodes, such as: .in, The total amount of target data, The size of each data shard. This represents the number of storage nodes.

[0058] Distributing the target data evenly avoids frequent adjustments to data shard sizes, dynamically balances storage pressure, and resolves storage performance bottlenecks caused by excessively large or small data blocks. The choice of 'n' can also be dynamically adjusted based on the total data volume and node scale, ensuring that the size of each shard is within a reasonable range to optimize storage and access performance.

[0059] After the target data is evenly divided, the data is divided into D pieces. i Mapping to storage nodes, since data sharding and storage nodes are mapped to hash space, not only does it support the dynamic addition and removal of nodes, reducing the data migration overhead caused by node changes, but it also optimizes the uniformity of data distribution and avoids data skew problems.

[0060] In this process, data sharding is mapped to a hash space, which can employ consistent hashing, such as... , where H ( H(D) is a hash function, which can be an algorithm such as SHA-256. i ) is data sharding D i The characteristic value is N, where N is the number of storage nodes. It is data sharding D i Mapping information.

[0061] Optionally, if the target data needs to be stored in shards, the target data can be sharded. After sharding, it can be checked whether each data shard was successfully sharded, such as whether there are any data shards that are lost or duplicated.

[0062] Optionally, in this embodiment, each data shard is transmitted to the corresponding storage node via a secure transmission protocol. The secure transmission protocol can be HTTPS or an encrypted tunnel.

[0063] Optionally, in this embodiment, after each data shard is transmitted to its corresponding storage node, distribution information can be recorded on the blockchain. This distribution information can be the nodes where each data shard is stored.

[0064] Optionally, in this embodiment, the method further includes: Verification information is generated and stored based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0065] In this way, for the target data, verification information can be generated from the feature values ​​of each data fragment to ensure that the data fragments downloaded by the user are complete and tamper-proof original data, thus avoiding access to all data fragments.

[0066] Optionally, after each data fragment of the target data is stored in its corresponding storage node, these data fragments are extracted from each storage node, their feature values ​​are further determined, and then verification information is generated. The determined feature values ​​can be hash values, obtained using a hash function such as the SHA-256 algorithm.

[0067] Optionally, in this embodiment, generating and storing verification information based on the feature values ​​of each data segment includes: A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation. The root feature value is stored in the blockchain as the verification information.

[0068] That is, the feature values ​​of the parent nodes are calculated layer by layer according to the binary tree structure until the root feature value is generated. The root feature value is then used as verification information to check the integrity of the data. In this process, the use of a binary tree can compress the amount of information in the verification information, while the parallel calculation of the root feature value can effectively reduce the latency of large-scale data verification.

[0069] For example, if the feature value is a hash value, a binary tree node only has two child nodes, left and right. H left H is the left child node. right The right child node is represented by ||, which is the string concatenation operation. root This is the root hash value.

[0070] and T Merkle Let P be the time for parallel computation, Depth be the tree depth, and P be the number of parallel processing units. Thus, the more parallel processing units there are, the shorter the computation time for the root eigenvalue. For a binary tree, P=2.

[0071] In one alternative implementation, the binary tree is a Merkle tree.

[0072] Optionally, in this embodiment, the method further includes: When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

[0073] In other words, when a user requests to obtain the first target data fragment (which can also be understood as the first data fragment), the integrity of the first target data fragment will be verified based on the verification information, the feature value of the first target data fragment, and the path information. Only after the verification of the first target data fragment passes will the first target data fragment be transmitted to the user.

[0074] Optionally, verifying the integrity of the first target data fragment based on the verification information, the feature value of the first target data fragment, and the path information includes: Based on the feature values ​​of the first target data segment and the path information, the information to be verified is determined; Verify whether the verification information and the information to be verified are consistent. If they are consistent, the verification passes; otherwise, the verification fails.

[0075] In one alternative implementation, taking the feature value as a hash value as an example, the process of determining the sibling nodes s1 and s2 based on the path information can be represented as follows: , where H i H is the hash value of the first target data shard. s1 H s2 H is the hash value of the sibling node. root This is the root hash value.

[0076] Of course, the device that applies the method of the embodiments of this application can also send the first target data fragment and the corresponding path information to the user when the user requests the download of the first target data fragment, so that the user can verify the integrity of the data, wherein the user obtains the verification information from the blockchain.

[0077] Optionally, in this embodiment, the method further includes: If the user completes the download of the first target data segment, the path information will be cached; or, If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

[0078] Thus, in this embodiment of the application, on the one hand, the most recently verified path information P will be cached. i This allows for direct reuse of the cache during repeated verification, reducing the overhead of redundant calculations. Furthermore, for path information accessed more frequently than a preset threshold, it can be updated with compressed path feature values, improving the efficiency of path access.

[0079] In one alternative implementation, Huffman coding is used to compress the path and reduce its length, as shown in the formula: , where pi L represents the probability of accessing path information. path This represents the compressed path length.

[0080] Optionally, in this embodiment, the method further includes: Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

[0081] Thus, after determining the number of replicas for each data shard based on the storage node information, a corresponding number of replica data can be generated and transmitted to the storage nodes to ensure normal operation even in the event of node failure. The number of replicas can be dynamically adjusted based on the storage node information to balance redundant storage costs and fault tolerance.

[0082] In one alternative implementation, the number of replicas for each data shard can be determined using the following formula: r is the number of replicas of a data shard, and n is the number of storage nodes, ensuring that the system can still operate normally even with the failure of a fault-tolerant node.

[0083] For data fragment D i After determining the number of replicas r, data shards D can be generated. i The data is processed into r replicas, and then the r replicas are transferred to other storage nodes.

[0084] Optionally, in this embodiment, the characteristic values ​​of the replica data are also used to map each replica data to a hash space, and the storage nodes are also mapped to the hash space, so that the replica data can be transmitted to the storage nodes mapped to the same hash space.

[0085] In one optional implementation, consistent hashing is used to distribute replica data: j is the replica data sequence number, with a value range of j=0,1,…,r-1. Δ is the fixed step size. It is data sharding D i The location of the j-th sub-data that is mapped to the hash space.

[0086] Consistent hashing distributes replicas evenly, preventing them from being concentrated on a few nodes. Multi-hop hashing is achieved by dynamically optimizing the replica distribution through adjusting the step size Δ.

[0087] Optionally, in this embodiment, the replica data and the corresponding storage node are recorded in the blockchain.

[0088] Optionally, in this embodiment, the method further includes: Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0089] In this way, by monitoring the operating status of all storage nodes, data recovery can be carried out in a timely manner when a faulty storage node is detected, ensuring that the system can quickly return to normal when a node fails.

[0090] The monitoring method involves determining whether a heartbeat signal is received within a specified time. If not, the storage node is marked as "failed"; if so, it is marked as "healthy". The formula is as follows: T now T represents the current time. hearthbeat Δ is the time since the last monitoring. t Allowed monitoring latency. NodeID is the node identifier.

[0091] Heartbeat signals can be monitored based on a set heartbeat interval. Of course, the heartbeat interval can be dynamically adjusted to change the monitoring frequency, reducing unnecessary detection overhead.

[0092] In this application, data recovery can be performed using replicas or erasure coding. Specifically, when a faulty storage node is detected, based on the second target data shard on the faulty storage node, the first target replica data on the non-faulty storage node is queried. The first target replica data is a replica of the second target data shard. If all replica data of the second target data shard is found (i.e., the queried first target replica data includes all replica data of the second target data shard), this first target replica data is copied and stored on the target storage node so that the second target data shard can be recovered using this replica data when needed later. Alternatively, the second target data shard can be recovered first using this first target replica data, and then the recovered second target data shard can be stored on the target storage node. The formula for recovering a faulty shard using replicas is as follows: , This is a copy of the data for the faulty shard.

[0093] Of course, if not all replicas of the second target data shard are found, meaning the found first target replicas include only a portion of the second target data shard, then the second target data shard needs to be recovered using erasure coding, and the recovered second target data shard stored on the target storage node. The formula for recovery using erasure coding is: G is the encoding matrix, and V is a vector containing the partially retrieved copy data.

[0094] In addition, in this embodiment, the target storage node is the optimal storage node dynamically selected during the recovery process. Optionally, in this embodiment, the method further includes: Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0095] In this way, by obtaining the metric information of the recovery metrics corresponding to the storage nodes, the optimal storage node can be selected in real time to perform recovery, thereby achieving the purpose of load balancing.

[0096] Recovery metrics include load, network latency, and version number.

[0097] In this embodiment, the Raft protocol is used to synchronize the recovery process, ensuring the consistency and efficiency of distributed storage.

[0098] Optionally, in this embodiment, the method further includes: Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0099] Therefore, version numbers can be used to resolve distributed storage node conflicts, ensuring data consistency of replicas of the same data shard across different storage nodes. The version number is set based on the data shard, using the following formula: T update Data sharding D i The update time, NodeID is the data shard D i Storage node identifier, Data sharding D i The version number. The set version number can be stored on the storage node along with the replica data.

[0100] According to the version number setting formula, the version numbers of all replica data of the same data shard should be the same. When the version numbers of the replica data of the third target data shard on different storage nodes are inconsistent, the update time of the third target data shard may change, and some replica data may not be updated in time. Therefore, it is necessary to synchronize the replica data with the highest version number among these replica data on each storage node. That is, replace the replica data with the lower version number stored in the storage node with the replica data with the highest version number.

[0101] In one optional implementation, determining whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes includes: scanning all storage nodes and checking whether the version numbers of the replica data of the same data shard are consistent, where the third target data shard is any data shard stored in the distributed storage system. If a version number inconsistency is detected, a "version conflict" flag is set.

[0102] For data shards with "version conflicts," the version numbers of all their replicas are compared, and the replica with the highest version number is selected as the valid replica. A distributed consensus protocol (such as Paxos or Raft) is then used to synchronize the valid replica to all relevant storage nodes, ensuring that the version numbers of the replicas are consistent across all storage nodes. At this point, the updated replica data and the corresponding storage node can be recorded on the blockchain.

[0103] In summary, the method in this application uses consistent hashing to achieve dynamic data mapping, supports the dynamic addition and removal of nodes, and reduces data migration overhead; it optimizes the verification process through Merkle trees and parallel computing, significantly reducing verification latency and resource consumption; and it employs dynamic replica management and fault monitoring to ensure that the system can recover quickly in the event of node failure.

[0104] like Figure 2 As shown, a data processing apparatus 200 according to an embodiment of this application includes: The first processing module 210 is used to determine the feature value of each data segment of the target data; The second processing module 220 is used to determine the characteristic value of each storage node; The third processing module 230 is used to map each data shard and each storage node to a hash space based on the feature value of each data shard and the feature value of each storage node; Transmission module 240 is used to transmit each of the data fragments to their respective corresponding storage nodes; The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0105] This device can map both data shards and storage nodes to a hash space using their characteristic values, establishing a correspondence between them and enabling the storage of data shards to the corresponding storage nodes. This way, only partial data migration is required when a node joins or leaves, reducing network overhead and latency.

[0106] Optionally, the device further includes: The fourth processing module is used to determine multiple data fragments of the target data based on the target data and storage node information.

[0107] Optionally, the device further includes: The fifth processing module is used to generate and store verification information based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0108] Optionally, the fifth processing module is further configured to: A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation. The root feature value is stored in the blockchain as the verification information.

[0109] Optionally, the device further includes: The sixth processing module is used for When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

[0110] Optionally, the device further includes: The seventh processing module is used to cache the path information after the user has completed downloading the first target data segment; The eighth processing module is used to update the path information to path feature values ​​when the access frequency of the path information is greater than a preset threshold. The path feature values ​​are obtained by compressing the path information.

[0111] Optionally, the device further includes: The ninth processing module is used for Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

[0112] Optionally, the device further includes: The tenth processing module is used for Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0113] Optionally, the device further includes: The eleventh processing module is used for Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0114] Optionally, the device further includes: The twelfth processing module is used for Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0115] The apparatus of this application embodiment is an apparatus that applies the above-described data processing method. The implementation of the above-described method embodiment is applicable to this apparatus and can achieve the same technical effect.

[0116] like Figure 3 As shown, a data processing device 300 according to an embodiment of this application includes a processor 310 and a transceiver 320, wherein the processor is used for: Determine the feature values ​​of each data slice of the target data; Determine the characteristic values ​​of each storage node; Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space; The transceiver is used to transmit each data fragment to its corresponding storage node. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

[0117] This device can map data shards and storage nodes to a hash space using their characteristic values, establishing a correspondence between them and enabling the storage of data shards to the corresponding storage nodes. This way, only partial data migration is required when a node joins or leaves, reducing network overhead and latency.

[0118] Optionally, the processor 310 is further configured to: Based on the target data and storage node information, multiple data fragments of the target data are determined.

[0119] Optionally, the processor 310 is further used for Verification information is generated and stored based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

[0120] Optionally, the processor 310 is further used for A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation. The root feature value is stored in the blockchain as the verification information.

[0121] Optionally, the processor 310 is further used for When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. The transceiver 320 is also used to transmit the first target data fragment to the user if the verification of the first target data fragment passes.

[0122] Optionally, the processor 310 is further used for If the user completes the download of the first target data segment, the path information will be cached; or, If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

[0123] Optionally, the processor 310 is further used for Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

[0124] Optionally, the processor 310 is further used for Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

[0125] Optionally, the processor 310 is further used for Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

[0126] Optionally, the processor 310 is further used for Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

[0127] It should be noted that, as Figure 4 As shown, the data processing device in this embodiment can be applied to a distributed storage system, which further includes multiple storage nodes and a blockchain. The storage nodes are distributed devices used to store data shards and replicas.

[0128] Another embodiment of the data processing device of this application, such as Figure 5 As shown, it includes a transceiver 510, a processor 500, a memory 520, and a program or instructions stored in the memory 520 and executable on the processor 500; when the processor 500 executes the program or instructions, it implements the above-described data processing method.

[0129] The transceiver 510 is used to receive and send data under the control of the processor 500.

[0130] Among them, Figure 5 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 during operation.

[0131] An embodiment of this application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the data processing method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0132] The processor is the processor in the data processing device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0133] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0134] It should be further noted that many of the functional components described in this specification are referred to as modules in order to emphasize the independence of their implementation.

[0135] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0136] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0137] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0138] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or components. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0139] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Determine the feature values ​​of each data slice of the target data; Determine the characteristic values ​​of each storage node; Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space; Each of the data fragments is transmitted to its corresponding storage node. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

2. The method according to claim 1, characterized in that, Also includes: Based on the target data and storage node information, multiple data fragments of the target data are determined.

3. The method according to claim 1, characterized in that, Also includes: Verification information is generated and stored based on the feature values ​​of each data segment, and the verification information is used to verify the integrity of the data.

4. The method according to claim 3, characterized in that, The step of generating and storing verification information based on the feature values ​​of each data segment includes: A binary tree is constructed based on the feature values ​​of each data segment; wherein the root feature value of the binary tree is determined through parallel computation. The root feature value is stored in the blockchain as the verification information.

5. The method according to claim 4, characterized in that, Also includes: When a user requests to obtain a first target data shard, the feature value and path information of the first target data shard are obtained; the path information includes the feature value of the sibling node of the first target data shard in the binary tree; The integrity of the first target data segment is verified based on the verification information, the feature value of the first target data segment, and the path information. If the verification of the first target data fragment passes, the first target data fragment is transmitted to the user.

6. The method according to claim 5, characterized in that, Also includes: Once the user has completed downloading the first target data segment, the path information will be cached. or, If the access frequency of the path information is greater than a preset threshold, the path information is updated to a path feature value, which is obtained by compressing the path information.

7. The method according to claim 1, characterized in that, Also includes: Based on the storage node information, determine the number of replicas for each data shard; Based on the number of replicas for each data shard, generate replica data; The copy data of each data shard is transmitted to the corresponding storage node.

8. The method according to claim 7, characterized in that, Also includes: Monitor the operational status of all storage nodes; If a faulty storage node is detected, the first target replica data on the non-faulty storage node is queried based on the second target data shard on the faulty storage node. The first target replica data is the replica data of the second target data shard. If all replicas of the second target data shard are found, then the first target replica data is copied and stored to the target storage node; If not all replicas of the second target data shard are found, the second target data shard is restored using erasure coding based on the first target replica data found, and then stored in the target storage node.

9. The method according to claim 8, characterized in that, Also includes: Obtain the indicator information of the recovery indicators for each storage node; Based on the aforementioned indicator information, storage nodes that meet the preset conditions are identified as the target storage nodes.

10. The method according to claim 7, characterized in that, Also includes: Determine whether the version numbers of the replica data of the third target data shard are consistent across different storage nodes; In the event of inconsistent version numbers, the replica data with the highest version number is synchronized across all storage nodes based on the version number of the replica data of the third target data shard.

11. A data processing apparatus, characterized in that, include: The first processing module is used to determine the feature values ​​of each data segment of the target data; The second processing module is used to determine the characteristic values ​​of each storage node; The third processing module is used to map each data shard and each storage node to a hash space based on the feature value of each data shard and the feature value of each storage node. The transmission module is used to transmit each of the data fragments to their respective corresponding storage nodes. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

12. A data processing device, characterized in that, Includes a processor and a transceiver, the processor being used for: Determine the feature values ​​of each data slice of the target data; Determine the characteristic values ​​of each storage node; Based on the feature values ​​of each data shard and each storage node, each data shard and each storage node are mapped to a hash space; The transceiver is used to transmit each data fragment to its corresponding storage node. The storage node is a distributed storage node, and the data shards and their corresponding storage nodes are mapped to the same location in the hash space.

13. A data processing device, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the data processing method as described in any one of claims 1-10.

14. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the data processing method as described in any one of claims 1-10.

Images