A data distributed storage data error detection method and system

Abstract

The present application relates to a data distributed storage data error detection method and system, the method comprises the following steps: screening each network node according to the set condition, obtaining N target network nodes; N redundant data packets are generated, and the redundant data packets are stored in the target network nodes one by one; in the data reading stage, when the data segment of the data packet is sent out by the target network node, the passed data segment is verified with the redundant data packet, if consistent, the data segment is transmitted to the next node, otherwise, the data segment is repaired and then transmitted to the next node; when data repair occurs in the target network node, the target network node sends a data repair log to the distributed storage server; the redundant data is loaded into the N target network nodes selected, and data verification and detection are carried out during the data segment transmission process, which can not only detect data errors, but also save a large amount of storage space resources.

Description

Technical Field

[0001] This invention relates to the field of distributed data storage technology, and more specifically, to a method and system for detecting data errors in distributed data storage. Background Technology

[0002] In distributed storage systems, data loss and unavailability are common due to network transmission, human factors, and hardware failures. Data redundancy is typically used to combat these errors, repairing data when errors occur. Error detection usually relies on analyzing data logs, identifying errors based on log entries and then using redundant data for repair. While data logs effectively monitor data changes, their size increases dramatically over time, consuming significant storage resources. Although algorithms have been implemented to identify and clear useless log entries, the fundamental problem remains unresolved. Therefore, a data error detection method and system for distributed storage is needed to address this issue. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a data error detection method for distributed data storage, and a data error detection system for distributed data storage, in view of the above-mentioned defects of the prior art.

[0004] The technical solution adopted by this invention to solve its technical problem is:

[0005] A data error detection method for distributed data storage is constructed, comprising the following steps:

[0006] Step 1: Receive user data storage requests, and after successful verification, receive the user's stored data, divide the stored data into multiple sub-data packets, and perform distributed storage.

[0007] Step 2: Filter each network node in the sub-data packet access path according to the set conditions to obtain N target network nodes;

[0008] Step 3: Copy each sub-data packet in the distributed storage to generate N redundant data packets, and store the redundant data packets one by one into the target network node;

[0009] Step 4: During the data reading phase, when a data segment of a data packet is sent out by the target network node, the data segment is checked against the redundant data packet. If they match, the data segment is transmitted to the next node; otherwise, the data segment is repaired before being transmitted to the next node.

[0010] Step 5: When data repair occurs in the target network node, the target network node sends a data repair log to the distributed storage server.

[0011] The data error detection method for distributed data storage according to the present invention further includes the following steps:

[0012] Step Six: After receiving the data repair log, the distributed storage server calls the redundant data packets in the corresponding target network node to repair the data packets in the sub-data packets.

[0013] The data error detection method for distributed data storage according to the present invention further includes, in step six, a method:

[0014] After successfully repairing the sub-data packet, delete the data repair log.

[0015] The data error detection method for distributed data storage described in this invention includes a plurality of target network nodes that detect different data segments, and the combination of the data segments detected by the plurality of target network nodes constitutes the data accessed during the data access phase.

[0016] The data error detection method for distributed data storage according to the present invention further includes, in step four, a method:

[0017] During the data modification phase, when the data segment modifying the sub-data packet passes through the target network node, the passed data segment is compared and replaced with the redundant data packet, thus completing the update of the redundant data packet of the target network node.

[0018] The data error detection method for distributed data storage described in this invention, wherein the network nodes in the sub-data packet access path are filtered according to set conditions to obtain N target network nodes, employs the following method:

[0019] The overall performance of each network node is ranked, and N target network nodes are selected from high to low according to the ranking.

[0020] A data distributed storage error detection system is applied to the data distributed storage error detection method described above, wherein the system includes a distributed storage server, multiple network nodes, and multiple storage terminal devices;

[0021] The distributed storage server receives user data storage requests, and after verification, receives the user's stored data, divides the stored data into multiple sub-data packets, and performs distributed storage on multiple storage devices.

[0022] The distributed storage server is also used to filter each network node in the sub-data packet access path according to set conditions to obtain N target network nodes; to copy each sub-data packet in the distributed storage to generate N redundant data packets, and to store the redundant data packets one by one into the target network nodes; during the data reading stage, when the data fragment of the data packet is sent out through the target network node, the data fragment is checked with the redundant data packet. If they match, the data fragment is transmitted to the next node; otherwise, the data fragment is repaired before being transmitted to the next node.

[0023] When data repair occurs, the target network node sends a data repair log to the distributed storage server.

[0024] The distributed data storage error detection system of the present invention, wherein, after receiving the data repair log, the distributed storage server calls redundant data packets in the corresponding target network node to repair the sub-data packets; after successfully repairing the sub-data packets, the data repair log is deleted.

[0025] The distributed data storage error detection system of the present invention includes a plurality of target network nodes that detect different data segments, and the combination of the data segments detected by the plurality of target network nodes constitutes the data accessed during the data access phase.

[0026] The distributed data storage error detection system of the present invention includes a data modification phase in which, when a data segment modifying a sub-data packet passes through a target network node, the data segment is compared and replaced with a redundant data packet to update the redundant data packet of the target network node.

[0027] The beneficial effects of this invention are as follows: By applying the method of this application, redundant data is loaded into N selected target network nodes, and data verification and detection are performed during the data segment transmission process. This not only enables targeted data error detection, but also saves a lot of storage space resources. In addition, since the selected target network nodes are used, the impact on the overall data transmission efficiency is also minimal. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort:

[0029] Figure 1 This is a flowchart of a data error detection method for distributed data storage according to a preferred embodiment of the present invention;

[0030] Figure 2 This is a block diagram illustrating the principle of a data distributed storage data error detection system according to a preferred embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0032] The preferred embodiment of the data distributed storage data error detection method of the present invention, such as... Figure 1 As shown, it includes the following steps:

[0033] S01: Receive user data storage request, and after successful verification, receive user storage data, divide the storage data into multiple sub-data packets, and perform distributed storage; this step can use the existing distributed storage steps and will not be improved.

[0034] S02: Filter each network node in the sub-data packet access path according to the set conditions to obtain N target network nodes; the filtering can be done by: sorting the comprehensive performance of each network node, and selecting N target network nodes from high to low according to the sorting to minimize the impact on data transmission. Of course, relatively idle network nodes can also be selected. The purpose of the two methods is the same.

[0035] S03: Copy each sub-data packet in the distributed storage to generate N redundant data packets, and store the redundant data packets one by one in the target network node; it is understandable that the content of the redundant data packets generated by this step is consistent with the content of the sub-data packets, and will change as the content of the sub-data packets changes, so as to facilitate subsequent correction work.

[0036] S04: During the data reading phase, when a data segment of a data packet is sent out by a target network node, the data segment is checked against the redundant data packet. If they match, the data segment is transmitted to the next node; otherwise, the data segment is repaired before being transmitted to the next node. Preferably, the data segments detected by multiple target network nodes are different, and the combination of the data segments detected by multiple target network nodes is the data accessed during the data access phase.

[0037] During the data modification phase, when the data segment modifying the sub-data packet passes through the target network node, the data segment passing through is compared and replaced with the redundant data packet, thus completing the update of the redundant data packet of the target network node.

[0038] S05: When data repair occurs in the target network node, the target network node sends a data repair log to the distributed storage server;

[0039] S06: After receiving the data repair log, the distributed storage server calls the redundant data packets in the corresponding target network node to repair the data packets in the sub-data packets; after successfully repairing the data packets in the sub-data packets, the data repair log is deleted.

[0040] By applying the method of this application, redundant data is loaded into N selected target network nodes, and data verification and detection are performed during the data segment transmission process. This not only enables targeted data error detection but also saves a significant amount of storage space resources. Furthermore, since the selected target network nodes are used, the impact on the overall data transmission efficiency is minimal.

[0041] A data error detection system for distributed data storage, applied to the data error detection method for distributed data storage as described above, such as... Figure 2 As shown, the system includes a distributed storage server 100, multiple network nodes 101, and multiple storage terminal devices 102;

[0042] Distributed storage server 100 receives user data storage requests, and after verification, receives user storage data, divides the storage data into multiple sub-data packets, and performs distributed storage on multiple storage terminal devices 102.

[0043] The distributed storage server 100 is also used to filter each network node 101 in the sub-data packet access path according to set conditions to obtain N target network nodes; to copy each sub-data packet of the distributed storage to generate N redundant data packets, and to store the redundant data packets one by one into the target network nodes; during the data reading stage, when the data fragment of the data packet is sent out through the target network node, the data fragment is checked with the redundant data packet. If they match, the data fragment is transmitted to the next node; otherwise, the data fragment is repaired before being transmitted to the next node.

[0044] When data repair occurs, the target network node sends a data repair log to the distributed storage server.

[0045] By applying the method of this application, redundant data is loaded into N selected target network nodes, and data verification and detection are performed during the data segment transmission process. This not only enables targeted data error detection but also saves a significant amount of storage space resources. Furthermore, since the selected target network nodes are used, the impact on the overall data transmission efficiency is minimal.

[0046] Preferably, after receiving the data repair log, the distributed storage server calls the redundant data packets in the corresponding target network node to repair the sub-data packets; after successfully repairing the sub-data packets, the data repair log is deleted; this allows for precise location of data errors and repair, which is a completely different approach from existing methods that rely on data logs, and can save a significant amount of storage space.

[0047] Preferably, the data segments detected by the multiple target network nodes are different, and the data segments detected by the multiple target network nodes are combined to form the data accessed in the data access stage; that is, multiple target network nodes are used to split the directional data (the splitting rules can be customized), and verification and repair are performed separately to distribute the data verification compliance evenly; of course, under normal circumstances, the probability of multiple errors is not very high, and the overall verification load is within an acceptable range.

[0048] Preferably, during the data modification stage, when the data segment modifying the sub-data packet passes through the target network node, the passed data segment is compared and replaced with the redundant data packet to complete the update of the redundant data packet in the target network node; this facilitates updating the redundant data packet in the target network node while updating the sub-data packet, simplifying the data update operation.

[0049] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for detecting data errors in distributed data storage, characterized in that, Includes the following steps: Step 1: Receive user data storage requests, and after successful verification, receive the user's stored data, divide the stored data into multiple sub-data packets, and perform distributed storage. Step 2: Filter each network node in the sub-data packet access path according to the set conditions to obtain N target network nodes; Step 3: Copy each sub-data packet in the distributed storage to generate N redundant data packets, and store the redundant data packets one by one into the target network node; Step 4: During the data reading phase, when a data segment of a data packet is sent out by the target network node, the data segment is checked against the redundant data packet. If they match, the data segment is transmitted to the next node; otherwise, the data segment is repaired before being transmitted to the next node. Step 5: When data repair occurs in the target network node, the target network node sends a data repair log to the distributed storage server; The data segments detected by the multiple target network nodes are different, and the combination of the data segments detected by the multiple target network nodes constitutes the data accessed during the data access phase.

2. The data error detection method for distributed data storage according to claim 1, characterized in that, The data error detection method for distributed data storage also includes the following steps: Step Six: After receiving the data repair log, the distributed storage server calls the redundant data packets in the corresponding target network node to repair the data packets in the sub-data packets.

3. The data error detection method for distributed data storage according to claim 2, characterized in that, Step six also includes a method: After successfully repairing the sub-data packet, delete the data repair log.

4. The data error detection method for distributed data storage according to claim 1, characterized in that, Step four also includes a method: During the data modification phase, when the data segment modifying the sub-data packet passes through the target network node, the passed data segment is compared and replaced with the redundant data packet, thus completing the update of the redundant data packet of the target network node.

5. The data error detection method for distributed data storage according to claim 1, characterized in that, The network nodes in the sub-data packet access path are filtered according to set conditions to obtain N target network nodes. The overall performance of each network node is ranked, and N target network nodes are selected from high to low according to the ranking.

6. A data error detection system for distributed data storage, applied to the data error detection method for distributed data storage as described in any one of claims 1-5, characterized in that, The system includes a distributed storage server, multiple network nodes, and multiple storage terminal devices; The distributed storage server receives user data storage requests, and after verification, receives the user's stored data, divides the stored data into multiple sub-data packets, and performs distributed storage on multiple storage devices. The distributed storage server is also used to filter each network node in the sub-data packet access path according to set conditions to obtain N target network nodes; to copy each sub-data packet in the distributed storage to generate N redundant data packets, and to store the redundant data packets one by one into the target network nodes. During the data reading phase, when a data segment of a data packet is sent out by the target network node, the data segment is checked against the redundant data packet. If they match, the data segment is transmitted to the next node; otherwise, the data segment is repaired before being transmitted to the next node. When data repair occurs, the target network node sends a data repair log to the distributed storage server.

7. The data error detection system for distributed data storage according to claim 6, characterized in that, After receiving the data repair log, the distributed storage server calls the redundant data packets in the corresponding target network node to repair the data packets in the sub-data packets; after successfully repairing the data packets in the sub-data packets, it deletes the data repair log.

8. The data error detection system for distributed data storage according to claim 6, characterized in that, The data segments detected by the multiple target network nodes are different, and the combination of the data segments detected by the multiple target network nodes constitutes the data accessed during the data access phase.

9. The data error detection system for distributed data storage according to claim 6, characterized in that, During the data modification phase, when the data segment modifying the sub-data packet passes through the target network node, the passed data segment is compared and replaced with the redundant data packet, thus completing the update of the redundant data packet of the target network node.

Images