Data processing method and apparatus

Abstract

This specification provides a data processing method and apparatus. The data processing method includes: creating a verification node with the same hierarchical structure based on the attribute information of a system node, and storing the verification data in the verification node, wherein the verification data is generated based on first sub-data and second sub-data stored in the system node; when a failed node is detected in a distributed system, determining a target system node in the distributed system that has a repair relationship with the failed node; extracting target original data stored in the target system node and the verification data stored in the verification node; generating failed original data based on the target original data and the verification data, and storing it in the failed node.

Description

Technical Field

[0001] This specification relates to the field of computer storage technology, and in particular to data processing methods and apparatus. Background Technology

[0002] With the development of internet technology, social networks, video and file sharing, and cloud storage have become increasingly prevalent in people's daily lives. As the user base grows, the data generated by these products is increasing exponentially. However, traditional centralized storage cannot meet practical needs in terms of cost and reliability in the face of massive data storage challenges. Distributed storage systems, due to their high scalability, low cost, and high reliability, have been widely applied and researched in industry and academia. Furthermore, to prevent data loss, distributed storage systems have evolved from initially using replication technology for data fault tolerance to employing erasure coding for data storage, achieving high storage efficiency within a given fault tolerance capability. While existing distributed storage systems can improve database storage efficiency through erasure coding, the construction of erasure coding requires an exponential number of data packets, resulting in high complexity and resource consumption. Simultaneously, the repair phase also requires significant bandwidth to complete data repair, greatly impacting the performance of the distributed storage system. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0005] Verification nodes with the same hierarchical structure are created based on the attribute information of system nodes, and verification data is stored in the verification nodes, wherein the verification data is generated based on the first sub-data and the second sub-data stored in the system nodes;

[0006] If a failed node is detected in a distributed system, a target system node with a repair relationship to the failed node is identified in the distributed system.

[0007] Extract the target raw data stored in the target system node, and the verification data stored in the verification node;

[0008] Failure original data is generated based on the target original data and the verification data, and stored in the failure node.

[0009] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0010] The creation module is configured to create verification nodes with the same hierarchical structure based on the attribute information of system nodes, and store verification data in the verification nodes, wherein the verification data is generated based on the first sub-data and the second sub-data stored in the system nodes;

[0011] The determination module is configured to, upon detecting a failed node in a distributed system, determine a target system node in the distributed system that has a repair relationship with the failed node;

[0012] The extraction module is configured to extract the target raw data stored in the target system node and the verification data stored in the verification node;

[0013] The repair module is configured to generate failure original data based on the target original data and the verification data, and store it in the failure node.

[0014] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0015] Memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor is used to implement the steps of any of the above-described data processing methods when executing the computer-executable instructions.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0019] The data processing method provided in this specification aims to improve data storage efficiency and reduce storage costs in fault-tolerant scenarios. It creates verification nodes based on the attribute information of system nodes in a distributed system, and stores verification data generated from the first and second sub-data stored in the system nodes on these verification nodes. This allows the creation of verification nodes to be controlled by system node attributes, thereby reducing storage costs. Subsequently, if a failed node is detected in the distributed system, a target system node with a repair relationship to the failed node can be identified. By extracting the target original data stored in the target system node and the verification data stored in the verification node, the corresponding failed original data for the failed node is generated and stored on the failed node, thus repairing the data in the failed node. This process, by identifying a target system node with a repair relationship and combining it with the verification node to complete data repair, allows for repair with fewer system nodes involved, effectively reducing the repair bandwidth of system nodes and thus significantly improving the storage efficiency and reliability of the distributed system. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0021] Figure 2 This is a schematic diagram of the encoding in a data processing method provided in one embodiment of this specification;

[0022] Figure 3 This is a schematic diagram of the encoding in another data processing method provided in one embodiment of this specification;

[0023] Figure 4 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0024] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0025] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0026] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0027] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0029] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0030] Distributed Storage System: A distributed storage system provides an interface to access this distributed storage system and a local data buffer to reduce network pressure. It is a system that distributes a large amount of data across multiple storage nodes.

[0031] Erasure coding (EC) is a data protection method that segments data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. Its basic principle is to divide the stored data into fragments, generate k+m fragments from k original data fragments through a certain verification calculation method, and then restore the original data using any k fragments from the k+m fragments. This way, even if some data is lost, the system can still recover the original data.

[0032] MDS code (maximum distance separable code) refers to a code that reaches the Singleton limit. If M = q n+1-d Then the q-ary (n, M, d) code is called a maximum distance separable code, where M is the total number of distinct codewords in the (n, M, d) code; another equivalent description is that when C is a [n, k, d] linear code, the necessary and sufficient condition for C to be an MDS code is d = n - k + 1.

[0033] The Singleton bound is a measure of the number of codewords. It is an upper bound on the number of codewords given the codeword length and minimum distance. A q-ary (n, M) code with a minimum distance of d is usually called a q-ary (n, M, d) code, and the total number of distinct codewords it contains is denoted as A(n, d). The Singleton bound states that A(n, d) ≤ q n+1-d .

[0034] This specification provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0035] In practical applications, distributed storage systems employ erasure coding to support data fault tolerance and reduce storage costs. Reed-Solomon (RS) codes, a type of maximum distance separable code, are widely used in data storage due to their maturity. Node failure is a common phenomenon in real-world distributed storage systems. To maintain system reliability, failed nodes need to be repaired promptly, but the repair bandwidth of (n,k)RS codes is k times the amount of data stored on a single node. Locally Repairable Code (LRC) and Minimum Storage Regenerating Code (MSR) are effective means to reduce repair bandwidth; the latter, due to its MDS property, has the highest storage efficiency (lowest storage cost). However, the construction of MSR codes requires an exponential number of sub-packetization levels, and discontinuous data needs to be read during node repair and degradation reads, resulting in high complexity and making it difficult to widely adopt MSR codes in practical storage applications.

[0036] The data processing method provided in this specification aims to improve data storage efficiency and reduce storage costs in fault-tolerant scenarios. It creates verification nodes based on the attribute information of system nodes in a distributed system, and stores verification data generated from the first and second sub-data stored in the system nodes on these verification nodes. This allows the creation of verification nodes to be controlled by system node attributes, thereby reducing storage costs. Subsequently, if a failed node is detected in the distributed system, a target system node with a repair relationship to the failed node can be identified. By extracting the target original data stored in the target system node and the verification data stored in the verification node, the corresponding failed original data for the failed node is generated and stored on the failed node, thus repairing the data in the failed node. This process, by identifying a target system node with a repair relationship and combining it with the verification node to complete data repair, allows for repair with fewer system nodes involved, effectively reducing the repair bandwidth of system nodes and thus significantly improving the storage efficiency and reliability of the distributed system.

[0037] Figure 1 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0038] Step S102: Create a verification node with the same hierarchical structure based on the attribute information of the system node, and store the verification data in the verification node, wherein the verification data is generated based on the first sub-data and the second sub-data stored in the system node.

[0039] Specifically, the distributed system refers to a distributed storage system; correspondingly, the system node refers to the node contained in the distributed storage system, used for storing data. The data types that system nodes can store include, but are not limited to, transaction data, user information data, video data, image data, etc. The data stored in different scenarios can be selected according to the actual application scenario, and this embodiment does not impose any limitations. Correspondingly, the verification node refers to the node in the distributed system used to store verification data. Unlike system nodes, which directly store data sent from upstream, verification nodes generate verification data based on the data stored in system nodes. This verification node is dedicated to storing verification data, enabling decoding and recovery in the event of system node failure. It should be noted that the number of verification nodes in the distributed system is less than the number of system nodes, and the hierarchical structure of verification nodes and system nodes is the same to support the data reading distribution requirements during repair processing. Correspondingly, the original data refers to the data sent from upstream and stored in system nodes. Correspondingly, the verification data refers to the data generated based on the original data, and the verification data can be combined with the original data stored in other system nodes to complete the data recovery processing in the failed node.

[0040] Therefore, to reduce the storage cost of distributed systems while supporting data fault tolerance, and to enable data recovery with minimal bandwidth after data loss, verification nodes adapted to the current distributed system can be created based on the attribute information of the system nodes. This adaptation should be such that it occupies minimal storage space and can recover data stored on any system node, forming a verification node with the same hierarchical structure as the system nodes. After the verification node is created, verification data can be generated based on the first and second sub-data stored in the system nodes, where the first and second sub-data are the original data stored in the system nodes. The verification data is then stored in the verification node for recovery operations in the event of system node failure.

[0041] It should be noted that, since the system nodes in a distributed system grow data in real time, the verification data needs to be updated simultaneously after each update of the original data to ensure that the verification data stored in the verification nodes is generated based on the latest original data. Based on this, the verification data stored in the verification nodes can be updated after each update of the original data, or it can be updated according to preset time nodes; this embodiment does not impose any limitations on this.

[0042] Furthermore, in the process of creating verification nodes, in order to ensure that the number of verification nodes matches the number of system nodes in the distributed system, and to enable data repair for any failed system node under any condition, it is necessary to create a set number of verification nodes based on the number of system nodes in the distributed system. In this embodiment, the specific implementation method is as follows:

[0043] Obtain the attribute information of system nodes in the distributed system; determine the number of system nodes in the distributed system based on the attribute information; create a set number of verification nodes according to preset verification node creation conditions and the number of system nodes; wherein the hierarchical structure of the verification nodes and the system nodes is the same.

[0044] Specifically, the number of system nodes refers to the total number of all available system nodes in the distributed system. Correspondingly, the preset verification node creation conditions refer to the conditions that determine the appropriate number of verification nodes for distributed systems of different scales, in order to support the repairability of distributed system data with minimal storage resources. The preset verification node creation conditions can be k≥2(r-1), i.e., r≤k / 2+1; where k is the number of system nodes and r is the number of verification nodes.

[0045] Based on this, the first step is to obtain the attribute information corresponding to the system nodes in the distributed system. This attribute information determines the number of system nodes available to support the upstream system. Then, according to the preset verification node creation conditions and the number of system nodes, the set number of verification nodes that can be created in the current distributed system is determined. Finally, the set number of verification nodes are created in the distributed system. In practical applications, when creating verification nodes, the disaster recovery system phase can be selected as the verification node in the distributed system, or a new storage space can be created as the verification node.

[0046] For example, in a distributed storage system with 8 system nodes, 3 verification nodes can be created according to the preset verification node creation conditions; or, in a distributed storage system with 6 system nodes, 2 verification nodes can be created according to the preset verification node creation conditions.

[0047] In summary, by combining preset verification node creation conditions and reasonably creating a set number of verification nodes for a distributed system, it is possible to support data repairable processing with less storage resources, thereby reducing storage costs and effectively improving the reliability of the distributed system.

[0048] Furthermore, after creating the verification node for the distributed system, it is also necessary to store verification data in the verification node. The verification data is the basis for repairing the data stored on the failed node after any system node fails. Therefore, when generating the verification data, it is necessary to complete it according to the relationship between the system node and the verification node. In this embodiment, the specific implementation method is as shown in steps S1022 to S1028.

[0049] Step S1022: Determine the first system sub-node and the second system sub-node in the system nodes.

[0050] Specifically, the first system sub-node refers to the system node used to generate the verification data required for the first verification sub-node, and the second system sub-node refers to the system node used to generate the verification data required for the second verification sub-node. The first and second system sub-nodes may overlap. For example, generating the first verification sub-node requires combining the original data stored in system nodes 1, 2, and 3, while generating the second verification sub-node requires combining the original data stored in system nodes 2, 3, and 4. In this case, system nodes 2 and 3 can serve as either the first or second system sub-nodes, depending on the verification data required to generate the different verification sub-nodes.

[0051] Furthermore, in determining the first and second system sub-nodes, considering that different system sub-nodes need to create the required verification data for different verification sub-nodes, and in order to achieve a data repair mechanism with less storage resources, the created verification nodes can be divided into two categories: the first verification sub-node, which has a direct verification relationship with the system node, and the second verification sub-node, which has an indirect verification relationship with the system node. Then, the system node corresponding to each verification sub-node is selected as the first and second system sub-nodes. In this embodiment, the specific implementation method is as follows:

[0052] The first verification sub-node is defined as a verification sub-node that has a direct verification relationship with the system node, and the second verification sub-node is defined as a verification sub-node that has an indirect verification relationship with the system node. The first system sub-node is determined in the system node based on the first verification sub-node, and the second system sub-node is determined in the system node based on the second verification sub-node.

[0053] Specifically, direct verification relationships refer to preset verification node categories, which specify that the raw data required by this type of verification node is the data stored in all system nodes. Correspondingly, indirect verification relationships refer to another preset verification type, which specifies that the raw data required by this type of verification node is the data stored in local system nodes. This distinction facilitates the subsequent creation of verification data for different verification child nodes. Accordingly, the first verification child node is the verification node corresponding to the direct verification relationship, and the second verification child node is the verification node corresponding to the indirect verification relationship.

[0054] Therefore, after creating verification nodes for the distributed system, from the set number of verification nodes created, a first verification child node that needs to combine data stored in all system nodes for verification data creation can be selected, and a second verification child node that needs to combine data stored in local system nodes can be selected. After determining the first and second verification child nodes, a first system child node associated with the first verification child node and a second system child node associated with the second verification child node can be determined within the system nodes. This facilitates the subsequent creation and storage of verification data based on the mapping relationship between the verification child nodes and the system nodes.

[0055] The data processing method is illustrated using a distributed storage system containing 6 system nodes and 2 verification nodes as an example. For descriptions of other scenarios, please refer to the same or corresponding descriptions in this embodiment, which will not be elaborated on here.

[0056] The distributed storage system includes nodes 1 to 8, where nodes 1 to 6 are system nodes, and nodes 7 and 8 are verification nodes. When creating verification data for the verification nodes, it can be determined that node 7 needs to combine all the data stored by nodes 1 to 6 to create the verification data, so node 7 is designated as the first verification child node. Similarly, node 8 needs to combine some data to create the verification data, so node 8 is designated as the second verification child node. Furthermore, since nodes 1 to 6 are all associated with node 7, nodes 1 to 6 will serve as the first system child nodes corresponding to node 7; similarly, nodes 1 to 6 are also all associated with node 8, so nodes 1 to 6 can be designated as the second system child nodes corresponding to node 8.

[0057] In summary, by combining direct and indirect verification relationships to determine verification sub-nodes and system sub-nodes, the mapping relationship between verification sub-nodes and system sub-nodes can be clearly defined during the determination process. This facilitates the subsequent generation of verification data based on this mapping relationship, avoiding data waste caused by loading excessive data.

[0058] Step S1024: Generate first verification sub-data based on the first original sub-data stored in the first system sub-node.

[0059] Specifically, after determining the first system sub-node and the second system sub-node, the next step is to generate first verification sub-data based on the first original sub-data stored in the first system sub-node. This allows the first verification sub-data to be combined with the second verification sub-data to form verification data for storage in the verification node. Here, the first original sub-data specifically refers to the original data stored in the first system sub-node, and correspondingly, the first verification sub-data specifically refers to a portion of the verification data stored in the verification node.

[0060] Furthermore, in the process of generating the first verification sub-data, in order to save repair bandwidth, it is necessary to combine a small amount of original sub-data during the generation stage. Therefore, the data processing method provided in this embodiment generates the first verification sub-data based on the number of packets for system sub-nodes and verification sub-nodes. The specific implementation method is as follows:

[0061] In the first system sub-node, a first system node sub-packet is determined, and based on the first original sub-packet data stored in the first system node sub-packet, first verification sub-packet data corresponding to the first verification node sub-packet in the first verification sub-node is generated; in the first system sub-node, a second system node sub-packet is determined, and based on the second original sub-packet data stored in the second system node sub-packet, second verification sub-packet data corresponding to the second verification node sub-packet in the first verification sub-node is generated; wherein, the first original sub-packet data and the second original sub-packet data constitute the first original sub-data, and the first verification sub-packet data and the second verification sub-packet data constitute the first verification sub-data.

[0062] Specifically, since both system sub-nodes and verification sub-nodes are divided into two packets, each system sub-node and each verification sub-node will consist of two packets: the first system node packet is one of the two packets of the first system sub-node, and the second system node packet is the other packet. The first system node packet and the second system node packet together constitute the first system sub-node. Correspondingly, the first original packet data is the original data stored in the first system node packet, and the second original packet data is the original data stored in the second system node packet. The first original packet data and the second original packet data together constitute the original data stored in the first system sub-node. Similarly, the first verification node packet is one of the two packets of the first verification sub-node, and the second verification node packet is the other packet. The first verification node packet and the second verification node packet together constitute the first verification sub-node. Correspondingly, the first verification packet data is the verification data to be written in the first verification node packet, and the second verification packet data is the verification data to be written in the second verification node packet. The first verification packet data and the second verification packet data together constitute the first verification sub-data. This refers to the verification data stored in the first verification sub-node.

[0063] Based on this, after determining the first system sub-node associated with the first verification sub-node, in order to save storage resources, each system sub-node and each verification sub-node are divided into packets with a packet count of 2. Based on this, when generating the required first verification sub-data for the first verification sub-node, the first system node packet can be determined within the first system sub-node, and the first verification sub-packet data corresponding to the first verification node packet in the first verification sub-node can be generated based on the first original sub-packet data stored in the first system node packet. Simultaneously, the second system node packet can be determined within the first system sub-node, and the second verification sub-packet data corresponding to the second verification node packet in the first verification sub-node can be generated based on the second original sub-packet data stored in the second system node packet. The first original sub-packet data and the second original sub-packet data together form the first original sub-data, and the first verification sub-packet data and the second verification sub-packet data together form the first verification sub-data. This allows for the generation of data for the first verification sub-node by combining data stored in packets with different layer structures.

[0064] In practice, the first verification sub-package data can be determined using formula (1):

[0065]

[0066] Where p1,1 represents the first verification packet data; a1,j represents the first original packet data; k represents the number of system child nodes; j∈{1,2…k} represents the system node packet;

[0067] Accordingly, the second verification packet data can be determined using formula (2):

[0068]

[0069] Where p2,1 represents the second verification packet data; a2,j represents the second original packet data.

[0070] In practical applications, see Figure 2 The diagram shows that in a scenario with k system nodes and r verification nodes, the k system nodes can be divided into t = 2(r-1) groups, each containing m = k / 2(r-1) system nodes. The t verification data p1,2,p2,2,...p1,r,p2,r correspond to the system nodes in the t groups respectively. That is, the verification data is obtained by linearly combining the k original data in the row where the verification data is located and the m data in different rows in the corresponding group.

[0071] Continuing with the previous example, with 6 system nodes k and 2 verification nodes r, and each node having 2 packet segments, the generation of verification data for node 7 includes: (See...) Figure 3 As shown, p1,1 in node 7 is first used as the first verification packet data to be generated. According to the above formula (1), it needs to be generated by combining the first system node packets corresponding to nodes 1 to 6. That is, it needs to combine the data a1,1 stored in the first system node packet in node 1, the data a1,2 stored in the first system node packet in node 2, ... the data a1,6 stored in the first system node packet in node 6. After loading the data {a1,1; a1,2; ... a1,6}, p1,1 is determined according to formula (1) as a1,1 + a1,2 + a1,3 + a1,4 + a1,5 + a1,6. Then, the data can be encoded by combining this expression to obtain the first verification packet data p1,1 in node 7. Similarly, p2,1 in node 7 is taken as the second verification packet data to be generated. According to the above formula (2), it needs to be generated by combining the second system node packets corresponding to nodes 1 to 6. That is, it needs to combine the data a2,1 stored in the second system node packet of node 1, the data a2,2 stored in the second system node packet of node 2, ... the data a2,6 stored in the second system node packet of node 6. After loading the data {a2,1; a2,2; ... a2,6}, p2,1 is determined according to formula (2) as a2,1 + a2,2 + a2,3 + a2,4 + a2,5 + a2,6. Then, the data can be encoded by combining this expression to obtain the second verification packet data p2,1 in node 7. Among them, "+" and "-" are addition and subtraction operations in the finite field Fq.

[0072] In summary, by dividing the system child nodes and verification child nodes into packets with a packet size of 2, and generating verification data for the verification child nodes accordingly, it is possible to generate verification data with less data and less storage resources. This improves the bandwidth usage during the data repair phase, increases repair efficiency, and ensures the reliability of the distributed system.

[0073] Step S1026: Generate second verification sub-data based on preset verification parameters and the second original sub-data stored in the second system sub-node.

[0074] Specifically, after generating the first verification sub-node data as described above, it is further necessary to generate second verification sub-data for the second verification sub-node. Since the second verification sub-node and the second system sub-node have an indirect repair relationship, during the encoding stage, the second verification sub-data needs to be generated by combining preset verification parameters and the second original sub-data stored in the second system sub-node. Here, the verification parameters specifically refer to the non-zero elements combined during the encoding stage. Correspondingly, the second original sub-data specifically refers to the original data stored in the second system sub-node, and the second verification sub-data specifically refers to a portion of the verification data stored in the verification node, which, together with the first verification sub-data, constitutes the verification data.

[0075] Furthermore, in the process of generating the second verification sub-data, in order to save repair bandwidth and ensure the availability of verification data during the repair phase, it is necessary to combine a small amount of original sub-data during the generation phase. Therefore, the data processing method provided in this embodiment generates the second verification sub-data based on the number of packets for system sub-nodes and verification sub-nodes. The specific implementation method is as follows:

[0076] In the second system sub-node, a first global system node sub-packet and a first local system node sub-packet, as well as a second global system node sub-packet and a second local system node sub-packet, are determined. Based on preset verification parameters, the first global sub-packet data stored in the first global system node sub-packet, and the first local sub-packet data stored in the first local system node sub-packet, third verification sub-packet data corresponding to the third verification node sub-packet in the second verification sub-node is generated. Based on preset verification parameters, the second global sub-packet data stored in the second global system node sub-packet, and the second local sub-packet data stored in the second local system node sub-packet, fourth verification sub-packet data corresponding to the fourth verification node sub-packet in the second verification sub-node is generated. The third verification sub-packet data and the fourth verification sub-packet data constitute the second verification sub-data.

[0077] Specifically, since both system sub-nodes and verification sub-nodes are divided into two parts by a packet count of 2, each system sub-node and each verification sub-node will consist of two packets. The first global system node packet is one of the two packets of all second system sub-nodes, and the second all system node packet is the other packet of all second system sub-nodes. The first global system node packet and the second all system node packet together constitute all second system sub-nodes. Correspondingly, the first local system node packet is one of the two packets of some second system sub-nodes, and the second local system node packet is the other packet of some second system sub-nodes. The first local system node packet and the second local system node packet together constitute the second system sub-nodes.

[0078] Accordingly, the first global sub-packet data is the original data stored in the first global system node sub-packet; the second global sub-packet data is the original data stored in the second global system node sub-packet; the first global sub-packet data and the second global sub-packet data constitute the original data stored in all the second system sub-stages. Similarly, the first local sub-packet data is the original data stored in the first local system node sub-packet; the second local sub-packet data is the original data stored in the second local system node sub-packet; the first local sub-packet data and the second local sub-packet data constitute the original data stored in the second system sub-nodes.

[0079] Accordingly, the third verification node packet is one of the two packets of the second verification sub-node, and the fourth verification node packet is the other packet. The third and fourth verification node packets together constitute the second verification sub-node; correspondingly, the data in the third verification packet is the verification data to be written in the third verification node packet, and the data in the fourth verification packet is the verification data to be written in the fourth verification node packet. The data in the third and fourth verification packets together constitute the second verification data, which is the verification data stored in the second verification sub-node.

[0080] Based on this, after determining the second system sub-node associated with the second verification sub-node, in order to save storage resources, each system sub-node and each verification sub-node are divided into packets with a packet count of 2. Based on this, when generating the required second verification sub-data for the second verification sub-node, the first global system node packet and the first local system node packet, as well as the second global system node packet and the second local system node packet, can be determined in the second system sub-node.

[0081] Furthermore, based on the preset verification parameters, the first global packet data stored in the first global system node packet, and the first local packet data stored in the local system node packet, the third verification packet data corresponding to the third verification node packet in the second verification sub-node is generated.

[0082] Furthermore, based on the preset verification parameters, the second global sub-packet data stored in the second global system node sub-packet, and the second local sub-packet data stored in the second local system node sub-packet, the fourth verification sub-packet data corresponding to the fourth verification node sub-packet in the second verification sub-node is generated; wherein, the third verification sub-packet data and the fourth verification sub-packet data constitute the second verification sub-data.

[0083] In practice, the third verification packet data can be determined using formula (3):

[0084]

[0085] Where p1,r represents the third verification packet data; The following parameters represent the verification parameters; ax,j represent the first global packet data; m represents the number of packets in the local system node; r∈{2,3…r} represents the verification node packets; ι represents the local system node packets; a2,ι represents the first local packet data.

[0086] Accordingly, the fourth verification sub-package data can be determined using formula (4):

[0087]

[0088] Where p2,r represents the fourth verification packet data; a1,ι represents the second local packet data; and ay,j represents the second global packet data.

[0089] In practical applications, These are k(r-1) non-zero elements in the finite field Fq, obtained by choosing appropriate... This allows the encoding process for generating checksum data to have MDS properties.

[0090] Following the previous example, with 6 system nodes k, 2 verification nodes r, and each node having 2 packet segments, the generation of verification data for node 8 includes: (See...) Figure 3As shown, firstly, p1,2 in node 8 is taken as the third verification packet data to be generated. According to the above formula (3), it needs to be generated by combining the first system node packets corresponding to nodes 1 to 6, and the second system node packets corresponding to nodes 1, 2 and 3. That is, it needs to combine the data a1,1 stored in the first system node packet in node 1, the data a1,2 stored in the first system node packet in node 2, ... the data a1,6 stored in the first system node packet in node 6, and the data a2,1 stored in the second system node packet in node 1, the data a2,2 stored in the second system node packet in node 2 and the data a2,3 stored in the second system node packet in node 3. After loading the data {a1,1; a1,2; a1,3; a1,4; a1,5; a1,6; a2,1; a2,2; a2,3}, it is determined according to formula (3). At this point, the data can be encoded using this expression to obtain the third verification packet data p1,2 in node 8.

[0091] Similarly, p2,2 in node 8 is taken as the fourth verification packet data to be generated. According to the above formula (4), it needs to be generated by combining the second system node packets corresponding to nodes 1 to 6, and the first system node packets corresponding to nodes 4, 5 and 6. That is, it needs to combine the data a2,1 stored in the second system node packet in node 1, the data a2,2 stored in the second system node packet in node 2, ... the data a2,6 stored in the second system node packet in node 6, and the data a1,4 stored in the first system node packet in node 4, the data a1,5 stored in the first system node packet in node 5 and the data a1,6 stored in the first system node packet in node 6. After loading the data {a2,1; a2,2; a2,3; a2,4; a2,5; a2,6; a1,4; a1,5; a1,6}, it is determined according to formula (4). At this point, the data can be encoded using this expression to obtain the fourth verification packet data p2,2 in node 8. Wherein, For a non-zero element in the finite field Fq, by choosing an appropriate value, the above encoding process can have the MDS property.

[0092] In summary, by combining preset verification parameters and the original data corresponding to different sub-packets, the verification data corresponding to the second verification sub-node is generated. This enables the generation of verification data to have MDS properties during the encoding stage, thereby ensuring that node repair nodes can complete data recovery with less data, effectively reducing repair bandwidth.

[0093] Step S1028: Generate the verification data based on the first verification sub-data and the second verification sub-data, wherein the first original sub-data and the second original sub-data constitute the original data stored in the system node.

[0094] Specifically, after obtaining the first and second verification sub-data, verification data for storage on the verification node can be generated based on the first and second verification sub-data, so that in the event of system node failure, the system node repair operation can be completed by combining the verification data.

[0095] In summary, by generating verification data with a packet size of 2, the storage efficiency of the distributed system is improved while reducing the system node repair bandwidth, and storage resources are saved.

[0096] Step S104: If a failed node is detected in the distributed system, a target system node with a repair relationship with the failed node is determined in the distributed system.

[0097] Specifically, after generating the verification nodes and verification data, the distributed system can invoke system nodes to perform data storage processing according to the data storage requests from upstream. Furthermore, if a failed node is detected in the distributed system, it indicates that the data stored on that node will be lost. To avoid impacting upstream and downstream systems, erasure coding can be used to recover the data. During recovery, target system nodes with which the failed node is associated can first be identified within the distributed system. Then, data repair processing is performed using the target system nodes and the verification nodes to recover the data stored in the failed node.

[0098] In this context, a failed node refers to a node in a distributed system that is unable to continue storing data. Causes of failure include, but are not limited to, power outages, disk corruption, and reboots. Correspondingly, a target system node refers to the system node required to repair the data stored on the failed node; the relationship between the two is the repair relationship.

[0099] In practical applications, considering that the number of failed nodes at the same time varies in different scenarios, in order to complete node recovery with the minimum amount of data downloaded, different processing levels can be defined based on the number of failed nodes. That is, different recovery strategies can be determined based on the number of failed nodes to recover the data corresponding to the failed nodes. In specific implementation, when dividing the levels, it can be divided into whether the number of failed nodes is greater than the threshold 1. If it is greater than 1, the first repair processing strategy is selected. If it is less than or equal to 1, the second repair processing strategy is selected. The data required for the first repair processing strategy and the second repair processing strategy can be determined by combining the above formulas (1) to (4).

[0100] Furthermore, the data required for the repair process all come from system nodes and verification nodes. Therefore, by determining the source of the required data, the target system node corresponding to the failed node can be identified.

[0101] Step S106: Extract the target original data stored in the target system node and the verification data stored in the verification node.

[0102] Specifically, after identifying the target system nodes that are related to the failed node, considering that the role of the verification node is also to repair the failed node, the target original data stored in the target system node and the verification data stored in the verification node can be extracted to facilitate the subsequent data repair processing of the failed node by combining the verification data and the target original data.

[0103] Step S108: Generate failure original data based on the target original data and the verification data, and store it in the failure node.

[0104] Specifically, based on the target original data stored in the target system node and the verification data stored in the verification node, the failed original data can be generated according to the verification data and the target original data, and stored in the failed node after restarting, thus completing the recovery processing operation of the original data stored in the failed node.

[0105] Furthermore, during the repair process, if there is only one failed node, in order to save repair bandwidth, it is necessary to first accurately filter out its associated target system nodes. In this embodiment, the specific implementation method is as follows:

[0106] During the creation of the verification node, at least two candidate system nodes are associated with the verification node; the candidate system nodes other than the failed node among the at least two candidate system nodes are taken as target system nodes that have a repair relationship with the failed node.

[0107] Specifically, candidate system nodes refer to the verification nodes to which the original data belongs during the verification node creation phase. Based on this, in order to recover failed nodes, at least two candidate system nodes associated with the verification nodes during the verification node creation phase can be identified first. Then, all nodes other than the failed node can be used as target system nodes for subsequent data repair processing.

[0108] In summary, by selecting candidate system nodes from the associated verification nodes to determine the target system node, the correlation between the target system node and the failed node can be guaranteed, thereby improving the accuracy of data repair.

[0109] Furthermore, when generating the original failed data, considering that the number of packets in the failed node is 2, a repair process needs to be performed on each packet separately during the recovery phase. In this embodiment, the specific implementation method is as follows:

[0110] A first target original sub-data and a second target original sub-data are determined from the target original data, and a third verification sub-data and a fourth verification sub-data are determined from the verification data; wherein the data volume of the second target original sub-data is less than the data volume of the first target original sub-data; based on the first target original sub-data and the third verification sub-data, a first failure original sub-data corresponding to the first failure node sub-packet in the failure node is generated; based on the second target original sub-data and the fourth verification sub-data, a second failure original sub-data corresponding to the second failure node sub-packet in the failure node is generated; and the failure original data is generated based on the first failure original sub-data and the second failure original sub-data.

[0111] Specifically, the first target raw sub-data refers to the raw data required to decode the first failed node packet in the failed node, and the first target raw sub-data originates from the first target system node packet of the target system node. Correspondingly, the second target raw sub-data refers to the raw data required to decode the second failed node packet in the failed node, and the second target raw sub-data originates from the second target system node packet of the target system node. Correspondingly, the third verification sub-data refers to the verification data stored in the verification sub-node packet of the verification node; the fourth verification sub-data refers to the verification data stored in the other verification sub-node packet of the verification node.

[0112] Based on this, firstly, the first target original sub-data and the second target original sub-data are determined in the target original data, and simultaneously, the third and fourth verification sub-data are determined in the verification data; and the data volume of the second target original sub-data is less than that of the first target original sub-data; secondly, based on the first target original sub-data and the third verification sub-data, the first failure original sub-data corresponding to the first failure node sub-packet in the failure node is generated; based on the second target original sub-data and the fourth verification sub-data, the second failure original sub-data corresponding to the second failure node sub-packet in the failure node is generated; and finally, failure original data is generated based on the first and second failure original sub-data.

[0113] In practice, for the failure of a single system node, there is always a method to reduce bandwidth for repair. Taking the failure of system node 1 as an example, by downloading data {a1,2; a1,3; ...a1,k} and p1,1, according to formula (1), the data a1,1 in node 1 can be calculated as follows: Node 1 still contains invalid data a2,1. Therefore, according to formulas (3) and (4), it is determined that data {a2,2; a2,3; ... a2,m} and p1,2 still need to be downloaded. According to formula (3), the data a2,1 in node 1 can be calculated as follows: Finally, the decoded a1,1 is written into the first packet of node 1, and a2,1 is written into the second packet of node 1, thus completing the data recovery of node 1. In this process, the amount of data to be downloaded to repair node 1 is determined to be k+k / 2(r-1). Similarly, the repair of any system node i can be calculated by reading the corresponding data a1,i and a2,i stored in node i according to formulas (1)(2) or (3)(4).

[0114] Furthermore, under the condition that any system node fails, i.e., the number of failed nodes is greater than 1, the transformed equations can be obtained by transforming formulas (1) to (4):

[0115]

[0116]

[0117]

[0118]

[0119] The variables in the system of equations are {a1,1; a2,1; a1,2; a2,2; ... a1,r; a2,r}. According to the MDS property, the coefficient matrix corresponding to the unknowns is of full rank. Therefore, downloading all the data from nodes r+1 to r+k will yield the unique solution to the system of equations, thus decoding the data and recovering the data from the failed nodes.

[0120] Following the example above, see [link to example]. Figure 3 As shown, when node 1 fails, by downloading data {a1,2; a1,3; a1,4; a1,5; a1,6} and p1,1, a1,1 = p1,1 - a1,2 - a1,3 - a1,4 - a1,5 - a1,6 is calculated according to formula (1). Then, by downloading data {a2,2; a2,3} and p1,2, a1,1 = p1,1 - a1,2 - a1,3 - a1,4 - a1,5 - a1,6 is calculated according to formula (3). The data of node 1 can be recovered based on a1,1 and a2,1.

[0121] When both node 1 and node 2 fail, the system of equations can be obtained by transforming formulas (1) to (4):

[0122] a1,1+a1,2+a1,3+a1,4+a1,5+a1,6-p1,1=0

[0123] a2,1+a2,2+a2,3+a2,4+a2,5+a2,6-p2,1=0

[0124]

[0125]

[0126] The unknowns are determined by the above system of equations: a1,1, a1,2, a2,1 and a2,2. According to the MDS property, the coefficient matrix corresponding to the unknowns is of full rank. Therefore, downloading all the data from nodes 3 to 6 will yield the unique solution to the equations, which can be used to repair nodes 1 and 2.

[0127] The data processing method provided in this specification aims to improve data storage efficiency and reduce storage costs in fault-tolerant scenarios. It creates verification nodes based on the attribute information of system nodes in a distributed system, and stores verification data generated from the first and second sub-data stored in the system nodes on these verification nodes. This allows the creation of verification nodes to be controlled by system node attributes, thereby reducing storage costs. Subsequently, if a failed node is detected in the distributed system, a target system node with a repair relationship to the failed node can be identified. By extracting the target original data stored in the target system node and the verification data stored in the verification node, the corresponding failed original data for the failed node is generated and stored on the failed node, thus repairing the data in the failed node. This process, by identifying a target system node with a repair relationship and combining it with the verification node to complete data repair, allows for repair with fewer system nodes involved, effectively reducing the repair bandwidth of system nodes and thus significantly improving the storage efficiency and reliability of the distributed system.

[0128] The following is in conjunction with the appendix Figure 4 Taking the application of the data processing method provided in this specification in a scenario with 6 system nodes and 2 verification nodes as an example, the data processing method will be further explained. Figure 4 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0129] Step S402: Obtain the attribute information of the system nodes in the distributed system.

[0130] Step S404: Determine the number of system nodes included in the distributed system based on the attribute information.

[0131] Step S406: Create a set number of verification nodes according to the preset verification node creation conditions and the number of system nodes.

[0132] Step S408: Store the verification data in the verification node, wherein the verification data is generated based on the original data stored in the system node.

[0133] Specifically, the generation of verification data includes: determining a first system sub-node and a second system sub-node in the system nodes; generating first verification sub-data based on the first original sub-data stored in the first system sub-node; generating second verification sub-data based on preset verification parameters and the second original sub-data stored in the second system sub-node; and generating the verification data based on the first verification sub-data and the second verification sub-data; wherein the first original sub-data and the second original sub-data constitute the original data.

[0134] Furthermore, the verification sub-nodes in the verification nodes that have a direct verification relationship with the system node are designated as first verification sub-nodes, and the verification sub-nodes in the verification nodes that have an indirect verification relationship with the system node are designated as second verification sub-nodes; the first system sub-node is determined in the system node based on the first verification sub-node, and the second system sub-node is determined in the system node based on the second verification sub-node.

[0135] When both the system sub-nodes in the system node and the verification sub-nodes in the verification node have two packets, generating the first verification sub-data based on the first original sub-data stored in the first system sub-node includes:

[0136] In the first system sub-node, a first system node sub-packet is determined, and based on the first original sub-packet data stored in the first system node sub-packet, first verification sub-packet data corresponding to the first verification node sub-packet in the first verification sub-node is generated; in the first system sub-node, a second system node sub-packet is determined, and based on the second original sub-packet data stored in the second system node sub-packet, second verification sub-packet data corresponding to the second verification node sub-packet in the first verification sub-node is generated; wherein, the first original sub-packet data and the second original sub-packet data constitute the first original sub-data, and the first verification sub-packet data and the second verification sub-packet data constitute the first verification sub-data.

[0137] Furthermore, in the second system sub-node, a first global system node sub-packet and a first local system node sub-packet, as well as a second global system node sub-packet and a second local system node sub-packet, are determined. Based on preset verification parameters, the first global sub-packet data stored in the first global system node sub-packet, and the first local sub-packet data stored in the first local system node sub-packet, third verification sub-packet data corresponding to the third verification node sub-packet in the second verification sub-node is generated. Based on preset verification parameters, the second global sub-packet data stored in the second global system node sub-packet, and the second local sub-packet data stored in the second local system node sub-packet, fourth verification sub-packet data corresponding to the fourth verification node sub-packet in the second verification sub-node is generated. The third verification sub-packet data and the fourth verification sub-packet data constitute the second verification sub-data.

[0138] Step S410: If a failed node is detected in the distributed system, a target system node with a repair relationship with the failed node is determined in the distributed system.

[0139] Step S412: Extract the target original data stored in the target system node and the verification data stored in the verification node.

[0140] Step S414: Generate failure original data based on the target original data and the verification data, and store it in the failure node.

[0141] When the number of failed nodes is one, determining the target system node with a repair relationship with the failed node in the distributed system includes: determining at least two candidate system nodes associated with the verification node during the creation of the verification node; and selecting the candidate system nodes other than the failed node from the at least two candidate system nodes as the target system node with a repair relationship with the failed node.

[0142] Further, a first target original sub-data and a second target original sub-data are determined from the target original data, and a third verification sub-data and a fourth verification sub-data are determined from the verification data; wherein, the data volume of the second target original sub-data is less than the data volume of the first target original sub-data; based on the first target original sub-data and the third verification sub-data, a first failure original sub-data corresponding to the first failure node sub-packet in the failure node is generated; based on the second target original sub-data and the fourth verification sub-data, a second failure original sub-data corresponding to the second failure node sub-packet in the failure node is generated; and the failure original data is generated based on the first failure original sub-data and the second failure original sub-data.

[0143] In summary, to improve data storage efficiency and reduce storage costs in fault-tolerant scenarios, verification nodes can be created based on the attribute information of system nodes in the distributed system. Verification data generated from the first and second sub-data stored in the system nodes is then stored on these verification nodes. This allows the creation of verification nodes to be controlled by system node attributes, thereby reducing storage costs. Subsequently, if a failed node is detected in the distributed system, a target system node with a repair relationship to the failed node can be identified. By extracting the target original data stored in the target system node and the verification data stored in the verification node, the corresponding failed original data for the failed node can be generated and stored on the failed node, thus repairing the data in the failed node. This process, by identifying the target system node with a repair relationship and combining it with the verification node to complete data repair, allows for repair with fewer system nodes involved, effectively reducing the repair bandwidth of system nodes and thus significantly improving the storage efficiency and reliability of the distributed system.

[0144] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0145] The creation module 502 is configured to create a verification node with the same hierarchical structure based on the attribute information of the system node, and store the verification data in the verification node, wherein the verification data is generated based on the first sub-data and the second sub-data stored in the system node;

[0146] The determination module 504 is configured to determine, in the distributed system, a target system node that has a repair relationship with the failed node when a failed node is detected in the distributed system.

[0147] Extraction module 506 is configured to extract the target raw data stored in the target system node and the verification data stored in the verification node;

[0148] Repair module 508 is configured to generate failure original data based on the target original data and the verification data, and store it in the failure node.

[0149] In an optional embodiment, the creation module 502 is further configured to:

[0150] Obtain the attribute information of system nodes in the distributed system; determine the number of system nodes in the distributed system based on the attribute information; create a set number of verification nodes according to preset verification node creation conditions and the number of system nodes; wherein the hierarchical structure of the verification nodes and the system nodes is the same.

[0151] In an optional embodiment, the creation module 502 is further configured to:

[0152] A first system sub-node and a second system sub-node are determined in the system node; a first verification sub-data is generated based on the first original sub-data stored in the first system sub-node; a second verification sub-data is generated based on preset verification parameters and the second original sub-data stored in the second system sub-node; the verification data is generated based on the first verification sub-data and the second verification sub-data; wherein the first original sub-data and the second original sub-data constitute the original data stored in the system node.

[0153] In an optional embodiment, the creation module 502 is further configured to:

[0154] The first verification sub-node is defined as a verification sub-node that has a direct verification relationship with the system node, and the second verification sub-node is defined as a verification sub-node that has an indirect verification relationship with the system node. The first system sub-node is determined in the system node based on the first verification sub-node, and the second system sub-node is determined in the system node based on the second verification sub-node.

[0155] In an optional embodiment, when both the system sub-nodes in the system node and the verification sub-nodes in the verification node have two packet counts, the creation module 502 is further configured to:

[0156] In the first system sub-node, a first system node sub-packet is determined, and based on the first original sub-packet data stored in the first system node sub-packet, first verification sub-packet data corresponding to the first verification node sub-packet in the first verification sub-node is generated; in the first system sub-node, a second system node sub-packet is determined, and based on the second original sub-packet data stored in the second system node sub-packet, second verification sub-packet data corresponding to the second verification node sub-packet in the first verification sub-node is generated; wherein, the first original sub-packet data and the second original sub-packet data constitute the first original sub-data, and the first verification sub-packet data and the second verification sub-packet data constitute the first verification sub-data.

[0157] In an optional embodiment, the creation module 502 is further configured to:

[0158] In the second system sub-node, a first global system node sub-packet and a first local system node sub-packet, as well as a second global system node sub-packet and a second local system node sub-packet, are determined. Based on preset verification parameters, the first global sub-packet data stored in the first global system node sub-packet, and the first local sub-packet data stored in the first local system node sub-packet, third verification sub-packet data corresponding to the third verification node sub-packet in the second verification sub-node is generated. Based on preset verification parameters, the second global sub-packet data stored in the second global system node sub-packet, and the second local sub-packet data stored in the second local system node sub-packet, fourth verification sub-packet data corresponding to the fourth verification node sub-packet in the second verification sub-node is generated. The third verification sub-packet data and the fourth verification sub-packet data constitute the second verification sub-data.

[0159] In an optional embodiment, when the number of failed nodes is one, the determining module 504 is further configured to:

[0160] During the creation of the verification node, at least two candidate system nodes are associated with the verification node; the candidate system nodes other than the failed node among the at least two candidate system nodes are taken as target system nodes that have a repair relationship with the failed node.

[0161] In an optional embodiment, the repair module 508 is further configured to:

[0162] A first target original sub-data and a second target original sub-data are determined from the target original data, and a third verification sub-data and a fourth verification sub-data are determined from the verification data; wherein the data volume of the second target original sub-data is less than the data volume of the first target original sub-data; based on the first target original sub-data and the third verification sub-data, a first failure original sub-data corresponding to the first failure node sub-packet in the failure node is generated; based on the second target original sub-data and the fourth verification sub-data, a second failure original sub-data corresponding to the second failure node sub-packet in the failure node is generated; and the failure original data is generated based on the first failure original sub-data and the second failure original sub-data.

[0163] In an optional embodiment, the first verification packet data is determined by formula (1):

[0164]

[0165] Where p1,1 represents the first verification packet data; a1,j represents the first original packet data; k represents the number of system child nodes; j∈{1,2…k} represents the system node packet;

[0166] Accordingly, the second verification packet data is determined by formula (2):

[0167]

[0168] Where p2,1 represents the second verification packet data; a2,j represents the second original packet data.

[0169] In an optional embodiment, the third verification packet data is determined by formula (3):

[0170]

[0171] Where p1,r represents the third verification packet data; The following parameters represent the verification parameters; ax,j represent the first global packet data; m represents the number of packets in the local system node; r∈{2,3…r} represents the verification node packets; ι represents the local system node packets; a2,ι represents the first local packet data.

[0172] Accordingly, the fourth verification packet data is determined by formula (4):

[0173]

[0174] Where p2,r represents the fourth verification packet data; a1,ι represents the second local packet data; and ay,j represents the second global packet data.

[0175] The data processing apparatus provided in this specification, in order to improve data storage efficiency and reduce storage costs in scenarios supporting data fault tolerance, can create verification nodes based on the attribute information of system nodes in a distributed system. Verification data generated from the first and second sub-data stored in the system nodes is stored in the verification nodes. This allows the creation of verification nodes to be controlled by system node attributes, thereby reducing storage costs. Subsequently, if a failed node is detected in the distributed system, a target system node with a repair relationship to the failed node can be identified. By extracting the target original data stored in the target system node and the verification data stored in the verification node, the corresponding failed original data for the failed node is generated and stored in the failed node, thus repairing the data in the failed node. This process, by identifying the target system node with a repair relationship and combining it with the verification node to complete data repair, allows repair to be completed with fewer system nodes involved, effectively reducing the repair bandwidth of system nodes and thus significantly improving the storage efficiency and reliability of the distributed system.

[0176] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0177] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0178] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0179] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0180] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0181] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0182] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0183] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0184] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0185] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0186] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0187] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0188] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0189] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0190] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0191] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Verification nodes with the same hierarchical structure are created based on the attribute information of system nodes, and verification data is stored in the verification nodes. The verification data is generated based on the first and second sub-data stored in the system nodes. The generation of the verification data includes: designating verification sub-nodes in the verification nodes that have a direct verification relationship with the system nodes as first verification sub-nodes, and designating verification sub-nodes in the verification nodes that have an indirect verification relationship with the system nodes as second verification sub-nodes; determining a first system sub-node in the system nodes based on the first verification sub-nodes; and determining a second system sub-node in the system nodes based on the second verification sub-nodes. Each system sub-node and each verification sub-node are divided into packets with a packet size of 2. First verification sub-data is generated based on the first sub-data stored in the first system sub-node, and second verification sub-data is generated based on the verification packet data. The verification packet data is generated based on preset verification parameters, first global packet data stored in the first global system node packet, and first local packet data stored in the first local system node packet. The first global system node packet is one of the two packets of all the second system sub-nodes, and the first local system node packet is one of the two packets of some of the second system sub-nodes. If a failed node is detected in a distributed system, a target system node with a repair relationship to the failed node is identified in the distributed system. Extract the target raw data stored in the target system node, and the verification data stored in the verification node; Failure original data is generated based on the target original data and the verification data, and stored in the failure node.

2. The method according to claim 1, wherein creating verification nodes with the same hierarchical structure based on the attribute information of system nodes includes: Obtain the attribute information of system nodes in the distributed system; The number of system nodes contained in the distributed system is determined based on the attribute information. Create a set number of verification nodes according to the preset verification node creation conditions and the number of system nodes; wherein the hierarchical structure of the verification nodes and the system nodes is the same.

3. The method according to claim 1, wherein the number of packets in both the system sub-nodes in the system node and the verification sub-nodes in the verification node is two, the step of generating the first verification sub-data based on the first original sub-data stored in the first system sub-node includes: In the first system sub-node, the first system node sub-packet is determined, and based on the first original sub-packet data stored in the first system node sub-packet, the first verification sub-packet data corresponding to the first verification node sub-packet in the first verification sub-node is generated; In the first system sub-node, the second system node sub-packet is determined, and based on the second original sub-packet data stored in the second system node sub-packet, the second verification sub-packet data corresponding to the second verification node sub-packet in the first verification sub-node is generated; The first original sub-packet data and the second original sub-packet data constitute the first original sub-data, and the first verification sub-packet data and the second verification sub-packet data constitute the first verification sub-data.

4. The method according to claim 3, wherein generating the second verification sub-data based on preset verification parameters and the second original sub-data stored in the second system sub-node includes: In the second system sub-node, the first global system node sub-packet and the first local system node sub-packet, as well as the second global system node sub-packet and the second local system node sub-packet, are determined; Based on the preset verification parameters, the first global packet data stored in the first global system node packet, and the first local packet data stored in the first local system node packet, the third verification packet data corresponding to the third verification node packet in the second verification sub-node is generated. Based on the preset verification parameters, the second global packet data stored in the second global system node packet, and the second local packet data stored in the second local system node packet, the fourth verification packet data corresponding to the fourth verification node packet in the second verification sub-node is generated; The third and fourth verification packet data together constitute the second verification sub-data.

5. The method according to any one of claims 1-4, wherein when the number of failed nodes is one, determining the target system node in the distributed system that has a repair relationship with the failed node includes: During the stage of creating the verification node, at least two candidate system nodes are associated with the verification node; The candidate system nodes other than the failed node among the at least two candidate system nodes are taken as target system nodes that have a repair relationship with the failed node.

6. The method according to claim 5, wherein generating failure original data based on the target original data and the verification data comprises: A first target original sub-data and a second target original sub-data are determined from the target original data, and a third verification sub-data and a fourth verification sub-data are determined from the verification data; wherein, the data volume of the second target original sub-data is less than the data volume of the first target original sub-data; Based on the first target original sub-data and the third verification sub-data, generate the first failure original sub-data corresponding to the first failure node sub-packet in the failure node; Based on the second target original sub-data and the fourth verification sub-data, generate the second failure original sub-data corresponding to the second failure node sub-packet in the failure node; The original failure data is generated based on the first original failure sub-data and the second original failure sub-data.

7. The method according to claim 4, wherein the first verification packet data is determined by formula (1): p1,1= （1） in, p1,1 represents the first verification packet data; a1,j represents the first original packet data; k represents the number of system child nodes; j∈{1,2…k} represents the system node packet; Accordingly, the second verification packet data is determined by formula (2): p2,1= （2） Where p2,1 represents the second verification packet data; a2,j represents the second original packet data.

8. The method according to claim 7, wherein the third verification packet data is determined by formula (3): p1,r= ·ax,j+ a2, (3) in, p1,r represents the third verification packet data; The parameters represent the verification parameters; ax,j represent the first global packet data; m represents the number of packets in the local system node; r∈{2,3…r} represents the verification node packets; This indicates local system node packet segmentation; a2, This represents the packet data for the first game. Accordingly, the fourth verification packet data is determined by formula (4): p2,r= ·yes,j+ a1, (4) Where p2,r represents the fourth verification packet data; a1, This represents the second-round packet data; ay,j represents the second-round global packet data.

9. A data processing apparatus, comprising: A creation module is configured to create verification nodes with the same hierarchical structure based on the attribute information of system nodes, and to store verification data in the verification nodes. The verification data is generated based on first and second sub-data stored in the system nodes. The generation of the verification data includes: designating verification sub-nodes in the verification nodes that have a direct verification relationship with the system nodes as first verification sub-nodes, and designating verification sub-nodes in the verification nodes that have an indirect verification relationship with the system nodes as second verification sub-nodes; determining a first system sub-node in the system nodes based on the first verification sub-nodes; and determining the system nodes based on the second verification sub-nodes. The second system sub-node is determined, and each system sub-node and each verification sub-node are divided into packets with a packet size of 2. First verification sub-data is generated based on the first sub-data stored in the first system sub-node, and second verification sub-data is generated based on the verification packet data. The verification packet data is generated based on preset verification parameters, first global packet data stored in the first global system node packet, and first local packet data stored in the first local system node packet. The first global system node packet is one of the two packets of all the second system sub-nodes, and the first local system node packet is one of the two packets of some of the second system sub-nodes. The determination module is configured to, upon detecting a failed node in a distributed system, determine a target system node in the distributed system that has a repair relationship with the failed node; The extraction module is configured to extract the target raw data stored in the target system node and the verification data stored in the verification node; The repair module is configured to generate failure original data based on the target original data and the verification data, and store it in the failure node.

10. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images