Parallel network aggregation repair method, system and device

Abstract

The present application relates to a parallelized network aggregation repair method, system, and device, which includes: a helper splitting a relevant storage block into multiple sub-blocks and assigning them to the corresponding sending queues of multiple tree pipelines; based on the transmission throughput of the tree pipeline, the sub-blocks in the corresponding sending queue are packaged into corresponding ACP messages and sent to the next node; the next node aggregates the ACP messages based on the sub-block processing identifiers carried by the ACP messages, the unique identification numbers of the tree pipelines corresponding to the sub-blocks, and the sequence numbers, and then sends them to the subsequent nodes or forwards the ACP messages to the subsequent nodes; when the helper completes the transmission of the relevant storage blocks, the requester who collects data at the end of each tree pipeline unpacks all received ACP messages into sub-blocks based on the unique identification numbers and generates repair blocks for repairing the failed storage blocks. This greatly improves the performance of repairing failed storage blocks.

Description

Technical Field

[0001] The present invention belongs to the technical field of data block repair in erasure code storage systems, and relates to a parallelized network aggregation repair method, system, and device. Background Art

[0002] Erasure coding (EC) is a low-cost fault-tolerance mechanism widely adopted by many distributed storage systems (DSSs). Compared with replication-based fault-tolerance designs, erasure coding-based fault-tolerance schemes save a significant amount of space overhead for distributed storage systems. When an erasure coding-based DSS encounters a node failure, repairing the failed storage block (called a degraded read) requires retrieving many related storage blocks from other storage nodes, resulting in a huge amount of broadcast traffic. This problem has not been solved, and the main approach in the field is to use partially participating storage nodes to aggregate related storage blocks from other nodes, thereby reducing the total amount of data sent.

[0003] Based on this, traditional repair work can be roughly divided into two categories, namely block-level scheduling schemes and pipeline schemes. The former focuses on scheduling when and where to use incast trees to schedule related storage blocks to avoid potential network bottlenecks; the latter aggregates multiple small block units on some intermediate nodes into one unit, and then sends them to the next hop to avoid the problem of entry link bottlenecks. With the deepening of research, the emergence of programmable switches and in-network computing technologies has also provided new opportunities for improving failure repair strategies in erasure code storage systems. Based on this, there has been a repair technology that builds multiple parallel tree pipelines to improve repair performance, while using intermediate nodes on the routing path (such as switches or servers) to aggregate related data to avoid in-network aggregation such as incast transmission. However, in the process of implementing the present invention, the inventors found that the aforementioned traditional repair technology still has the technical problem of insufficient repair performance. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes a parallelized network aggregation repair method, a parallelized network aggregation repair system, a computer device and a computer-readable storage medium that can significantly improve the repair performance of failed storage blocks.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] In one aspect, a parallelized network aggregation repair method is provided, comprising the steps of:

[0007] The helper splits the relevant storage block into multiple sub-blocks and assigns them to the corresponding sending queues of multiple tree-shaped pipelines; each tree-shaped pipeline is identified by a unique identification number and its own transmission throughput;

[0008] Based on the transmission throughput of the tree pipeline, the sub-blocks in the corresponding sending queue are packaged into corresponding ACP messages and sent to the next node; the ACP message carries the sub-block processing identifier, the unique identifier of the tree pipeline corresponding to the sub-block, and the sequence number assigned to the sub-block. The sub-block processing identifier includes an aggregation identifier or a forwarding identifier.

[0009] The next node aggregates the ACP message based on the sub-block processing identifier, the unique identifier of the tree pipeline corresponding to the sub-block, and the sequence number, and then sends it to the subsequent node or forwards the ACP message to the subsequent node;

[0010] When the helper completes the transmission of the relevant storage block, the requester that collects the data at the end of each tree pipeline unpacks all received ACP messages into sub-blocks according to each unique identification number and generates a repair block;

[0011] The requester uses the repair block to repair the failed storage block.

[0012] On the other hand, a parallelized network aggregation repair system is also provided, which includes a helper, intermediate nodes, and a requester. After receiving a repair instruction from a coordinator, the helper splits the relevant storage block into multiple sub-blocks and assigns them to the corresponding sending queues of multiple tree-shaped pipelines; each tree-shaped pipeline is identified by a unique identification number and its own transmission throughput;

[0013] Based on the transmission throughput of the tree pipeline, the helper packages the sub-blocks in the corresponding sending queue into corresponding ACP messages and sends them to the next node. The ACP messages carry the sub-block processing identifier, the unique identifier of the tree pipeline corresponding to the sub-block, and the sequence number assigned to the sub-block. The sub-block processing identifier includes an aggregation identifier or a forwarding identifier.

[0014] The next node aggregates the ACP message based on the sub-block processing identifier, the unique identifier of the tree pipeline corresponding to the sub-block, and the sequence number, and then sends it to the subsequent node or forwards the ACP message to the subsequent node;

[0015] When the helper completes the transmission of the relevant storage blocks, the requester who collects data at the end of each tree pipeline unpacks all received ACP messages into sub-blocks according to each unique identification number and generates a repair block. The requester uses the repair block to repair the failed storage block.

[0016] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned parallel network aggregation repair method when executing the computer program.

[0017] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned parallel network aggregation repair method are implemented.

[0018] One of the above technical solutions has the following advantages and beneficial effects:

[0019] The above-mentioned parallelized network aggregation repair method, system and device, for each helper that provides relevant storage blocks, splits the relevant storage blocks into multiple sub-blocks through the helper and then uses multiple parallel tree pipelines constructed by Paint for parallel transmission. During the transmission process, each sub-block is transmitted in the form of a data packet of an ACP message. Since each tree pipeline has its own unique identification number, each ACP message also carries the unique identification number of the tree pipeline from which the sub-block comes, and each ACP message carries a sub-block processing identifier to distinguish whether the ACP message is an information packet that needs to be aggregated after sub-block data before transmission, or an ordinary information packet that does not need aggregation and is directly forwarded. At the same time, each ACP message also carries a serial number assigned to the sub-block.

[0020] Therefore, for any node on the tree pipeline, after receiving the ACP message from the previous node, the next node immediately adjacent to the previous node can first distinguish whether the ACP message is an information packet that needs to be aggregated and transmitted, or an information packet that is directly forwarded, based on the sub-block processing identifier carried in the ACP message. If it is an information packet that needs to be aggregated and transmitted, then based on the unique identification number and sequence number carried in the ACP message, the next node will aggregate the ACP messages received from the same tree pipeline with the same sequence number before sending them to the subsequent node. Otherwise, the ACP message will be directly forwarded to the subsequent node. The requester collects data at the end of each tree pipeline. When all helpers have completed the transmission of the relevant storage block, the requester assembles all received sub-blocks according to the unique identification number, thereby regenerating the repair block to complete the repair of the failed storage block.

[0021] Compared with traditional technologies, the above solution uses Paint to improve repair performance by building multiple parallel tree-like pipelines. At the same time, it uses intermediate nodes on the routing path (such as switches or servers) to aggregate related data to avoid incast transmission. On this basis, it proposes an aggregation control protocol (ACP) to manage different repair pipelines, solving problems such as message identification, reliability, and rate control faced by parallel network repair, thereby significantly improving the repair performance of failed storage blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 Schematic diagram of the erasure code encoding process;

[0024] Figure 2 1 is a flow chart of a parallelized network aggregation repair method according to an embodiment;

[0025] Figure 3 A schematic flow chart of a parallelized network aggregation repair method according to another embodiment;

[0026] Figure 4 Schematic overview of a Paint repair network in one embodiment;

[0027] Figure 5 Schematic diagram of the composition of an ACP message in one embodiment;

[0028] Figure 6 This is an example schematic diagram of a failed storage block to be repaired in one embodiment;

[0029] Figure 7 FIG1 is a schematic diagram of an example of repairing a failed storage block in one embodiment;

[0030] Figure 8 FIG1 is a schematic diagram illustrating an example of deploying intra-network aggregation on a programmable switch according to an embodiment;

[0031] Figure 9 Figure 1 is a performance diagram of two parallelized Paint pipelines in one embodiment, where (a) is a curve showing the number of packets sent in each batch versus pipeline throughput, and (b) is a curve showing the number of packets sent in each batch versus repair time.

[0032] Figure 10 Schematic diagram comparing the Paint and SP methods in one embodiment, wherein (a) is a curve showing the number of packets sent in each batch and throughput, and (b) is a curve showing the number of packets sent in each batch and repair time;

[0033] Figure 11 A schematic diagram showing a comparison of repair times under different block size settings in one embodiment;

[0034] Figure 12 Schematic diagram of the module structure of a parallelized network aggregation repair system in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0037] It should be noted that reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of such a phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments.

[0038] Those skilled in the art will appreciate that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the present specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0039] Incast transmission not only imposes a heavy network burden but also results in significant repair delays. When a destination node receives traffic from multiple sources simultaneously, its ingress link can become a network bottleneck. Consequently, the transmission process can introduce significant network delays and hinder repair performance. To address these challenges, existing repair methods in the field primarily include partially parallel repair (PPR), repair pipelines (RP), repair trees (RT), and repair methods using multiple parallel tree-like pipelines (Paint).

[0040] The rise of programmable switches has provided powerful computing and caching capabilities to traditional store-and-forward networks. In-network aggregation is being explored on programmable switches for distributed search, machine learning, virtual network functions, and other functions to achieve effective data reduction. Programmable switches are used in DSS to facilitate failed storage block repair. When a degraded read occurs, repair traffic can be aggregated across the programmable switches along the path, effectively reducing bandwidth consumption. This repair model is therefore called in-network repair. Furthermore, parallelization can provide significant performance improvements for failed storage block repair, as there are typically multiple parallel and equal paths between any pair of servers in a warehouse cluster or data center. Repair operations can be further decomposed into multiple pipelines and executed in parallel across different routing paths. This parallel repair model fully utilizes available bandwidth to reduce the latency of degraded reads, ensure quality of service, and mitigate the impact of high traffic on the cluster. However, research on this type of design is limited to date.

[0041] In erasure-coded storage systems, files are typically stored as fixed-size blocks, the basic unit of read and write. Typically, in a (k,m) erasure-coded storage system, every k original data blocks in a stripe are encoded together to produce m redundant blocks. These k original blocks from the source file are called data blocks, while the m redundant blocks are called parity blocks, primarily for fault tolerance. The k+m blocks in the same stripe are stored in a distributed manner across k+m nodes. Each block can be regenerated by decoding any k blocks in the stripe. Therefore, the system can tolerate up to m node failures without data loss.

[0042] Currently, widely used erasure codes satisfy linear properties, including RS coding, LRC coding, MSR coding, and MBR coding. Specifically, k related blocks are represented as B1, B2, ..., B k Any block B on the same stripe * It can be calculated by the linear combination of k blocks, expressed as where α i It's block B i The corresponding decoding coefficients, additions and multiplications are based on Galois field arithmetic over finite fields. In actual deployment, the encoding and decoding of blocks are performed on small coding units to achieve better performance, called words. The size of a word can be multiple bits, such as 8 or 16 bits. The words in the corresponding positions of each block are encoded together to form a coding group. The encoding and decoding of different coding groups are independent of each other, such as Figure 1 Therefore, when repairing failed storage blocks, the system can perform phrase-level decoding operations after receiving a small portion of each block, rather than waiting for the entire block to be received. This is also the theoretical basis of the pipeline repair solution.

[0043] During the study, the inventors found that the Paint repair method, which uses multiple parallel Minimum-Widest Directed Steiner Trees (MWDSTs), improves the repair throughput when repairing erasure-coded storage systems and utilizes programmable switch nodes on the transmission path to achieve data aggregation within the network, reducing transmission costs. However, in practical applications, there are still some key challenges that need to be addressed: for example, different pipelines may conflict with each other, requiring careful design to build and coordinate them; routing algorithms are very time-consuming, and generating routing paths for parallel network repairs is a type of NP-hard problem, even if only one pipeline is built; unlike servers, programmable switches have their own computing logic and architecture, which require careful design to achieve caching and aggregation of related data, and also need to handle network errors such as packet loss and disorder. To address these challenges, it is necessary to design a new repair control method to control the repair process within the network, so that the performance of failed storage block repair based on the Paint repair method can be further significantly improved.

[0044] In summary, the design concept of this application is based on Paint's parallelized network aggregation repair solution, which is used to repair failed blocks in erasure coded storage systems. This solution improves repair performance by constructing multiple parallel tree-like pipelines. Furthermore, it utilizes intermediate nodes (such as switches or servers) along the routing path to aggregate related data to avoid incast transmission. Furthermore, a new aggregation control method, the Aggregation Control Protocol (ACP), is designed. ACP addresses the issues of message identification, reliability, and rate control faced by parallel network repair, further significantly improving the performance of failed storage block repair.

[0045] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0046] See also Figure 2 In one embodiment, the present application provides a parallelized network aggregation repair method, comprising the following processing steps S12 to S20:

[0047] S12, the helper splits the relevant storage block into multiple sub-blocks and allocates them to the corresponding sending queues of the corresponding multiple tree-shaped pipelines; each tree-shaped pipeline is identified by a unique identification number and its own transmission throughput.

[0048] It can be understood that a helper refers to a network storage node in the network that can provide related storage blocks for repairing failed storage blocks in failed nodes, and there can be multiple helpers in the network. Multiple tree-shaped pipelines can be pre-calculated and determined, and the specific calculation implementation method of their routing paths can be referred to the calculation implementation method of multiple parallel tree-shaped pipelines in the Paint repair method. The difference is that in this application, multiple parallel tree-shaped pipelines are also assigned their own exclusive identification numbers (also called pipeline IDs), and each tree-shaped pipeline can be identified by a pipeline ID and its transmission throughput.

[0049] Specifically, for each helper, the helper encodes its related storage block and splits it into multiple sub-blocks. These sub-blocks (each sub-block contains multiple erasure code phrases) are assigned to the corresponding sending queue of each pipeline. For example, for the i-th helper, it is based on the local block B i The coding coefficient α i Calculate a coding block α i B i Assume that there are a total of λ tree pipelines, and the throughput of the jth tree pipeline is recorded as S j Therefore, each helper stores its block α i B i It is divided into λ sub-blocks, and each sub-block is assigned to a tree pipeline.

[0050] S14, according to the transmission throughput of the tree pipeline, the sub-blocks in the corresponding sending queue are packaged into corresponding ACP messages and sent to the next node; the ACP message carries the sub-block processing identifier, the unique identification number of the tree pipeline corresponding to the sub-block, and the sequence number assigned to the sub-block. The sub-block processing identifier includes an aggregation identifier or a forwarding identifier.

[0051] It can be understood that there is a one-to-one correspondence between the sending queue and the pipeline ID in the helper. Each sending queue sends data at the transmission throughput predicted by the corresponding pipeline. In order to avoid the occurrence of stragglers dragging down the entire repair speed, the completion time of each tree pipeline should be kept consistent. To this end, it is necessary to coordinate the amount of data transmitted according to the transmission throughput of each tree pipeline. Therefore, for a related storage block of size M, the size of the sub-block allocated to the jth tree pipeline is M j :

[0052]

[0053] Among them, S i represents the throughput of the i-th node on the j-th tree pipeline. For example, S jThe jth sub-block is sent at a speed of 10 Mbps, controlled by the Aggregation Control Protocol (ACP). These sub-blocks are packaged into ACP messages for transmission. The payload of each ACP message consists of multiple erasure code phrases from the sub-block. Therefore, data packets (i.e., ACP messages) from different helpers with the same pipeline ID can be aggregated as long as they are at the same position in the block. The header of each ACP message records the sub-block's sequence number and the pipeline ID corresponding to the sub-block. Because the routing path of the tree pipeline is pre-calculated, the intermediate nodes for data aggregation are also predetermined. Therefore, when ACP messages with the same sub-block sequence number and pipeline ID pass through the same node, the node aggregates these ACP messages by performing an exclusive OR operation and then sends the aggregated ACP message to the next node along the specified path. For ACP messages that do not require aggregation, the node directly forwards and transmits them through link sharing.

[0054] Whether an ACP message is a data packet that requires aggregation or a normal data packet that does not require aggregation is marked by the sub-block processing identifier carried by the ACP message. If the ACP message carries an aggregation identifier, it means that the ACP message received at the current node is a data packet that requires aggregation. If the ACP message carries a forwarding identifier, it means that the ACP message received at the current node is a normal data packet that does not require aggregation. The sequence number of the sub-block is automatically assigned to the sub-block by the aggregation control protocol (ACP). During normal transmission, the receiving end receives data packets with consecutive sequence numbers. In this way, even if a small number of ACP messages are out of order, the receiving end can recover the ACP message out of order based on the sub-block sequence number.

[0055] S16, the next node aggregates the ACP message according to the sub-block processing identifier, the unique identifier of the tree pipeline corresponding to the sub-block, and the sequence number, and then sends it to the subsequent node or forwards the ACP message to the subsequent node.

[0056] It is understood that upon receiving an ACP message from the previous node, the next node transmitting on the routing path of the tree pipeline can first determine whether the ACP message needs to be aggregated based on the sub-block processing identifier of the ACP message. If aggregation is required, the next node will aggregate other received ACP messages with the same sequence number and pipeline ID as the ACP message, and then send the aggregated new ACP message to the subsequent node. If aggregation is not required, the next node will directly forward and transmit the message to the subsequent node through link sharing.

[0057] S18, when the helper completes the transmission of the relevant storage block, the requester collecting data at the end of each tree pipeline decapsulates all received ACP messages into sub-blocks according to each unique identification number and generates a repair block;

[0058] S20: The requester repairs the failed storage block using the repair block.

[0059] It can be understood that the requester is responsible for collecting data at the end of each tree pipeline to repair the failed storage block on the failed node. When each helper completes the transmission of its related storage block, the requester assembles the sub-blocks after unpacking the received ACP message according to the pipeline ID, thereby regenerating the repair block B that is identical to the failed storage block. * .

[0060] The above-mentioned parallelized network aggregation repair method, for each helper that provides relevant storage blocks, splits the relevant storage blocks into multiple sub-blocks through the helper and then uses multiple parallel tree pipelines constructed by Paint for parallel transmission. During the transmission process, each sub-block is transmitted in the form of a data packet of an ACP message. Since each tree pipeline has its own unique identification number, each ACP message also carries the unique identification number of the tree pipeline from which the sub-block comes, and each ACP message carries a sub-block processing identifier to distinguish whether the ACP message is an information packet that needs to be aggregated after sub-block data before transmission, or an ordinary information packet that does not need aggregation and is directly forwarded. At the same time, each ACP message also carries a serial number assigned to the sub-block.

[0061] Therefore, for any node on the tree pipeline, after receiving the ACP message from the previous node, the next node immediately adjacent to the previous node can first distinguish whether the ACP message is an information packet that needs to be aggregated and transmitted, or an information packet that is directly forwarded, based on the sub-block processing identifier carried in the ACP message. If it is an information packet that needs to be aggregated and transmitted, then based on the unique identification number and sequence number carried in the ACP message, the next node will aggregate the ACP messages received from the same tree pipeline with the same sequence number before sending them to the subsequent node. Otherwise, the ACP message will be directly forwarded to the subsequent node. The requester collects data at the end of each tree pipeline. When all helpers have completed the transmission of the relevant storage block, the requester assembles all received sub-blocks according to the unique identification number, thereby regenerating the repair block to complete the repair of the failed storage block.

[0062] Compared with traditional technologies, the above solution uses Paint to improve repair performance by building multiple parallel tree-like pipelines. At the same time, it uses intermediate nodes on the routing path (such as switches or servers) to aggregate related data to avoid incast transmission. On this basis, it proposes an Aggregation Control Protocol (ACP) to manage different repair pipelines and solve the problems of message identification, reliability, and rate control faced by parallel network repair, thereby achieving the goal of significantly improving the repair performance of failed storage blocks.

[0063] In one embodiment, Figure 3 and Figure 4 As shown, before the above step S12, the following processing steps S08 to S11 may be further included:

[0064] S08, when a degraded read is triggered, the requester sends a repair request to the coordinator;

[0065] S09, after receiving the repair request, the coordinator queries the local metadata and sends the information of the relevant helper to the network controller;

[0066] S10, the network controller calculates routing paths of multiple parallel tree pipelines based on information from the relevant helpers and predicts the transmission throughput of the paths;

[0067] S11, after the network controller assigns a unique identification number to each tree-like pipeline, it sends the information of each parallel tree-like pipeline to each related helper through the coordinator, and at the same time transmits the flow rule information to the related nodes involved in the routing path of each parallel tree-like pipeline.

[0068] It is understandable that Figure 4 Figure 2 shows an overview of the Paint repair network. When a degraded read operation is triggered by reading a failed block, the requester sends a repair request to the network coordinator (which can be a secondary management server) to repair the failed storage block on the failed node. Upon receiving the repair request, the coordinator queries local metadata and sends information about the relevant helpers to the network controller (which can be a primary management server). This information includes the addresses of the helpers that can provide the relevant storage blocks and the relevant storage blocks they store.

[0069] Based on the currently received network information, the network controller calculates the routing paths of multiple tree-shaped pipelines and predicts their transmission throughput. The routing path can be calculated by separately calculating multiple parallel Steiner trees. For details, please refer to the calculation implementation method of multiple parallel tree-shaped pipelines in the traditional Paint repair method. Each tree-shaped pipeline is identified by a pipeline ID number and its predicted transmission throughput. Then, the network controller sends the calculated relevant information of each tree-shaped pipeline to the corresponding helpers through the coordinator. At the same time, the network controller transmits the relevant flow rule information to the relevant nodes involved in the network. The flow rule information includes data processing (such as aggregation or forwarding, etc.) rule information on the node, data format (such as sub-blocks packaged and transmitted in the form of ACP messages) information and the involved routing path information, etc. The flow rule information informs the relevant nodes of the routing path involved and the data processing operations required.

[0070] Through the above coordination steps, when a degraded read is triggered, the relevant helpers can be efficiently determined and multiple parallel tree-like pipelines with unique identification numbers can be calculated to quickly prepare for subsequent repair processing.

[0071] In one embodiment, Figure 5 As shown, an ACP message includes a UDP header, a Magic field, a PipeID field, a BlockID field, and a Payload field. The Magic field is used to record the aggregation identifier or forwarding identifier. The aggregation identifier indicates to nodes that the current ACP message is an information packet that needs to be aggregated, and the forwarding identifier indicates to nodes that the current ACP message is a normal information packet that is directly forwarded. The PipeID field is used to record the unique identification number of the tree pipeline corresponding to the sub-block. The BlockID field is used to record the sequence number assigned to the sub-block. The Payload field is used to record the data of the sub-block.

[0072] It can be understood that the proposed Aggregation Control Protocol (ACP) allows Paint to complete pipelined in-network aggregation. The ACP protocol proposed in this application is an extension of the traditional UDP protocol. The main reason for choosing the UDP protocol instead of the TCP protocol is that TCP is oriented towards reliable connections, but when performing in-network aggregation, some messages need to be aggregated. Therefore, when using the TCP protocol, the client will think that its message is lost, which may continuously trigger retransmission, causing the repair process to enter an infinite loop. In contrast, the UDP protocol is not subject to these constraints and is relatively scalable.

[0073] However, the inventors have found in practice that the UDP protocol is only responsible for transmitting data and cannot guarantee the reliability of transmission. Therefore, when deploying intra-network aggregation in practical applications, the following three challenges need to be further addressed: The first is identification. Ordinary messages in the network will interfere with the aggregation process. Intermediate nodes should identify and determine which data packets (also called information packets) need to be aggregated and which data packets are ordinary messages that do not need to be aggregated. In addition, due to the encoding characteristics of the erasure code itself, only related data in the same coding group can be aggregated, while data in different coding groups are unrelated and should not be aggregated. This requires intermediate nodes to identify which packets are related and which packets are unrelated before performing intra-network aggregation. Otherwise, unrelated data packets may be aggregated together and result in erroneous blocks.

[0074] The second is reliability. The UDP protocol cannot guarantee transmission reliability. Therefore, when some packets are lost or out of order, the requester may generate storage blocks with errors, which may cause disasters to the storage system.

[0075] The third is rate control. The UDP protocol cannot control the sending rate. However, blocks in erasure-coded storage systems are typically relatively large (for example, HDFS defaults to 64MB). Without effective rate control, the send and receive queues can easily overflow, resulting in persistent packet loss.

[0076] To address the above challenges, the ACP message of this application extends the traditional UDP packet. A standard ACP message is as follows: Figure 5 As shown in the figure, three fields are added after the UDP header (i.e., the existing UDP Header). The Magic field consists of a unique bit sequence that records the aggregation identifier or forwarding identifier, used to distinguish packets to be aggregated from ordinary packets in the network. The PipeID field records the unique identification number of the tree pipeline corresponding to the sub-block, used to mark which tree pipeline the data packet belongs to. This is based on Paint's use of multiple parallel tree pipelines to repair failed blocks. The BlockID field further identifies the sequence number of each packet. The sequence number can be numbered in a cyclic manner, and the maximum value depends on the number of intermediate nodes. In the current implementation, one example implementation is to set the Magic field to 16 bits, the PipeID field to 8 bits, and the BlockID field to 16 bits. In different application scenarios, other bit sizes can be selected according to actual needs. At the end of the ACP message is the Payload field, which stores data from the helper storage block. The size of this field can be adjusted according to different network conditions, but should be an integer multiple of the erasure code word size. For example, in the current implementation, the Payload field size can be set to 8KB.

[0077] Solving the Identification Problem: To repair a failed memory block, the network controller first calculates a routing path based on network information, which consists of multiple directed Steiner trees (DSTs). Each DST is a routing path for an independent tree pipeline, labeled with a unique pipeline ID. The network controller then transmits this information to the relevant nodes in the network. Each memory node selected as a helper divides the relevant memory block into multiple sub-blocks and transmits them through each tree pipeline separately. Packets passing through each tree pipeline are labeled with the pipeline ID in their PipeID field.

[0078] like Figure 6 Figure 2 shows an example of a failed storage block to be repaired. H1, H2, and H3 are nodes that store the corresponding related storage blocks (B1, B2, and B3) and serve as helpers. H5 is a requester. S1 through S7 are programmable switches. Assume that host H4 experiences a degraded read failure, and requester H5 needs to repair failed storage block B4 on H4.

[0079] like Figure 7The figure shows an example diagram of repairing a failed storage block. The repair process is split into two concurrent tree-shaped pipelines, where the predicted throughput of pipeline 1 is 14 Mbps and the predicted throughput of pipeline 2 is 8 Mbps, assuming that the block size is 8 MB.

[0080] The other nodes involved in the tree pipeline can be divided into two categories. One is the forwarding node, which only forwards the received ACP message according to the matching action table. The forwarding of the ACP message is completed by identifying the Magic field and PipeID field of the message. The other is called the aggregation node. The network controller specifies the number of its (aggregation node) child nodes, that is, how many packets each tree pipeline needs to aggregate at these nodes. Figure 6 For example, a feasible repair path can be constructed as Figure 7 For example, the programmable switch S3 is an aggregation node that should aggregate packets from the 2 child nodes of pipeline 1 and the 2 child nodes of pipeline 2.

[0081] The design of the above ACP message can effectively solve the identification problem.

[0082] In one embodiment, in step S16, the process of aggregating the ACP messages and sending them to subsequent nodes may specifically include the following processing:

[0083] When the next node receives an ACP message with a new sequence number, it temporarily stores the ACP message;

[0084] When the number of temporarily stored ACP messages with the same unique identification number and sequence number of the tree pipeline reaches a set value, the next node aggregates the temporarily stored ACP messages and sends them to the subsequent node.

[0085] It can be understood that each time the next node (such as Figure 5 When a programmable switch S3 receives an ACP message with a new BlockID field, it temporarily stores the data in the ACP message and waits for other messages with the same PipeID and BlockID fields. When the number of temporarily stored ACP messages reaches a specified value, these messages are aggregated and forwarded to the next node. The specified value can be set for the intermediate nodes as needed.

[0086] Through the above processing, intra-network aggregation control of ACP messages that need to be aggregated can be effectively and reliably achieved on the intermediate nodes.

[0087] In one embodiment, the parallelized network aggregation repair method may further include the following processing steps:

[0088] After receiving a set number of ACP messages, the receiver sends multiple confirmation messages to notify all senders; the receiver is the requester and the senders are the helpers;

[0089] After receiving the confirmation message, each sender continues to send the next batch of ACP messages with a set number.

[0090] As you can understand, the ACP protocol addresses the identification problem by setting the Magic field, PipeID field, and BlockID field in the data packet. To further address the remaining two challenges, the ACP protocol also establishes certain specifications at the application layer. As mentioned above, the BlockID field in the ACP message assigns a sequence number to each message. Under normal circumstances, when there is no packet loss or out-of-order transmission in the network, the receiver should receive packets with consecutive numbers in the BlockID field. Even if a small number of packets are out-of-order, the receiver can recover the out-of-order packets based on the BlockID field.

[0091] However, when many messages are erroneous or even lost, the information in the ACP message alone is insufficient to resolve the issue. To address this, the ACP protocol introduces an acknowledgment mechanism similar to the TCP protocol at the application layer. After receiving a set number of messages, the receiver sends multiple acknowledgment messages (i.e., ACK messages) to notify all senders. These senders then proceed to send the next batch of data packets. The number of packets in each batch (denoted as τ) can be dynamically adjusted based on network latency and packet loss, or a fixed value can be used. This allows both the sender and receiver to achieve dynamic rate control. Specifically, the value of τ should be less than or equal to the maximum value of the BlockID field. Thus, through the aforementioned processing steps, when multiple messages are out of order, the receiver can accurately recover the out-of-order messages based on the BlockID field of the messages.

[0092] In one embodiment, the parallelized network aggregation repair method may further include the following processing steps:

[0093] When the receiver does not receive the set number of ACP packets within the set time, the timer will time out and instruct all senders to resend the set number of ACP packets;

[0094] Before resending an ACP message, each sender sends a specific ACP message through the corresponding tree pipeline. The specific ACP message is used to instruct the intermediate nodes on the corresponding tree pipeline to clear the previously stored ACP message.

[0095] After receiving the specific ACP message from each sender, the receiver returns a confirmation message to each sender;

[0096] After receiving the confirmation message, each sender retransmits the previous set number of ACP messages.

[0097] As you can understand, when τ packets are not received for a certain period of time (i.e., a set duration), the receiver triggers a timer timeout and requires all senders to resend τ packets. It is important to note that in the case of intra-network aggregation, retransmission is not simply equivalent to traditional message retransmission. This is because intermediate nodes in the network retain data from previous packets, which can lead to aggregation errors. To avoid this, intermediate nodes should clear their stored data before retransmitting packets. The set duration can be flexibly set according to the actual needs of intra-network aggregation repair.

[0098] Specifically, the solution provided by this embodiment is that each sender sends a special ACP packet, that is, a specific ACP message, through each corresponding tree pipeline before retransmitting data. The Magic field and PipeID field of this specific ACP message are the same as other packets on the tree pipeline. The difference is that all bits in the BlockID field are 1 and there is no load information in the Payload field. When an intermediate node receives a specific ACP message, the node will automatically clear the data of the previously stored data packet. After receiving these specific ACP messages from each sender, the receiver returns an ACK message to each sender. Then, these senders return to the starting point of the previous batch of data packets and retransmit the previous batch of data packets.

[0099] Through the above processing steps, the problem of packet disorder or even packet loss can be handled more effectively, thereby better ensuring the reliability of the repair process.

[0100] In one embodiment, a register array idx, a register array data, and a register array cnt are pre-set in each programmable switch. The register array idx is used to store the BlockID field of each ACP message, the register array data is used to store the Payload field of each ACP message, and the register array cnt is used to calculate the XOR number of each BlockID value.

[0101] As you can understand, this embodiment also describes a specific implementation on a programmable switch. Paint is implemented on a programmable switch to aggregate related data packets across the network, with the P4 language being the preferred language for this aggregation process. When an ACP message arrives at the programmable switch, it parses it and performs a series of actions based on the matching action table, matching the message header. Because some ACP messages require aggregation on the programmable switch, previously arriving ACP messages need to be temporarily stored pending the arrival of related ACP messages. To achieve this, the register array of the P4 programmable switch is used to cache data, namely the payload fields of these packets. When related packets arrive, the programmable switch aggregates them by performing an exclusive-OR operation on the payload fields of all related packets. After all related packets are aggregated, the data in the register array is written to the payload field of the last arriving ACP message, which is then sent to the next node. This eliminates incast traffic on the network through programmable switch aggregation, thereby improving repair performance.

[0102] Implementation details such as Figure 8 As shown. In the programmable switch, three types of register arrays are pre-set for each pipeline, namely register array idx, register array data and register array cnt. The number of storage buckets in these three arrays is the same and corresponds one to one. Register array idx is mainly used to store the BlockID field of each packet. Array data is used to store the Payload field of the data packet, and array cnt is used to calculate the XOR number of each BlockID value. In some embodiments, the inventors found that the size of each bucket in the register array is limited to 2048 bits by the system. Therefore, in order to store the Payload field of the data packet, each bucket of the register array data is composed of multiple register arrays.

[0103] When each packet arrives at a programmable switch, it first parses the message. It then determines whether it requires aggregation based on the message's Magic field. If so, it updates multiple register arrays based on the message's PipeID and BlockID fields. Otherwise, the message is forwarded according to the standard flow table.

[0104] like Figure 8The right side of the figure shows a simplified P4 language pseudocode example to describe the specific implementation process in detail. Lines 1-4 of the pseudocode example are used to determine whether the incoming data packet needs to be aggregated, where MAGIC is a unique value that matches the ACP message. If the value of the Magic field is equal to MAGIC (line 4), the programmable switch will perform an operation based on the PipeID field of the message to process the data packet (lines 5-8).

[0105] For a data packet belonging to a certain tree pipeline, the programmable switch queries its corresponding register array idx to determine which bucket the data is stored in (line 10). If the value of BlockID is not found, the function findID() returns the location of an empty bucket and writes the value of BlockID there. The programmable switch reads the data data in the register array tmp based on the location information (line 11) and determines whether it is empty. If not, the programmable switch performs an XOR operation on tmp and Payload (lines 12-13). Otherwise, the value of Payload will be passed directly to tmp (lines 14-15). Then, the programmable switch updates the arrays data (line 16) and cnt (lines 17-19).

[0106] After updating the register array, the programmable switch checks whether the data should be forwarded. If the value of t_cnt is not equal to its number of child nodes, the programmable switch performs no action; instead, the aggregated data should be sent to the next node. To do this, the packet's payload field is first updated (line 1) and the corresponding position in the register array is initialized (lines 2-4). The updated data packet is then forwarded to the next node via the function set_forward_upper() (line 5). This aggregation process occurs at a linear rate on the programmable switch, and the second-generation p4 programmable Ethernet switch ASIC, Intel(R) Tofino(TM) 2, offers a throughput of up to 12.8Tb / s. Therefore, the aggregation operation on the programmable switch rarely constitutes a throughput bottleneck for the pipeline.

[0107] Through the deployment of the above-mentioned programmable switches, the performance improvement of parallelized network aggregation repair is better guaranteed.

[0108] In one embodiment, some experimental examples are provided to facilitate intuitive explanation and understanding of the above-mentioned parallelized network aggregation repair method. It should be noted that the experimental examples provided in this embodiment are not limitations of the above-mentioned method of the present application, but are examples used to assist in understanding the beneficial effects of the above-mentioned method.

[0109] Due to the device limitations of the P4 programmable switch, the prototype system is implemented based on a realistic virtual network created by Mininet. This network runs real kernel, switch, and application code. Therefore, the prototype system code can be easily migrated to a real physical network. To simulate the behavior of a programmable switch, the default Open vSwitch software switch in Mininet is replaced with a BMV2 switch. The BMV2 switch is used to implement the packet processing behavior specified by the P4 program.

[0110] In order to verify the feasibility of the prototype system, Figure 6 The example in [1] constructs a network consisting of seven programmable switches and five hosts, with the bandwidth of each link set according to the instance. In addition, the network includes a network controller that directly interacts with the programmable switches and hosts to manage the repair process for failed blocks. The Jerasure library is used to build an erasure-coding storage system, employing basic RS encoding with parameters k = 3 and m = 1. The default block size is 8MB. The coordinator runs on host H5, and the remaining hosts serve as storage servers. Before the experiment, four storage blocks are evenly distributed across hosts H1 to H4. Then, the connection between host H4 and switch S6 is disconnected to simulate a failure. On host H5, an attempt is made to read the block stored on host H4. Since host H4 is offline, the system detects that the required block is inaccessible and triggers a degraded read. Therefore, the coordinator initiates a repair operation for the failed block using a predefined repair method.

[0111] Consider the following two repair methods compared with the above Paint-based repair method. The first is the conventional repair method, denoted as CR, which is the default repair method for many erasure coded storage systems. When a block fails, CR randomly selects k helpers from the set of available storage nodes and then transmits the corresponding k blocks to the requester. In the experiment, these blocks are routed to CR using the shortest path. The second is the single-pipeline Paint repair method, denoted as SP. Like Paint, this method uses in-network aggregation to repair failed blocks, and intermediate nodes aggregate related data. The difference is that the SP method only builds one pipeline, while Paint can build multiple parallel pipelines.

[0112] like Figure 7 As shown in Figure 2, Paint will generate two independent pipelines in parallel to repair failed blocks. To verify the effectiveness of the system, the performance of the two pipelines is first evaluated. The throughput and repair time are shown in Figure 2. Figure 9 (a) and Figure 9 As shown in (b), the horizontal axis represents the number of messages sent in each batch, and Pipe represents the pipeline. Figure 9(a) shows the throughput of the two pipelines of Paint. As the number of packets sent per batch (i.e., the value of τ) increases, the throughput of the two pipelines increases. This is because the ACP protocol instructs the sender to wait for the ACK message before sending the next batch of packets.

[0113] When the number of packets processed per batch is small, the waiting time increases. This causes the sender to have a long idle time during the entire process, reducing the efficiency of bandwidth utilization. When τ = 50, the throughput of pipeline 1 reaches a maximum of 11.76Mbps, while the throughput of pipeline 2 is 7.14Mbps. However, when the value of τ exceeds 50, the throughput of pipeline 1 decreases. Too high a sending rate can cause the switch performance to degrade, resulting in packet loss. Similar effects can be seen in Figure 9 As can be seen in (b), the repair times of the two pipelines are tightly coupled at the beginning, which is in line with the system expectation. Later, when τ exceeds 60, the repair times are no longer coupled because the packets of the two pipelines are no longer processed in a timely manner.

[0114] However, analysis found that the throughput of the two pipelines did not reach the theoretical values ​​of 14 Mbps and 8 Mbps. The significant gap between the actual throughput and the theoretical value is mainly due to the performance issues of the BMV2 switch, which cannot provide high throughput for aggregation. To verify this, the throughput of a single pipeline (i.e., the SP method) is compared with the throughput of Paint's pipeline 1, as both have the same routing path. The results are shown in Figure 10(a). It can be seen that when the network processes only one pipeline, the throughput is higher than that of two parallel pipelines. For example, when τ = 70, the throughput of SP can reach 13.54 Mbps, while the throughput of pipeline 1 is only 10.67 Mbps. This verifies the aforementioned assumptions about the performance of the BMV2 switch. In addition, the official documentation of the BMV2 switch also points out that the BMV2 switch is not a production-grade software switch and its throughput is affected by various external factors. Nevertheless, the SP method verifies that the ACP protocol can achieve a throughput close to the theoretical value of 14 Mbps. Therefore, when running on a physical programmable switch, it can be confident that the ACP protocol can achieve the required repair throughput.

[0115] Due to the limitations of experimental conditions, Paint has poor throughput in a single pipeline. Nevertheless, compared with the SP method, the above-mentioned parallelized network aggregation repair method based on Paint still has significant advantages in repair performance. Figure 10As shown in (b), the repair time of the parallelized network aggregation repair method based on Paint is lower than that of the SP method, at least 3.5 seconds, which shows that parallelization can accelerate the repair process. We further set different block sizes and compare SP, the parallelized network aggregation repair method based on Paint, and the conventional repair method (CR). Figure 11 The figure shows the repair time of the three repair methods when τ = 50. SP reduces the repair time by about half compared to CR through in-network aggregation. Furthermore, the aforementioned parallelized network aggregation repair method based on Paint further reduces the repair time by using a parallel repair mode, to almost one-third of CR.

[0116] It should be understood that although Figure 2 and Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2 and Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0117] See also Figure 12 In one embodiment, a parallelized network aggregation repair system 100 is provided, comprising a helper 11, intermediate nodes 12, a requester 13, and a coordinator 14. Upon receiving a repair instruction from coordinator 14, helper 11 splits the relevant storage block into multiple sub-blocks and assigns each sub-block to the corresponding send queues of multiple tree-shaped pipelines; each tree-shaped pipeline is identified by a unique identification number and its own transmission throughput.

[0118] Based on the transmission throughput of the tree pipeline, the helper 11 packages the sub-blocks in the corresponding sending queue into corresponding ACP messages and sends them to the next node 12; the ACP message carries the sub-block processing identifier, the unique identification number of the tree pipeline corresponding to the sub-block, and the sequence number assigned to the sub-block. The sub-block processing identifier includes an aggregation identifier or a forwarding identifier. The next node 12 aggregates the ACP message based on the sub-block processing identifier, the unique identification number of the tree pipeline corresponding to the sub-block, and the sequence number, and then sends it to the subsequent node 12 or forwards the ACP message to the subsequent node 12. When the helper 11 completes the transmission of the relevant storage block, the requester 13 that collects data at the end of each tree pipeline unpacks all the received ACP messages into sub-blocks according to each unique identification number and generates a repair block. The requester 13 uses the repair block to repair the failed storage block.

[0119] The above-mentioned parallelized network aggregation repair system 100, for each helper 11 that provides relevant storage blocks, splits the relevant storage blocks into multiple sub-blocks through the helper 11 and then uses multiple parallel tree pipelines constructed by Paint for parallel transmission. During the transmission process, each sub-block is transmitted in the form of a data packet of an ACP message. Since each tree pipeline has its own unique identification number, each ACP message also carries the unique identification number of the tree pipeline from which the sub-block comes, and each ACP message carries a sub-block processing identifier to distinguish whether the ACP message is an information packet that needs to be aggregated after sub-block data is transmitted, or an ordinary information packet that does not need aggregation and is directly forwarded. At the same time, each ACP message also carries a serial number assigned to the sub-block.

[0120] Therefore, for any node 12 on the tree pipeline, after receiving the ACP message from the previous node 12, the next node 12 adjacent to the previous node 12 can first distinguish whether the ACP message is an information packet that needs to be aggregated and transmitted, or an information packet that is directly forwarded, based on the sub-block processing identifier carried in the ACP message. If it is an information packet that needs to be aggregated and transmitted, then based on the unique identification number and sequence number carried in the ACP message, the next node 12 will aggregate the ACP messages received from the same tree pipeline with the same sequence number and then send them to the subsequent node 12. Otherwise, the ACP message will be directly forwarded to the subsequent node 12. The requester 13 collects data at the end of each tree pipeline. When all helpers 11 complete the transmission of the relevant storage block, the requester 13 assembles all received sub-blocks according to the unique identification number, thereby regenerating the repair block to complete the repair of the failed storage block.

[0121] Compared with traditional technologies, the above solution uses Paint to improve repair performance by building multiple parallel tree-like pipelines, and uses intermediate nodes 12 (such as switches or servers) on the routing path to aggregate related data to avoid incast transmission. On this basis, an aggregation control protocol ACP is proposed to manage different repair pipelines to solve the problems of message identification, reliability and rate control faced by parallel network repair, thereby achieving the purpose of significantly improving the repair performance of failed storage blocks.

[0122] In one embodiment, the parallelized network aggregation repair system 100 may also be used to implement functions corresponding to the additional steps or sub-steps in each embodiment of the parallelized network aggregation repair method.

[0123] For specific limitations of the parallelized network aggregation repair system 100 , please refer to the corresponding limitations of the parallelized network aggregation repair method above, which will not be repeated here.

[0124] In one embodiment, a computer device is further provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following processing steps are implemented: a helper splits a related storage block into multiple sub-blocks and allocates them to corresponding sending queues of multiple corresponding tree-shaped pipelines; each tree-shaped pipeline is identified by a unique identification number and its own transmission throughput; based on the transmission throughput of the tree-shaped pipeline, the sub-blocks in the corresponding sending queue are packaged into corresponding ACP messages and then sent to the next node; the ACP messages carry the sub-block processing identifier, the corresponding sub-block processing identifier, and the corresponding transmission throughput of the sub-block. The unique identification number of the tree pipeline and the serial number assigned to the sub-block, the sub-block processing identifier includes an aggregation identifier or a forwarding identifier; the next node aggregates the ACP message according to the sub-block processing identifier, the unique identification number of the tree pipeline corresponding to the sub-block, and the serial number, and then sends it to the subsequent node or forwards the ACP message to the subsequent node; when the helper completes the transmission of the relevant storage block, the requester who collects data at the end of each tree pipeline unpacks all the received ACP messages into sub-blocks according to each unique identification number and generates a repair block; the requester uses the repair block to repair the failed storage block.

[0125] It can be understood that in addition to the memory and processor mentioned above, the above-mentioned computer equipment also includes other software and hardware components not listed in this specification. The specific components can be determined according to the specific server and / or programmable switch models in different application scenarios. This specification will not list them one by one in detail.

[0126] In one embodiment, when executing the computer program, the processor may further implement the additional steps or sub-steps in each embodiment of the above-mentioned parallelized network aggregation repair method.

[0127] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When executed by a processor, the computer program implements the following processing steps: a helper splits a relevant storage block into multiple sub-blocks and assigns them to the corresponding sending queues of multiple tree pipelines; each tree pipeline is identified by a unique identification number and its own transmission throughput; based on the transmission throughput of the tree pipeline, the sub-blocks in the corresponding sending queue are packaged into corresponding ACP messages and sent to the next node; the ACP messages carry a sub-block processing identifier, the unique identification number of the tree pipeline corresponding to the sub-block, and a sequence number assigned to the sub-block, where the sub-block processing identifier includes an aggregation identifier or a forwarding identifier; the next node aggregates the ACP messages based on the sub-block processing identifier, the unique identification number of the tree pipeline corresponding to the sub-block, and the sequence number, and then sends the ACP messages to a subsequent node or forwards the ACP messages to a subsequent node; when the helper completes the transmission of the relevant storage block, a requester that collects data at the end of each tree pipeline unpacks all received ACP messages into sub-blocks based on the unique identification numbers and generates a repair block; the requester uses the repair block to repair the failed storage block.

[0128] In one embodiment, when the computer program is executed by a processor, it can also implement the additional steps or sub-steps in each embodiment of the above-mentioned parallel network aggregation repair method.

[0129] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, abbreviated as RDRAM) and interface dynamic random access memory (DRDRAM).

[0130] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A parallelized network aggregation repair method, characterized in that: Including steps: The helper splits the relevant storage block into multiple sub-blocks and allocates them to the corresponding sending queues of the corresponding multiple tree-shaped pipelines; each of the tree-shaped pipelines is identified by a unique identification number and its own transmission throughput; According to the transmission throughput of the tree-shaped pipeline, the sub-blocks in the corresponding sending queue are packaged into corresponding ACP messages and then sent to the next node; the ACP messages carry the sub-block processing identifier, the unique identifier of the tree-shaped pipeline corresponding to the sub-block, and the sequence number assigned to the sub-block, wherein the sub-block processing identifier includes an aggregation identifier or a forwarding identifier; The next node aggregates the ACP message according to the sub-block processing identifier, the unique identification number of the tree-shaped pipeline corresponding to the sub-block, and the sequence number, and then sends the ACP message to a subsequent node or forwards the ACP message to a subsequent node; When the helper completes the transmission of the relevant storage block, the requester collecting data at the end of each tree-shaped pipeline unpacks all the received ACP messages into sub-blocks according to each unique identification number and generates a repair block; The requester repairs the failed storage block using the repair block.

2. The parallelized network aggregation repair method according to claim 1, characterized in that: The ACP message includes a UDP message header, a Magic field, a PipeID field, a BlockID field, and a Payload field. The Magic field is used to record the aggregation identifier or the forwarding identifier. The aggregation identifier is used to indicate to the node that the current ACP message is an information packet that needs to be aggregated. The forwarding identifier is used to indicate to the node that the current ACP message is an ordinary information packet that is directly forwarded. The PipeID field is used to record the unique identification number of the tree pipeline corresponding to the sub-block, the BlockID field is used to record the serial number allocated to the sub-block, and the Payload field is used to record the data of the sub-block.

3. The parallelized network aggregation repair method according to claim 1 or 2, characterized in that: The process of aggregating the ACP message and sending it to the subsequent node includes: When the next node receives the ACP message with the new sequence number, it temporarily stores the ACP message; When the temporarily stored number of the ACP messages having the same unique identification number and the sequence number of the tree-shaped pipeline reaches a set value, the next node aggregates the temporarily stored ACP messages and sends them to the subsequent node.

4. The parallelized network aggregation repair method according to claim 3, characterized in that: The method further comprises: After receiving a set number of the ACP messages, the receiver sends multiple confirmation messages to notify all senders; the receiver is the requester, and the sender is each of the helpers; After receiving the confirmation message, each sender continues to send the next batch of the set number of ACP messages.

5. The parallelized network aggregation repair method according to claim 4, characterized in that: The method further comprises: When the receiver does not receive the set number of ACP messages within a set time, triggering a timer timeout and instructing all the senders to resend the set number of ACP messages; Each of the senders sends a specific ACP message through the corresponding tree pipeline before resending the ACP message; the specific ACP message is used to instruct the intermediate node on the corresponding tree pipeline to clear the previously stored ACP message; After receiving the specific ACP message from each sender, the receiver returns the confirmation message to each sender respectively; After receiving the confirmation message, each sender retransmits the previous batch of the set number of ACP messages.

6. The parallelized network aggregation repair method according to claim 2, characterized in that: In each programmable switch, register array idx, register array data and register array cnt are pre-set. The register array idx is used to store the BlockID field of each ACP message, the register array data is used to store the Payload field of each ACP message, and the register array cnt is used to calculate the XOR number of each BlockID value.

7. The parallelized network aggregation repair method according to claim 1, characterized in that: Before the step of splitting the relevant storage block into multiple sub-blocks and assigning them to the corresponding sending queues of the corresponding multiple tree-shaped pipelines, the step further includes: When a degraded read is triggered, the requester sends a repair request to the coordinator; After receiving the repair request, the coordinator queries local metadata and sends information about the helper to the network controller; The network controller calculates routing paths of a plurality of parallel tree-shaped pipelines according to information related to the helper and predicts transmission throughput of the paths; After the network controller assigns a unique identification number to each of the tree-shaped pipelines, it sends the information of each of the parallel tree-shaped pipelines to the relevant helpers through the coordinator, and at the same time transmits the flow rule information to the relevant nodes involved in the routing path of each of the parallel tree-shaped pipelines.

8. A parallelized network aggregation repair system, characterized in that: The system includes a helper, intermediate nodes, and a requester. After receiving a repair instruction from the coordinator, the helper splits the relevant storage block into multiple sub-blocks and assigns them to the corresponding sending queues of multiple tree-shaped pipelines. Each of the tree-shaped pipelines is identified by a unique identification number and its own transmission throughput. According to the transmission throughput of the tree-shaped pipeline, the helper packages the sub-blocks in the corresponding sending queue into corresponding ACP messages and sends them to the next node; the ACP messages carry the sub-block processing identifier, the unique identifier of the tree-shaped pipeline corresponding to the sub-block, and the sequence number assigned to the sub-block, wherein the sub-block processing identifier includes an aggregation identifier or a forwarding identifier; The next node aggregates the ACP message according to the sub-block processing identifier, the unique identification number of the tree-shaped pipeline corresponding to the sub-block, and the sequence number, and then sends the ACP message to a subsequent node or forwards the ACP message to a subsequent node; When the helper completes the transmission of the relevant storage blocks, the requester who collects data at the end of each tree pipeline unpacks all the received ACP messages into sub-blocks according to each unique identification number and generates a repair block. The requester uses the repair block to repair the failed storage block.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the parallelized network aggregation repair method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the parallelized network aggregation repair method according to any one of claims 1 to 7 are implemented.

Images