Data transmission method, device, equipment and storage medium

By optimizing data transmission in the erasure code storage solution, and using virtual node dependency tree and predicted traffic transmission data to determine the repair sequence, the problem of network congestion during data repair in the erasure code storage is solved, and data repair efficiency is improved.

CN118378026BActive Publication Date: 2024-11-19CHINA TELECOM CORP LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410841284.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-26
Publication Date
2024-11-19
Estimated Expiration
2044-06-26

AI Technical Summary

Technical Problem

The erasure code storage solution causes network congestion during the data repair process, affecting the efficiency of data repair.

Method used

By acquiring data nodes, multiple data repair requirements are determined based on the predetermined virtual node dependency tree and the failed nodes, the repair sequence is determined using the connection relationship between nodes and predicted traffic transmission data, and data transmission is optimized to reduce network congestion.

Benefits of technology

It improves the efficiency of data repair, reduces network congestion, and ensures the efficiency of the data repair process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118378026B_ABST
    Figure CN118378026B_ABST
Patent Text Reader

Abstract

The present disclosure provides a data transmission method, apparatus, device and storage medium, which relate to the field of computer technology. The method comprises acquiring data nodes, determining multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairing failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving data repair efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Traditional distributed storage usually adopts multi-copy storage technology to provide higher reliability by storing multiple copies of data. However, multi-copy storage technology also wastes a lot of storage resources. In order to solve the above problems, the relevant technology adopts the storage solution of erasure code. The storage solution of erasure code needs to restore all the original data through data reconstruction, and then perform linear combination of data to restore the invalid coding block. However, in the storage solution of erasure code, since the coding block needs to be downloaded, a large amount of network transmission is required, which leads to network congestion and restricts the efficiency of data repair. Summary of the invention

[0003] The present disclosure provides a data transmission method, apparatus, device and storage medium, which reduce network congestion and improve the efficiency of data repair at least to a certain extent.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0005] According to one aspect of the present disclosure, a data transmission method is provided, including:

[0006] Obtain data nodes, which include failed nodes and repair nodes. The repair nodes are used to repair failed data of failed nodes.

[0007] Determine multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0008] Determine the repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0009] Repair the failed data based on the repair sequence.

[0010] In one embodiment of the present disclosure, the method further includes:

[0011] Obtain the upstream and downstream traffic data of each data node within a preset historical time period;

[0012] The predicted traffic transmission data of each data node within a preset time period is determined based on the uplink and downlink traffic data and the prediction model.

[0013] In one embodiment of the present disclosure, the predicted traffic transmission data of each data node within a preset time period is determined based on the uplink and downlink traffic data and the prediction model, including:

[0014] Normalize the uplink and downlink traffic data to obtain the feature vector corresponding to the data node;

[0015] The feature vector is input into the prediction model to obtain the predicted traffic transmission data of each data node within the preset time period.

[0016] In one embodiment of the present disclosure, the method further includes:

[0017] Obtaining uplink and downlink traffic data in the first historical time period and the second historical time period;

[0018] The intelligent model is trained based on the uplink and downlink traffic data in the first historical time period and the second historical time period to obtain a trained prediction model.

[0019] In one embodiment of the present disclosure, the method further includes:

[0020] Determine whether the available bandwidth of the repair node meets a preset threshold based on the predicted traffic transmission data of each data node within a preset time period;

[0021] Based on the predetermined virtual node dependency tree, failed nodes, and repair nodes, multiple data repair requirements are determined, including:

[0022] Multiple data repair requirements are determined based on a predetermined virtual node dependency tree, failed nodes, and repair nodes whose available bandwidth meets a preset threshold.

[0023] In one embodiment of the present disclosure, a repair sequence of multiple data repair requirements is determined based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm, including:

[0024] Weights are assigned to the edges between the nodes based on the connection relationship between the nodes, and the first weighted value of each edge in each data repair requirement is obtained;

[0025] Weights are assigned to the edges between the nodes based on the predicted traffic transmission data between the nodes to obtain a second weight value for each edge;

[0026] Determine a final weighted value of each edge based on the first weighted value and the second weighted value of each edge in each data repair requirement;

[0027] The data repair sequence of each edge is determined based on the final weight value of each edge, the time point corresponding to each edge in each data repair requirement, and the preset algorithm.

[0028] In one embodiment of the present disclosure, the method further includes:

[0029] Construct a virtual node dependency tree, where the virtual node dependency tree is a full binary tree, and the depth data of the full binary tree is the same as the number of failed nodes.

[0030] According to another aspect of the present disclosure, there is provided a data transmission device, comprising:

[0031] A first acquisition module is used to acquire data nodes, where the data nodes include failed nodes and repair nodes, and the repair nodes are used to repair failed data of failed nodes;

[0032] A first determination module is used to determine multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0033] A second determination module is used to determine a repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0034] The repair module is used to repair the failed data based on the repair sequence.

[0035] In one embodiment of the present disclosure, the device further includes:

[0036] The second acquisition module is used to obtain the uplink and downlink traffic data of each data node within a preset historical time period;

[0037] The third determination module is used to determine the predicted traffic transmission data of each data node within a preset time period based on the uplink and downlink traffic data and the prediction model.

[0038] In one embodiment of the present disclosure, the third determining module includes:

[0039] A normalization unit, used to normalize the uplink and downlink traffic data to obtain a feature vector corresponding to the data node;

[0040] The prediction unit is used to input the feature vector into the prediction model to obtain the predicted traffic transmission data of each data node within a preset time period.

[0041] In one embodiment of the present disclosure, the device further includes:

[0042] A third acquisition module is used to acquire the uplink and downlink traffic data in the first historical time period and the second historical time period;

[0043] The training module is used to train the intelligent model based on the uplink and downlink traffic data in the first historical time period and the second historical time period to obtain a trained prediction model.

[0044] In one embodiment of the present disclosure, the device further includes:

[0045] A fourth determination module, configured to determine whether the available bandwidth of the repair node meets a preset threshold based on the predicted traffic transmission data of each data node within a preset time period;

[0046] The first determination module includes:

[0047] The first determining unit is used to determine multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node whose available bandwidth meets a preset threshold.

[0048] In one embodiment of the present disclosure, the second determining module includes:

[0049] A first weighting unit, used to weight the edges between the nodes based on the connection relationship between the nodes, and obtain a first weighted value of each edge in each data repair requirement;

[0050] A second weighting unit is used to weight the edges between the nodes based on the predicted traffic transmission data between the nodes to obtain a second weight value for each edge;

[0051] A second determining unit, configured to determine a final weighted value of each edge based on the first weighted value and the second weighted value of each edge in each data repair requirement;

[0052] The third determining unit is used to determine the data repair sequence of each edge based on the final weight value of each edge, the time point corresponding to each edge in each data repair requirement, and a preset algorithm.

[0053] In one embodiment of the present disclosure, the device further includes:

[0054] The construction module is used to construct a virtual node dependency tree, wherein the virtual node dependency tree is a full binary tree, and the depth data of the full binary tree is the same as the number of failed nodes.

[0055] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned data transmission method by executing the executable instructions.

[0056] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data transmission method is implemented.

[0057] The data transmission method, apparatus, device and storage medium provided by the embodiments of the present disclosure obtain data nodes, determine multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, and repair the failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0060] Figure 1 A flow chart of a data transmission method in an embodiment of the present disclosure is shown;

[0061] Figure 2 Another data transmission method flow chart in the embodiment of the present disclosure is shown;

[0062] Figure 3 A flow chart of another data transmission method in an embodiment of the present disclosure is shown;

[0063] Figure 4 A flow chart of another data transmission method in an embodiment of the present disclosure is shown;

[0064] Figure 5 A flow chart of another data transmission method in an embodiment of the present disclosure is shown;

[0065] Figure 6 A flow chart of another data transmission method in an embodiment of the present disclosure is shown;

[0066] Figure 7 A flow chart of another data transmission method in an embodiment of the present disclosure is shown;

[0067] Figure 8 A data transmission device architecture diagram according to an embodiment of the present disclosure is shown;

[0068] Fig. 9 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0069] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0070] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0071] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0072] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0073] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0074] In order to solve the above problems, the embodiments of the present disclosure provide a data transmission method, apparatus, device and storage medium.

[0075] In order to explain the present disclosure in detail, Figure 1 A flow chart of a data transmission method in an embodiment of the present disclosure is shown.

[0076] like Figure 1 As shown, the method may include:

[0077] S110, obtaining data nodes, where the data nodes include failed nodes and repair nodes, and the repair nodes are used to repair failed data of failed nodes.

[0078] In some embodiments, the data node may be a node for storing data blocks, and the data blocks may be data blocks obtained by splitting based on erasure coding technology.

[0079] The failed node may be the storage node corresponding to the failed data.

[0080] Repairing the failed data of the failed node may include: if the data blocks and parity blocks of the erasure code fail, the failed data needs to be reconstructed.

[0081] In some embodiments, an erasure coding scheme can be represented by RS(n, k), with two parameters n and k, where n and k are constants and k < n. Through linear combinations over the Galois field, k original data chunks are encoded into n encoded chunks of the same size. After that, it can be ensured that any k of the n encoded chunks can reconstruct the original k data chunks. That is, any n - k encoded chunks can be tolerated to fail. These n encoded chunks are called a stripe, and the stripe is redistributed among n data nodes, with each node having one encoded chunk.

[0082] In some embodiments, the data transmission method flow in the present disclosure may be triggered by a failure event, and the failure event may be the failure of a data node. The failure of the data node can be detected based on the abnormal heartbeat message between the name node and the data node, and the name node notifies the coodringator. In another case, the failure event may be the failure of a disk, and in this case, it can be detected by the name node's self-check and then notified to the coodringator.

[0083] S120, determining multiple data repair requirements based on a pre-determined virtual node dependency tree, the failed node, and the repair nodes; each data repair requirement includes a failed node and multiple repair nodes.

[0084] The virtual node dependency tree may be a subtree of a full binary tree with a depth of k = log2(n + 1), and the root node of the subtree is vk+1.

[0085] S130, determining the repair sequence of multiple data repair requirements based on the connection relationship between the nodes in each data repair requirement, predicting the traffic transmission between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm.

[0086] In some embodiments, there may be multiple data repair requirements within a time period.

[0087] S140, repairing the failed data based on the repair sequence.

[0088] In some embodiments, the failed data may be repaired in the order of the repair sequence.

[0089] In some embodiments, the coordinator can transmit data according to the predicted traffic volume provided by the centralized controller to complete the allocation and scheduling of data transmission tasks required for repair.

[0090] For example, for the reconstruction of a failed node, the coordinator will arrange agents on k data nodes to send data and perform encoding and decoding calculations. One data node completes the final calculation summary and serves as the new data node of the failed block.

[0091] The data transmission method provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0092] In order to explain the present disclosure in detail, Figure 2 A flow chart of another data transmission method in an embodiment of the present disclosure is shown.

[0093] like Figure 2 As shown, the method may include:

[0094] S210, acquiring data nodes, where the data nodes include failed nodes and repair nodes, where the repair nodes are used to repair failed data of the failed nodes;

[0095] S220, determining multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0096] S230, obtaining the uplink and downlink traffic data of each data node within a preset historical time period.

[0097] In some embodiments, the centralized controller can periodically obtain the traffic data of each top-of-rack switch and send it to the model training cluster at a fixed time. The model training cluster then uses the preset model parameters as initial values ​​to refit the acquired data. After the training is completed, the updated model parameters are fed back to the centralized controller. The centralized controller sends the upstream and downstream traffic data within the preset time period to the coordinator.

[0098] S240: Determine predicted traffic transmission data of each data node within a preset time period based on the uplink and downlink traffic data and the prediction model.

[0099] S250, determining a repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0100] S260: Repair the failed data based on the repair sequence.

[0101] The data transmission method provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0102] In order to explain the present disclosure in detail, Figure 3 A flow chart of another data transmission method in an embodiment of the present disclosure is shown.

[0103] like Figure 3 As shown, the method may include:

[0104] S310, acquiring data nodes, where the data nodes include failed nodes and repair nodes, where the repair nodes are used to repair failed data of failed nodes;

[0105] S320, determining multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0106] S330, obtaining the uplink and downlink traffic data of each data node within a preset historical time period.

[0107] S340, normalize the uplink and downlink traffic data to obtain a feature vector corresponding to the data node.

[0108] S350, inputting the feature vector into a prediction model to obtain predicted traffic transmission data of each data node within a preset time period.

[0109] S360, determining a repair sequence of multiple data repair requirements based on a connection relationship between nodes in each data repair requirement, predicted traffic transmission data between nodes, a time point corresponding to each data repair requirement, and a preset algorithm;

[0110] S370, repairing the failed data based on the repair sequence.

[0111] The data transmission method provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0112] In order to explain the present disclosure in detail, Figure 4 A flow chart of another data transmission method in an embodiment of the present disclosure is shown.

[0113] like Figure 4 As shown, the method may include:

[0114] S410, acquiring data nodes, where the data nodes include failed nodes and repair nodes, where the repair nodes are used to repair failed data of the failed nodes;

[0115] S420, determining multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0116] S430, obtaining the uplink and downlink traffic data of each data node within a preset historical time period.

[0117] S440, obtaining uplink and downlink traffic data in the first historical time period and the second historical time period;

[0118] S450, training the intelligent model based on the uplink and downlink traffic data in the first historical time period and the second historical time period to obtain a trained prediction model.

[0119] In some embodiments, the intelligent model may include a multi-layer LSTM model.

[0120] In some embodiments, training the intelligent model may also include determining whether a loss function value corresponding to the intelligent model reaches a preset threshold, and determining that the training is completed when the loss function value reaches the preset threshold.

[0121] S460, normalize the uplink and downlink traffic data to obtain a feature vector corresponding to the data node.

[0122] S470, input the feature vector into the prediction model to obtain the predicted traffic transmission data of each data node within a preset time period.

[0123] S480, determining a repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0124] S490, repairing the failed data based on the repair sequence.

[0125] The data transmission method provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0126] In order to explain the present disclosure in detail, Figure 5 A flow chart of another data transmission method in an embodiment of the present disclosure is shown.

[0127] like Figure 5 As shown, the method may include:

[0128] S510, acquiring data nodes, where the data nodes include failed nodes and repair nodes, where the repair nodes are used to repair failed data of failed nodes;

[0129] S520: Determine whether the available bandwidth of the repair node meets a preset threshold based on the predicted traffic transmission data of each data node within a preset time period.

[0130] S530, determining multiple data repair requirements based on a predetermined virtual node dependency tree, failed nodes, and repair nodes whose available bandwidth meets a preset threshold.

[0131] In some embodiments, determining multiple data repair requirements based on a predetermined virtual node dependency tree, failed nodes, and repair nodes whose available bandwidth meets a preset threshold may include mapping the multiple data repair requirements determined by the failed nodes and the repair nodes whose available bandwidth meets a preset threshold to the virtual node dependency tree to obtain a transmission relationship for each failed node.

[0132] In some embodiments, failed nodes may be mapped in advance.

[0133] In some embodiments, the above-mentioned failed nodes and repair nodes whose available bandwidth meets a preset threshold may be randomly mapped, and then the virtual node dependency tree of the unmapped time-limited nodes may be excluded.

[0134] Exemplarily, mapping may be performed according to the following algorithm.

[0135] enter:

[0136] Phy_nodes #Data node collection

[0137] vdts #vdt collection of all data nodes in the pending queue

[0138] repairBW #Repair the minimum bandwidth required for data transmission

[0139] Output:

[0140] map_set #Enter the mapping of all virtual nodes to data nodes contained in vdts

[0141] Node_innum=[0] sizeof(phy_nodes) #node_innum records the number of download tasks for all current data nodes

[0142] Leaf_set=

[0143] Root_set=

[0144] Inner_set=

[0145] For vdt in vdts:

[0146] root_set.add(vdt.root)

[0147] leaf_set.add(vdt.root)

[0148] inner_st.add(vdt.nodes-root_set-leaf_set)

[0149] # Map data nodes to virtual internal nodes

[0150] Map_set = φ

[0151] Sort(innerset) compares the in-degrees of the nodes and arranges them in ascending order. If the in-degrees are the same, the comparison is based on the bandwidth information provided by the SDN controller.

[0152] while innerset != φ do

[0153] v = POP(innerset)

[0154] for i in phy_nodes:

[0155] if i has v.chunk and i.downloand_bandwidth>=repairBW: A data node needs to meet two conditions to enter the candidate mapping set. It has a chunk of data stripes to be repaired, that is, there is a failed node, and its available bandwidth is greater than the preset threshold repairBW

[0156] reviving_set.add(i)

[0157] dl=Sort(reviving_set) in ascending order of element's in_degree

[0158] Map_set.add((v, dl[0]))

[0159] Innerset= innerset-v

[0160] dl[0].in_degree = v.in_degree

[0161] dl[0].out_degree = v.out_degree

[0162] #Map physical nodes to virtual root nodes

[0163] while rootset != φ do

[0164] v = POP(rootset)

[0165] for i in phy_nodes:

[0166] if I has v.chunk and i.downloand_bandwidth>repairBW:

[0167] reviving_set.add(i) #

[0168] dl = Sort(reviving_set) in ascending order of element's in_degree

[0169] Map_set.add((v, dl[0]))

[0170] rootset=rootset-v

[0171] dl[0].in_degree = v.in_degree

[0172] dl[0].out_degree = v.out_degree

[0173] #Map data nodes to virtual leaf nodes

[0174] while leafset != φ do

[0175] v = POP(leafset)

[0176] for i in phy_nodes:

[0177] if I have v.chunk

[0178] reviving_set.add(i) and i.downloand_bandwidth>repairBW:

[0179] dl=Sort(reviving_set) in ascending order of element's out_degree

[0180] Map_set.add((v, dl[0]))

[0181] rootset=rootset-v

[0182] dl[0].in_degree = v.in_degree

[0183] dl[0].out_degree = v.out_degree

[0184] return map_set

[0185] S540, determining a repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0186] S550: Repair the failed data based on the repair sequence.

[0187] The data transmission method provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0188] In order to explain the present disclosure in detail, Figure 6 A flow chart of another data transmission method in an embodiment of the present disclosure is shown.

[0189] like Figure 6 As shown, the method may include:

[0190] S610, acquiring data nodes, where the data nodes include failed nodes and repair nodes, where the repair nodes are used to repair failed data of the failed nodes;

[0191] S620, determining multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0192] S630, weighting the edges between the nodes based on the connection relationship between the nodes, and obtaining a first weighted value of each edge in each data repair requirement.

[0193] In some embodiments, based on the connection relationship between the nodes, assigning weights to the edges between the nodes may include assigning weights to each existing edge.

[0194] S640, weighting the edges between the nodes based on the predicted traffic transmission data between the nodes to obtain a second weighted value for each edge.

[0195] In some embodiments, the transmission flow corresponding to each edge in each data repair requirement may be determined based on each data repair requirement, and then each edge may be weighted based on the transmission flow.

[0196] Exemplarily, the predicted traffic transmission data Bw between nodes can be determined based on the SDN controller, and then the amount of data s required to repair the failed node can be determined. During this transmission period, the data that can be transmitted is Bw / s. Each edge can be weighted based on the above transmitted data.

[0197] Exemplarily, the amount of data to be transmitted between the two nodes nchunk may also be obtained. Weights are assigned to each edge based on the minimum value between the amount of data to be transmitted between the two nodes nchunk and Bw / s.

[0198] S650: Determine a final weighted value of each edge based on the first weighted value and the second weighted value of each edge in each data repair requirement.

[0199] In some embodiments, the first weighted value and the second weighted value may be subjected to various mathematical operations to obtain a final weighted value, which is not specifically limited in the present disclosure.

[0200] S660, determining a data repair sequence for each edge based on the final weight value of each edge, the time point corresponding to each edge in each data repair requirement, and a preset algorithm.

[0201] In some embodiments, the preset algorithm may include a DICNIC algorithm.

[0202] S670: Repair the failed data based on the repair sequence.

[0203] In some embodiments, the priority of the node may also be determined based on the in-degree of the data node.

[0204] In some embodiments, node priority may also be determined based on the timestamp of the node.

[0205] In some embodiments, when the timestamps are the same, the destination data node with a smaller in-degree has a higher priority among all data transmission requests.

[0206] Exemplarily, determining a repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, predicted traffic transmission data between nodes, a time point corresponding to each data repair requirement, and a preset algorithm may include:

[0207] Create the initial data flow graph DG0:

[0208] The graph consists of 2n+2 nodes, including the source node S0, the terminal node R0,

[0209] The sending nodes are {S1, S2, ••• , Sn}, and the receiving nodes are {R1, R2, ••• , Rn}. Si and Ri represent all active nodes in the system.

[0210] If there is an edge from Si to Rj in pdts, add an edge (Si, Rj) to DG0 with a weight of 1.

[0211] Add an edge from S0 to any sending node Si, with an edge weight of n

[0212] Add an edge from any receiving node Ri to R0, with an edge weight of n

[0213] Adjust the weight of the initial data flow graph DG0 based on the network information provided by the SDN controller:

[0214] Assume that the SDN controller gives the available traffic Bw of the link between nodes, and assume that the data chunk size is s, then the number of chunks that can be sent in the current time slot is Bw / s. Query pdts to get the data sending request between nodes as nchunk. Take the minimum value between the two, update the weight of the edge between the sending node and the receiving node to min (Bw / s, nchunk), and the new graph obtained is DGn.

[0215] Execute the Dicnic algorithm on DGn to search for the maximum flow and obtain the maximum flow graph DGmax

[0216] Schedule the transmission order for each Si on DGmax and generate inter-node transmission commands inter_cmds

[0217] PDTs may have multiple edges starting from Si. In this case, the transmission demands represented by these edges need to be scheduled. The scheduling principles are as follows:

[0218] First, sort by the number of edges to the same destination node. The more edges to the same node Rj, the higher the priority of these edges.

[0219] When the number of edges with the same destination is the same, the priority is as follows:

[0220] Edges with smaller timestamps have higher priority

[0221] The timestamps are the same, and the destination node with smaller in-degree and smaller in-degree in all PDTs has higher priority.

[0222] When both of the above two items are the same, the order of the edges is randomly arranged.

[0223] The Coordinator generates a transmission instruction for each sorted edge and attaches a timestamp. It is packaged into a transmission macro instruction in the order of the above scheduling and sent to the agent of the source node Si, which is executed by the latter in the current time slot.

[0224] At the end of the time slot, the agent feeds back the unfinished sdraggler instructions to the coordinator, which will continue to schedule them next time.

[0225] The data transmission method provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0226] In order to explain the present disclosure in detail, Figure 7 A flow chart of another data transmission method in an embodiment of the present disclosure is shown.

[0227] like Figure 7 As shown, the method may include:

[0228] S710, acquiring data nodes, where the data nodes include failed nodes and repair nodes, where the repair nodes are used to repair failed data of the failed nodes;

[0229] S720, construct a virtual node dependency tree, wherein the virtual node dependency tree is a full binary tree, and the depth data of the full binary tree is the same as the number of failed nodes.

[0230] Exemplarily, constructing a virtual node dependency tree may include:

[0231] Input: Node list V1…Vk+1

[0232] Output: node dependency tree vdt

[0233] vdt.root = Vk+1

[0234] new_node_set={}

[0235] while True:

[0236] for i in vdt do:

[0237] #First, evenly add an edge to all data nodes that currently have no inbound and outbound edges

[0238] if i.degree<1:

[0239] new_node_set.add_node(vvdt.node_num+1)

[0240] vdt.add_edge(i, vvdt.node_num+1)

[0241] if (vdt.node_num+ new_node_set.size)==k+1:

[0242] dt.add_nodes(new_node_set)

[0243] #At this point, the number of edges of all data nodes is distributed between [0,1], achieving the goal of evenly distributed dependence, and vdt can be returned

[0244] return vdt

[0245] for i in vdt do:

[0246] #Then evenly add an edge to all data nodes that have only one inbound and outbound edge

[0247] if i.degree<2:

[0248] new_node_set.add_node(vdt.node_num+1)

[0249] vdt.add_edge(i, vvdt.node_num+1)

[0250] if (vdt.node_num+ new_node_set.size)==k+1:

[0251] vdt.add_nodes(new_node_set)

[0252] #At this point, the number of edges of all data nodes is distributed between [1,2], achieving the goal of evenly distributed dependence, and vdt can be returned

[0253] return vdt

[0254] vdt.add_nodes(new_node_set)

[0255] S730, determining multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0256] S740, determining a repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0257] S750: Repair the failed data based on the repair sequence.

[0258] The data transmission method provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0259] Based on the same inventive concept, the present disclosure also provides a data transmission device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0260] Figure 8 FIG. 4 shows an architecture diagram of a data transmission device in an embodiment of the present disclosure. Figure 8 As shown, the data transmission device 800 may include:

[0261] A first acquisition module 810 is used to acquire data nodes, where the data nodes include failed nodes and repair nodes, and the repair nodes are used to repair failed data of failed nodes;

[0262] A first determination module 820 is used to determine multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0263] A second determination module 830 is used to determine a repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0264] The repair module 840 is used to repair the failed data based on the repair sequence.

[0265] The data transmission device provided by the embodiments of the present disclosure obtains data nodes, determines multiple data repair requirements based on a pre-determined virtual node dependency tree, failed nodes and repair nodes, repairs the failed data based on the repair sequence based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement and the repair sequence of the multiple data repair requirements determined by a preset algorithm, thereby reducing network congestion and improving the efficiency of data repair.

[0266] In one embodiment of the present disclosure, the device further includes:

[0267] The second acquisition module is used to obtain the uplink and downlink traffic data of each data node within a preset historical time period;

[0268] The third determination module is used to determine the predicted traffic transmission data of each data node within a preset time period based on the uplink and downlink traffic data and the prediction model.

[0269] In one embodiment of the present disclosure, the third determining module includes:

[0270] A normalization unit, used to normalize the uplink and downlink traffic data to obtain a feature vector corresponding to the data node;

[0271] The prediction unit is used to input the feature vector into the prediction model to obtain the predicted traffic transmission data of each data node within a preset time period.

[0272] In one embodiment of the present disclosure, the device further includes:

[0273] A third acquisition module is used to acquire the uplink and downlink traffic data in the first historical time period and the second historical time period;

[0274] The training module is used to train the intelligent model based on the uplink and downlink traffic data in the first historical time period and the second historical time period to obtain a trained prediction model.

[0275] In one embodiment of the present disclosure, the device further includes:

[0276] A fourth determination module, configured to determine whether the available bandwidth of the repair node meets a preset threshold based on the predicted traffic transmission data of each data node within a preset time period;

[0277] The first determination module includes:

[0278] The first determining unit is used to determine multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node whose available bandwidth meets a preset threshold.

[0279] In one embodiment of the present disclosure, the second determining module includes:

[0280] A first weighting unit, used to weight the edges between the nodes based on the connection relationship between the nodes, and obtain a first weighted value of each edge in each data repair requirement;

[0281] A second weighting unit is used to weight the edges between the nodes based on the predicted traffic transmission data between the nodes to obtain a second weight value for each edge;

[0282] A second determining unit, configured to determine a final weighted value of each edge based on the first weighted value and the second weighted value of each edge in each data repair requirement;

[0283] The third determining unit is used to determine the data repair sequence of each edge based on the final weight value of each edge, the time point corresponding to each edge in each data repair requirement, and a preset algorithm.

[0284] In one embodiment of the present disclosure, the device further includes:

[0285] The construction module is used to construct a virtual node dependency tree, wherein the virtual node dependency tree is a full binary tree, and the depth data of the full binary tree is the same as the number of failed nodes.

[0286] The data transmission device provided in the embodiment of the present disclosure can be used to execute the positioning method provided in the above-mentioned method embodiments. The implementation principle and technical effect are similar and will not be described herein for the sake of simplicity.

[0287] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0288] Refer to the following Fig. 9 The electronic device 900 according to this embodiment of the present disclosure is described. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0289] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0290] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps of the above method embodiment:

[0291] Obtain data nodes, which include failed nodes and repair nodes. The repair nodes are used to repair failed data of failed nodes.

[0292] Determine multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each data repair requirement includes a failed node and multiple repair nodes;

[0293] Determine the repair sequence of multiple data repair requirements based on the connection relationship between nodes in each data repair requirement, the predicted traffic transmission data between nodes, the time point corresponding to each data repair requirement, and a preset algorithm;

[0294] Repair the failed data based on the repair sequence.

[0295] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0296] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0297] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0298] The electronic device 900 may also process images with one or more external devices 940 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also process images with one or more devices that enable users to interact with the electronic device 900, and / or process images with any device that enables the electronic device 900 to process images with one or more other computing devices (e.g., routers, modems, etc.). Such image processing may be performed through an input / output (I / O) interface 950. In addition, the electronic device 900 may also process images with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 960. As shown in the figure, the network adapter 960 processes images with other modules of the electronic device 900 through a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0299] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0300] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0301] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0302] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0303] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0304] In specific implementation, the program code for performing the operation of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0305] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0306] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0307] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the example implementation modes described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation mode of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation mode of the present disclosure.

[0308] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A data transmission method, characterized in that: include: Acquire data nodes, wherein the data nodes include failed nodes and repair nodes, and the repair nodes are used to repair failed data of the failed nodes; Determine multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each of the data repair requirements includes one failed node and multiple repair nodes; Determine a repair sequence for the plurality of data repair requirements based on a connection relationship between nodes in each of the data repair requirements, predicted traffic transmission data between nodes, a time point corresponding to each data repair requirement, and a preset algorithm; Repairing the failed data based on the repair sequence; The determining of the repair sequence of the plurality of data repair requirements based on the connection relationship between the nodes in each of the data repair requirements, the predicted traffic transmission data between the nodes, the time point corresponding to each data repair requirement and a preset algorithm comprises: Assigning weights to the edges between the nodes based on the connection relationship between the nodes to obtain a first weighted value for each edge in each data repair requirement; Weighting the edges between the nodes based on the predicted traffic transmission data between the nodes to obtain a second weighted value for each edge; Determine a final weighted value of each edge in each data repair requirement based on the first weighted value and the second weighted value of each edge; The data repair sequence of each edge is determined based on the final weight value of each edge, the time point corresponding to each edge in each data repair requirement, and a preset algorithm.

2. The data transmission method according to claim 1, characterized in that: The method further comprises: Obtain the upstream and downstream traffic data of each data node within a preset historical time period; The predicted traffic transmission data of each data node within a preset time period is determined based on the uplink and downlink traffic data and the prediction model.

3. The data transmission method according to claim 2, characterized in that: The determining the predicted traffic transmission data of each data node within a preset time period based on the uplink and downlink traffic data and the prediction model includes: Normalizing the uplink and downlink traffic data to obtain a feature vector corresponding to the data node; The feature vector is input into the prediction model to obtain the predicted traffic transmission data of each data node within a preset time period.

4. The data transmission method according to claim 2, characterized in that: The method further comprises: Obtaining uplink and downlink traffic data in the first historical time period and the second historical time period; The intelligent model is trained based on the uplink and downlink traffic data in the first historical time period and the second historical time period to obtain a trained prediction model.

5. The data transmission method according to claim 2, characterized in that: The method further comprises: Determining whether the available bandwidth of the repair node meets a preset threshold based on the predicted traffic transmission data of each data node within a preset time period; The determining of multiple data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node includes: Multiple data repair requirements are determined based on a predetermined virtual node dependency tree, failed nodes, and repair nodes whose available bandwidth meets a preset threshold.

6. The data transmission method according to claim 1, characterized in that: The method further comprises: Construct the virtual node dependency tree, wherein the virtual node dependency tree is a full binary tree, and the depth data of the full binary tree is the same as the number of failed nodes.

7. A data transmission device, characterized in that: include: An acquisition module, used for acquiring data nodes, wherein the data nodes include failed nodes and repair nodes, and the repair nodes are used for repairing failed data of the failed nodes; A first determination module is used to determine a plurality of data repair requirements based on a predetermined virtual node dependency tree, a failed node, and a repair node; each of the data repair requirements includes one failed node and a plurality of repair nodes; A second determination module is used to determine a repair sequence of the plurality of data repair requirements based on a connection relationship between nodes in each of the data repair requirements, predicted traffic transmission data between nodes, a time point corresponding to each data repair requirement, and a preset algorithm; A repair module, used for repairing the failed data based on the repair sequence; The second determining module includes: A first weighting unit, configured to weight the edges between the nodes based on the connection relationship between the nodes, to obtain a first weighted value for each edge in each data repair requirement; A second weighting unit, configured to weight the edges between the nodes based on the predicted traffic transmission data between the nodes, to obtain a second weighted value for each edge; A second determining unit, configured to determine a final weighted value of each edge in each data repair requirement based on the first weighted value and the second weighted value of each edge; The third determining unit is used to determine the data repair sequence of each edge based on the final weighted value of each edge, the time point corresponding to each edge in each data repair requirement, and a preset algorithm.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the data transmission method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data transmission method according to any one of claims 1 to 6 is implemented.