Self-adaptive parameter synchronous scheduling method for cross-domain distributed machine learning in optical wide area network

Abstract

The invention relates to the technical field of wide area networks, and discloses a cross-domain distributed machine learning-oriented adaptive parameter synchronous scheduling method in an optical wide area network, which comprises the following steps of: firstly, calculating two-hop bandwidths of each node, then sorting according to the sizes of the two-hop bandwidths, respectively selecting the first m nodes as aggregation nodes, respectively establishing mathematical models, the method comprises the following steps of: firstly, respectively calculating corresponding GMSTs according to relaxation and deterministic rounding thoughts, then comparing the GMSTs under different m conditions, finally selecting m nodes with the minimum GMST as aggregation nodes, and obtaining corresponding path, rate and link wavelength allocation strategies. According to the method, reconfigurability of network topology of the wide area network and rapidness and simplicity of parameter aggregation operation are considered at the same time, the number and positions of aggregation nodes can be adaptively selected according to network information and training task information, and wavelength allocation of optical fibers in the network is decided at the same time.

Description

Technical Field

[0001] The present application relates to the field of wide area network technology, and in particular to an adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network. Background Art

[0002] In recent years, machine learning (ML) has been widely used in machine translation, computer vision, speech recognition and other fields. With the increasing size of training data and machine learning models, distributed machine learning (DML) has become an ideal solution for large-scale applications. Nowadays, Internet service providers have established many data centers (DCs) around the world to serve their users. Correspondingly, a large amount of data generated by users is also stored in these DCs and used to enhance user experience through machine learning algorithms. Therefore, cross-domain distributed machine learning (Geo-distributed machine learning, Geo-DML) based on globally distributed data came into being.

[0003] like Figure 1 As shown in Figure 1, Geo-DML training usually adopts a two-layer training architecture, including local model synchronization (LMS) within the data center and global model synchronization (GMS) between data centers. In the LMS stage, a node called local aggregation (LA) is responsible for synchronizing the model parameters of all computing nodes in the data center. Then, in the GMS stage, the parameters of all LA nodes need to be synchronized across the wide area network (WAN) to form new model parameters, which are then distributed to the LA nodes of each data center. At this point, each data center can start a new round of iteration. With the use of acceleration hardware such as GPU, communication has become the main performance bottleneck of DML. Compared with the bandwidth of the local area network, the scarce and heterogeneous wide area network bandwidth makes the GMS stage across the wide area network more severe and prolongs the training time. Therefore, in order to improve the training performance of Geo-DML, scholars have done a lot of research in system optimization, resource scheduling and allocation, etc.

[0004] At present, some scholars have found that the network topology has a great impact on the training performance of DML, and even proposed some new physical topologies customized for DML. It is worth noting that most of these customized physical topologies are based on reconfigurable optical devices, such as optical switches. This provides a new idea for accelerating Geo-DML training. In fact, modern wide area networks are built on the optical layer, and cross-wide area network communications are achieved through reconfigurable optical add-drop multiplexers (ROADM). By adjusting the wavelength of each optical fiber, the wide area network topology can be reconfigured. At present, most solutions to accelerate Geo-DML ignore the reconfigurability of wide area network topology. In recent years, some scholars have proposed scheduling algorithms that use reconfigurable optical wide area networks to improve the performance of upper-layer applications. The experimental results of these algorithms show that joint optimization of scheduling and network topology can greatly reduce data transmission time, but they mainly target traditional traffic modes such as batch data transmission and multicast data transmission, and do not take into account the training characteristics of Geo-DML. If directly applied to Geo-DML, the training performance improvement is limited.

[0005] In fact, parameter aggregation in the parameter synchronization process only requires simple mathematical operations such as addition and averaging, which does not require high computing resources, so any node can be selected as an aggregation node. In addition, some studies have shown that the deployment of aggregation nodes, such as the location of parameter servers in the parameter server architecture, is crucial to training performance. If the appropriate node is selected to perform the aggregation operation, the total amount of data transmitted in the network can be greatly reduced, alleviating the data transmission pressure of the wide area network. Therefore, how to simultaneously utilize the reconfigurability of the wide area network topology and the training characteristics to optimize the parameter synchronization stage to greatly improve the training performance of Geo-DML is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to solve the above problems, the present application provides an adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network, so as to optimize Geo-DML by combining the reconfigurable characteristics and training characteristics of the optical wide area network, thereby realizing adaptive parameter synchronization scheduling in the optical wide area network.

[0007] In order to achieve the above purpose, the technical solution adopted in this application is as follows: The present application provides an adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network, the method comprising: Step S1: Input parameters: Get node set V , edge set E , link set L , wavelength bandwidth ∆, wavelength capacity of the edge , the wavelength capacity of the node , the set of links on the path ; Step S2: Initialize parameters: Initialize a temporary variable To set the value, , , , , , , ;in, Indicates assignment, Indicates the number of aggregation nodes, Indicates the selected FA node, Indicates the selected SA node. Indicates rate allocation, Indicates the selected path. Indicates the wavelength allocation of the link, Indicates the number of FA nodes; Step S3: Calculate the two-hop bandwidth of each LA node and sort them from large to small according to the two-hop bandwidth; Step S4: When When , go to step S5, otherwise go to step S16; wherein, Represents the number of LA nodes; Step S5: Set , select before LA nodes serve as FA nodes; Step S6: Solve the problem (P5) using a linear programming solver and obtain ; Wherein, the problem (P5) is expressed as: ; In the formula, Indicates the completion time of parameter aggregation in the first sub-stage, represents the constraints, Indicates the sequence number of the path. Indicates the set of LA nodes other than FA nodes. represents the set of FA nodes, express The node number in express The node number in e represents the edge, Represents edge e The wavelength capacity of Indicates a link, Indicates the sequence number of all nodes, Representation Node The wavelength capacity of Representation Node arrive No. The set of links on the path, S Indicates the machine model size, Indicates link The two nodes of Represents an indicator function. When * holds, the value is 1, otherwise it is 0. Represents a temporary variable. , is a temporary intermediate variable. , is a binary variable, Representation Node Select Node As its FA node, is a binary variable, Representation Node To Node Select the data transmission k Paths; The first sub-stage is a sub-stage in the GMS stage, which is used to optimize the number of decision-making FA nodes, the FA nodes selected by each LA node, the path and rate between FA and LA nodes, and the wavelength allocation of the link to reduce the parameter aggregation time of the first sub-stage ; Step S7: For each LA node ,Pick ,make , , ;in, Represents a temporary variable. represents a binary variable, Representation Node Select Node As a FA node, represents a binary variable, Representation Node To Node Select the data transmission Path, Represents the node obtained and path ; Step S8: and Substitute into problem (P7) and use the linear programming solver to obtain the scheduling strategy for the first sub-stage: ;in, Indicates link The number of wavelengths assigned, Representation Node and The data transfer rate between express The reciprocal of ; The problem (P7) is expressed as: ; In the formula, is a binary variable, =1 indicates a node To Node Select the data transmission Paths; Step S9: Maximum two-hop bandwidth node; where, Indicates SA node; Step S10: Solve the problem (P12) using a linear programming solver and obtain ; The problem (P12) is expressed as: ; In the formula, It represents the inverse of the completion time of parameter aggregation in the second sub-stage, that is, ; represents the parameter aggregation time of the second sub-stage; It is an intermediate temporary variable , is a binary variable, =1 indicates a node i To Node Select the data transmission Paths; The second sub-stage is a sub-stage in the GMS stage. The second sub-stage and the first sub-stage constitute the GMS stage. The second sub-stage is executed after the first sub-stage is completed. The second sub-stage is used to optimize the selection of SA nodes, the path and rate between SA nodes and FA nodes, and the wavelength allocation of links to reduce the parameter aggregation time of the second sub-stage. ; Step S11: For each FA node ,Pick ,make ;in, Indicates the maximum of The value of Representation Node i To Node Select the data transmission Paths; Step S12: The Substitute into problem (P9) and use the linear programming solver to obtain the scheduling strategy for the second sub-stage: ;in, Representation Node and The data transmission rate between them; the problem (P9) is expressed as: ; In the formula, Representation Node arrive No. The set of links on the path; Step S13: Calculate the model synchronization time: ; Where t represents the model synchronization time; Step S14: If , go to step S15, otherwise go to step S4; Step S15: , , , , , , , , go to step S4; wherein, Represents the set of selected paths, Represents the selected FA node set, represents the FA node selected by each node, Indicates the data transmission rate between nodes, Indicates the wavelength allocation of the link; Step S16: Output the number of aggregated nodes , Selected Path , Selected FA node , each node selects the aggregation node , wavelength allocation of links , Rate Allocation , Selected SA node .

[0008] Furthermore, it is characterized in that the two-hop bandwidth of each LA node is calculated by the following formula; ; In the formula, Indicates LA node i The two-hop bandwidth of , Represents LA nodes respectively Bandwidth and LA nodes The bandwidth of the neighbor nodes, where Indicates LA node The set of neighbor nodes of Represents the node's own bandwidth and neighbor's bandwidth The impact ratio, Indicates LA node The wavelength capacity of Indicates LA node The neighbor nodes of Indicates LA node The wavelength capacity of the neighboring nodes.

[0009] Further, the problem (P5) is determined by the following method: The mathematical optimization model for the first sub-stage is constructed as follows: ; In the formula, is a binary variable, =1 indicates a node Select Node as its FA node; When the FA nodes are given, the problem (P1) is simplified to: ; Substituting equation (2a) into equation (2d), we get: ; The two variables in the above formula are and The product of makes the constraint (2f) nonlinear, introducing the variable , according to formula (2b), (2c), (2f), we can deduce: ; The problem (P2) becomes: ; Introducing new variables Substituting into (1e), (1f) and (2h), we obtain: ; Binary variables Relaxing to continuous variables transforms problem (P4) into problem (P5).

[0010] Further, the problem (P7) is determined by the following method: In getting Then, the binary variable is obtained by the deterministic rounding algorithm and , and then and Substituting into problem (P2), we get: ; Introducing the first variable , then according to question (P6) we get the question (P7).

[0011] Further, the problem (P9) is determined by the following method: In the second sub-phase, one FA node is selected as the SA node, which is responsible for aggregating the parameters of other FA nodes and determining the wavelength allocation, the path and rate between SA and FA. If the FA node is selected based on the two-hop bandwidth As an SA node, the mathematical optimization model of the second sub-stage is expressed as: ; Among them, constraint (8a) indicates that the second stage ends when the parameter transmission between all FA nodes and SA nodes is completed, constraint (8b) ensures that only one path is selected between each FA and SA node to transmit parameters, and constraint (8c) is the link capacity constraint, which ensures that the traffic on the link is not greater than the link bandwidth capacity; Introducing temporary variables Substituting constraint (8a), we obtain the problem (P9).

[0012] Further, the problem (P12) is determined by the following method: Substituting constraint (9a) into constraint (8c), we obtain: ; Then, problem (P9) becomes: ; Binary variables Relaxing to a continuous variable, we get: ; Introducing variables , combining (8b) and (9b), we obtain: ; According to constraint (11b), constraint (11c) and problem (P11), the problem (P12) is obtained.

[0013] This application has at least the following beneficial effects: This application designs an adaptive parameter synchronization scheduling method for Geo-DML to accelerate the parameter synchronization stage, reduce GMST (model synchronization time), and thus speed up the training process. This method takes into account the reconfigurability of the wide area network topology and the speed and simplicity of parameter aggregation operations. It can adaptively select the number and location of aggregation nodes based on network information and training task information, and decide on the wavelength allocation of optical fibers in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the Geo-DML two-layer training architecture in the prior art.

[0015] Figure 2 A flowchart of an adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0017] The specific implementation methods of the present application are further described in detail below in conjunction with the embodiments.

[0018] The explanations of the abbreviations involved in this application are as follows: Geo-DML stands for cross-domain distributed machine learning; GMST stands for the time required to execute the GMS phase on the wide area network as the global model synchronization time; GMS stands for global model synchronization between data centers; LA node stands for local aggregation node; FA node stands for aggregation node in the first sub-phase; SA node stands for aggregation node in the second sub-phase; V Represents a set of LA nodes; E stands for Edge collection; S Indicates the size of the machine learning model; L represents a set of directed links; B e side e The wavelength capacity of the edge e The maximum number of wavelengths that can be carried; Q υ Representation Node υ The wavelength capacity of the node υ The maximum number of wavelengths that can be carried; node arrive No. The set of links on the path; V’’ Represents a set of FA nodes; V’ express V / V'' , that is, the set of LA nodes other than FA nodes; Indicates link The number of wavelengths assigned; represents a binary variable, Representation Node Selected as FA node; Indicates the selected FA node; Indicates the selected SA node; Indicates rate allocation; Indicates the selected path; Indicates the wavelength allocation of the link; Indicates the number of FA nodes; Indicates the number of LA nodes; express V’ The node number in ; express V’‘ The node number in ; Indicates the sequence number of the path; represents the parameter aggregation time of the first sub-stage; Parameter aggregation time of the second sub-stage; Represents a temporary variable. ; Represents an indicator function, which takes a value of 1 when the condition is met, otherwise 0; Indicates link Two nodes of represents a temporary variable, and The same meaning ; Representation and Same, it is a binary variable, =1 means node i selects node as its FA node; represents a temporary variable, and Meaning the same, binary variables, =1 indicates a node i To Node Select the data transmission k Paths; Indicates the node number, indicating the nodes; Indicates the sequence number of the path, indicating the k Paths; represents the first variable, ; represents the second variable, ; Indicates SA node; Intermediate temporary variables, ; represents a binary variable, =1 indicates a node To Node Select the data transmission Paths; represents a temporary variable, and Meaning the same, binary variables, Representation Node To Node Select the data transmission k Paths; Representation Node arrive Data transfer rate; Representation Node arrive No. k The set of links on the path; LA Node Two-hop bandwidth; Indicates LA node bandwidth; Indicates LA node The bandwidth of neighbor nodes; Indicates LA node The set of neighbor nodes of Represents the bandwidth of the node itself and the bandwidth of its neighbors The impact ratio; Indicates LA node wavelength capacity; Indicates LA node Neighbor nodes of Indicates LA node The wavelength capacity of the neighboring nodes; Represents a temporary variable, indicating the current minimum GMST.

[0019] An embodiment of the present application provides an adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network.

[0020] Specifically, the present invention defines the time required for Geo-DML to execute the GMS phase on the wide area network as the model synchronization time (Global Model Synchronization Time, GMST). The technical problem solved by the present invention is how to design an adaptive parameter synchronization scheduling method (named AdaptiveScheduling) for Geo-DML to accelerate the parameter synchronization phase, reduce GMST, and thus speed up the training process. This method takes into account the reconfigurability of the wide area network topology and the speed and simplicity of parameter aggregation operations. It can adaptively select the number and location of aggregation nodes based on network information and training task information, and at the same time decide on the wavelength allocation of optical fibers in the network.

[0021] When there are multiple aggregation nodes responsible for parameter synchronization of LA nodes, the GMS phase is divided into two sub-phases. In the first sub-phase, a part of LA nodes are selected as the first sub-phase aggregation nodes (called First Aggregation, FA) to synchronize the parameters of other LA nodes; in the second sub-phase, a FA node is selected as the second sub-phase aggregation node (called Second Aggregation, SA) to synchronize the parameters of other FA nodes. In an embodiment of the present invention, parameter aggregation and parameter distribution use the same path and rate, so they take the same time. Therefore, in an embodiment of the present invention, it is only necessary to calculate the parameter aggregation time to obtain the corresponding GMST. Therefore, it is only necessary to calculate and optimize the parameter aggregation time.

[0022] In this embodiment, we mainly focus on the GMS stage where parameter synchronization needs to be performed across the WAN. For simplicity, we do not consider encrypting or compressing the parameters here, and assume that the parameter size of all LA nodes is equal to the model size. S Considering the differences in computing power, training data sets, communication architectures, etc. of different data centers, the time required for each LA node to complete local parameter aggregation may be different. However, this stragglers problem can be alleviated by performing multiple local iterations within the data center. Therefore, here we take the example of all LA nodes starting the GMS phase almost at the same time to describe the detailed process.

[0023] In this embodiment, the network is abstracted into an undirected graph as in the conventional reconfigurable optical wide area network method. G =( V , E ),in V and E They represent LA nodes (or data centers or ROADMs) and edges (or optical fibers between data centers) in the network. The number of wavelengths that can be accommodated depends on the number of transponders deployed on the ROADM. , while e There are also wavelength capacity limitations Since different edges can transmit data in both directions, this embodiment provides e Introduce two virtual directed links whose total wavelength allocation does not exceed the capacity of the edge .

[0024] The present invention needs to find a suitable aggregation node allocation scheme to optimize the parameter synchronization time. When there are multiple aggregation nodes, the GMS stage is divided into a first sub-stage and a second sub-stage. When the first sub-stage is completed, the second sub-stage will start. Therefore, a mathematical optimization model is established for each sub-stage. In the first sub-stage, it is necessary to decide the number of FA nodes, the FA nodes selected by each LA node, the path and rate between the FA and LA nodes, the wavelength allocation of the link, etc., in order to reduce the parameter aggregation time of the first sub-stage. In the second stage, it is necessary to optimize the selection of SA nodes, the path and rate between SA nodes and FA nodes, the wavelength allocation of links, etc., in order to reduce the parameter aggregation time of the second sub-stage. .

[0025] The mathematical optimization model of the first sub-stage is as follows: ; As shown in (P1), the goal of this embodiment is to optimize the polymerization time of the first sub-stage Constraint (1a) indicates that the first sub-phase is completed when all FA nodes complete parameter transmission. Constraint (1b) indicates that m LA nodes are selected as FA nodes. Constraints (1c) and (1d) respectively indicate that each LA node should select one and only one FA node to synchronize its parameters, and only one path is selected between them to transmit data. (1e) and (1f) represent the edge capacity constraint and the node capacity constraint, respectively, that is, the total number of wavelengths allocated to edge and node-related links cannot exceed the edge capacity and node capacity constraints. It should be noted that in the present invention, the node capacity is limited by the number of transponders on each ROADM, so the present invention assumes that the maximum wavelength capacity on the node is a constant. (1g) ensures that the total traffic passing through a link does not exceed the bandwidth capacity of the link, where the indicator function Representation Node and Whether the flow between them passes through the link .

[0026] It can be seen from problem (P1) that there are multiple variables multiplied in the constraints, such as (1a) and (1g), which are difficult to solve directly using a linear programming (LP) solver. Therefore, this embodiment needs to simplify the original problem. Since the FA node is responsible for aggregating the parameters of other LA nodes, if a node with a smaller bandwidth is selected as the FA node, the limited bandwidth will become a bottleneck for data transmission. In the present invention, the two-hop bandwidth of the node is defined (which will be described in detail later). According to the two-hop bandwidth, a suitable LA node is selected as the FA node, and the set V'' Indicates that other LA nodes are represented by sets V'It can be seen from problem (P1) that when the FA node is given (that is, it is known and ), the original problem (P1) can be simplified to: ; Substituting (2a) into (2d), we obtain: ; The two variables in the above formula are and The product of makes the constraint (2f) nonlinear, so the variable , according to (2b), (2c), (2f), we can deduce: ; Therefore, the problem (P2) becomes: ; Next, introduce a new variable Substituting into (1e), (1f) and (2h), we get: ; Obviously, problem (P4) is a mixed integer linear program, which takes a long time to solve in a large-scale network. Therefore, this embodiment uses binary variables Relaxing to continuous variables, the problem (P4) becomes: ; At this point, problem (P5) can be solved by the linear programming solver to find the optimal value and continuous variables in a short time. When you get After that, the binary variable can be obtained by the deterministic rounding algorithm and , and then bring these variables into problem (P2), we get: ; (1e), (1f) Since (6a) is nonlinear, this embodiment introduces a new variable , then the problem (P6) becomes: ; (1e), (1f), (6b) Now we can directly use the linear programming solver to solve the problem (P7) and obtain the scheduling plan for the first sub-stage: , , , and .

[0027] In the second sub-phase, one FA node needs to be selected as the SA node, which is responsible for aggregating the parameters of other FA nodes and determining wavelength allocation, the path and rate between SA and FA. Similarly, assuming that the FA node is selected based on the two-hop bandwidth As an SA node, the mathematical optimization model of the second sub-stage can be expressed as: ; (1e), (1f) Constraint (8a) indicates that the second phase ends when the parameter transmission between all FA nodes and SA nodes is completed. (8b) ensures that only one path is selected between each FA and SA node to transmit parameters. (8c) is the link capacity constraint, which ensures that the traffic on the link is not greater than the link bandwidth capacity.

[0028] Since (8a) is nonlinear, we introduce the variable Substituting into (8a), we can obtain: ; (1e), (1f), (8b)-(8d) Next, substituting (9a) into (8c), we obtain: ; Now, the question (P9) becomes: ; (1e), (1f), (8b), (8d), (9b) As in problem (P5), here we also use the binary variable Relaxing to a continuous variable, we get: ; (1e), (1f), (8b), (9b) Next, introduce a new variable , combined with (8b) and (9b), we can get: ; Then the problem (P11) becomes: ; (1e), (1f), (11b), (11c) Now we can directly use the LP solver to solve (P12), and get Next, we use the deterministic rounding method to obtain the binary variable Next, Substitute into problem (P9) and solve it. So far, the solution to the second sub-stage is obtained: , , , and .

[0029] Aggregation node (LA and SA node) selection strategy - two-hop bandwidth: Obviously, FA and SA nodes, as parameter aggregation nodes, need to receive parameters from other nodes. If their bandwidth is small, it will lead to longer data transmission time. In addition, for the aggregation node, the data of non-adjacent LA nodes will pass through its adjacent nodes. If the sum of the bandwidths of its adjacent nodes is small, it will also become a transmission bottleneck. Similarly, the bandwidth of nodes two or more hops away from the aggregation node is also important, but the number of these nodes is large, and their impact on GMST is not as significant as the aggregation node itself and its neighbors. Therefore, this embodiment selects the aggregation node based on the two-hop bandwidth of the aggregation node. Define node Two-hop bandwidth ,in , Represents nodes Bandwidth and nodes The bandwidth of the neighbors, where represents the set of neighbor nodes of node i, Represents the node's own bandwidth and neighbor's bandwidth The impact ratio is in the range of 0 to 1. Generally speaking, the bandwidth of the node itself is more important, so the present invention will In the present invention, in order to select a suitable aggregation node, the two-hop bandwidth of each node is first calculated, and then the first m nodes ( , | represents the number of LA nodes) nodes as aggregation nodes, calculate the corresponding GMST respectively, and then compare the different m The GMST of the case, finally choose the one with the minimum GMST m nodes as aggregation nodes.

[0030] See also Figure 2 As shown, it is a flowchart of an adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network provided by an embodiment of the present application. The adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network includes the following steps when implemented: Step S1: Input parameters: node set V, edge set E, link set L, wavelength bandwidth ∆, wavelength capacity of edge , the wavelength capacity of the node , the set of links on the path .

[0031] Step S2: Initialization parameters: Initialization is a very large number, let , , , , , , ; Step S3: Calculate the two-hop bandwidth of each LA node and sort them from large to small according to the two-hop bandwidth.

[0032] Step S4: When When , go to step S5, otherwise go to step S16.

[0033] Step S5: , select before LA nodes serve as FA nodes.

[0034] Step S6: Use the linear programming solver Cplex to solve the problem (P5) and obtain .

[0035] Step S7: For each LA node i ,Pick ,make , , .

[0036] Step S8: and Substitute into problem (P7) and use Cplex to solve the scheduling strategy for the first sub-stage: .

[0037] Step S9: Maximum two-hop bandwidth node.

[0038] Step S10: Solve the problem (P12) using Cplex and obtain .

[0039] Step S11: For each FA node ,Pick ,make .

[0040] Step S12: The Substitute the problem (P9) and use Cplex to solve the second sub-stage scheduling strategy: .

[0041] Step S13: Calculate GMST: .

[0042] Step S14: If , go to step S15, otherwise go to step S4.

[0043] Step S15: , , , , , , , , go to step S4.

[0044] Step S16: Return parameter synchronization scheduling strategy: , , , , , , . That is, output: the number of aggregated nodes , the selected path ,Selected FA node , each node selects the aggregation node , wavelength allocation of links , Rate Allocation , selected SA node .

[0045] The above implementation modes are only used to illustrate the present application, and are not intended to limit the present application. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions also belong to the scope of the present application, and the scope of patent protection of the present application shall be limited by the claims.

Claims

1. An adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network, characterized in that: The method comprises: Step S1: Input parameters: Get node set V , edge set E , link set L , wavelength bandwidth ∆, wavelength capacity of the edge , the wavelength capacity of the node , the set of links on the path ; Step S2: Initialize parameters: Initialize a temporary variable To set the value, , , , , , , ;in, Indicates assignment, Indicates the number of aggregation nodes, Indicates the selected FA node, Indicates the selected SA node. Indicates rate allocation, Indicates the selected path. Indicates the wavelength allocation of the link, Indicates the number of FA nodes; Step S3: Calculate the two-hop bandwidth of each LA node and sort them from large to small according to the two-hop bandwidth; Step S4: When When , go to step S5, otherwise go to step S16; wherein, Represents the number of LA nodes; Step S5: Set , select before LA nodes serve as FA nodes; Step S6: Solve the problem (P5) using a linear programming solver and obtain ; Wherein, the problem (P5) is expressed as: ； In the formula, Indicates the completion time of parameter aggregation in the first sub-stage, represents the constraints, Indicates the sequence number of the path. Indicates the set of LA nodes other than FA nodes. represents the set of FA nodes, express The node number in express The node number in e represents the edge, Represents edge e The wavelength capacity of Indicates a link, Indicates the sequence number of all nodes, Representation Node The wavelength capacity of Representation Node arrive No. The set of links on the path, S Indicates the machine model size, Indicates link The two nodes of Represents an indicator function. When * holds, the value is 1, otherwise it is 0. Represents a temporary variable. , is a temporary intermediate variable. , is a binary variable, Representation Node Select Node As its FA node, is a binary variable, Representation Node To Node Select the data transmission k Paths; The first sub-stage is a sub-stage in the GMS stage, which is used to optimize the number of decision-making FA nodes, the FA nodes selected by each LA node, the path and rate between FA and LA nodes, and the wavelength allocation of the link to reduce the parameter aggregation time of the first sub-stage ; Step S7: For each LA node ,Pick ,make , , ;in, Represents a temporary variable. represents a binary variable, Representation Node Select Node As a FA node, represents a binary variable, Representation Node To Node Select the data transmission Path, Represents the node obtained and path ; Step S8: and Substitute into problem (P7) and use the linear programming solver to obtain the scheduling strategy for the first sub-stage: ;in, Indicates link The number of wavelengths assigned, Representation Node and The data transfer rate between express The reciprocal of ; The problem (P7) is expressed as: ； In the formula, is a binary variable, =1 indicates a node To Node Select the data transmission Paths; Step S9: Maximum two-hop bandwidth node; where, Indicates SA node; Step S10: Solve the problem (P12) using a linear programming solver and obtain ; The problem (P12) is expressed as: ； In the formula, It represents the inverse of the completion time of parameter aggregation in the second sub-stage, that is, ; represents the parameter aggregation time of the second sub-stage; It is an intermediate temporary variable , is a binary variable, =1 indicates a node To Node Select the data transmission Paths; The second sub-stage is a sub-stage in the GMS stage. The second sub-stage and the first sub-stage constitute the GMS stage. The second sub-stage is executed after the first sub-stage is completed. The second sub-stage is used to optimize the selection of SA nodes, the path and rate between SA nodes and FA nodes, and the wavelength allocation of links to reduce the parameter aggregation time of the second sub-stage. ; Step S11: For each FA node ,Pick ,make ;in, Indicates the maximum of The value of Representation Node i To Node Select the data transmission Paths; Step S12: The Substitute into problem (P9) and use the linear programming solver to obtain the scheduling strategy for the second sub-stage: ;in, Representation Node and The data transmission rate between them; the problem (P9) is expressed as: ； In the formula, Representation Node arrive No. The set of links on the path; Step S13: Calculate model synchronization time: ; Where t represents the model synchronization time; Step S14: If , go to step S15, otherwise go to step S4; Step S15: , , , , , , , , go to step S4; wherein, Represents the set of selected paths, Represents the selected FA node set, represents the FA node selected by each node, Indicates the data transmission rate between nodes, Indicates the wavelength allocation of the link; Step S16: Output the number of aggregated nodes , Selected Path , Selected FA node , each node selects the aggregation node , Link wavelength allocation , Rate Allocation , Selected SA node .

2. The adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network according to claim 1, characterized in that: The two-hop bandwidth of each LA node is calculated by the following formula: ； In the formula, Indicates LA node i The two-hop bandwidth of , Represents LA nodes respectively Bandwidth and LA nodes The bandwidth of the neighbor nodes, where Indicates LA node The set of neighbor nodes of Represents the node's own bandwidth and neighbor's bandwidth The impact ratio, Indicates LA node The wavelength capacity of Indicates LA node The neighbor nodes of Indicates LA node The wavelength capacity of the neighboring nodes.

3. The adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network according to claim 1, characterized in that: The problem (P5) is determined by the following method: The mathematical optimization model for the first sub-stage is constructed as follows: ； In the formula, is a binary variable, =1 indicates a node Select Node as its FA node; When the FA nodes are given, the problem (P1) is simplified to: ； Substituting equation (2a) into equation (2d), we get: ； The two variables in the above formula are and The product of makes the constraint (2f) nonlinear, introducing the variable , according to formula (2b), (2c), (2f), we can deduce: ； The problem (P2) becomes: ； Introducing new variables Substituting into (1e), (1f) and (2h), we obtain: ； Binary variables Relaxing to continuous variables transforms problem (P4) into problem (P5).

4. The adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network as claimed in claim 3, characterized in that: The problem (P7) was determined by the following method: In getting Then, the binary variable is obtained by the deterministic rounding algorithm and , and then and Substituting into problem (P2), we get: ； Introducing the first variable , then according to question (P6) we get the question (P7).

5. The adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network as claimed in claim 4, characterized in that: The problem (P9) was determined by the following method: In the second sub-phase, one FA node is selected as the SA node, which is responsible for aggregating the parameters of other FA nodes and determining the wavelength allocation, the path and rate between SA and FA. If the FA node is selected based on the two-hop bandwidth As an SA node, the mathematical optimization model of the second sub-stage is expressed as: ； Among them, constraint (8a) indicates that the second stage ends when the parameter transmission between all FA nodes and SA nodes is completed, constraint (8b) ensures that only one path is selected between each FA and SA node to transmit parameters, and constraint (8c) is the link capacity constraint, which ensures that the traffic on the link is not greater than the link bandwidth capacity; Introducing temporary variables Substituting constraint (8a), we obtain the problem (P9).

6. The adaptive parameter synchronization scheduling method for cross-domain distributed machine learning in an optical wide area network as claimed in claim 5, characterized in that: The problem (P12) was determined by the following method: Substituting constraint (9a) into constraint (8c), we obtain: ； Then, problem (P9) becomes: ； Binary variables Relaxing to a continuous variable, we get: ； Introducing variables , combining (8b) and (9b), we obtain: ； According to constraint (11b), constraint (11c) and problem (P11), the problem (P12) is obtained.

Images