Method and device for updating parameters of distributed learning model across data centers in wide area networks

Abstract

The present invention relates to a method and device for updating parameters of a distributed learning model across data centers in a wide area network. The method mainly includes: splitting the model and training data and deploying them on N worker machines; setting the same parameter importance evaluator on each parameter server, and the parameter importance evaluator includes three parameters: model convergence fixed parameter T f , Network bandwidth threshold T b And the dynamic threshold value T of parameter importance d Distributed machine learning model training begins. This invention addresses the problem of distributed machine learning training rate by introducing a parameter importance mechanism. It also addresses the problem of distributed machine learning requiring excessive network bandwidth. By using the parameter importance mechanism to eliminate updates to unimportant parameters, the bandwidth requirement for distributed machine learning across a wide area network (WAN) can be reduced, thereby reducing the computational effort of the parameter server.

Description

Technical field

[0001] The present invention relates to the field of distributed machine learning technology, and in particular to a method and device for updating parameters of a distributed learning model across data centers in a wide area network. The method mainly solves the problem that cross-domain (WAN) and cross-data center (DC) distributed machine learning has excessively high requirements for inter-domain communication network bandwidth and latency. [Background Technology]

[0002] Machine learning is now widely used, especially deep learning, which has achieved significant breakthroughs in pattern recognition, such as image and speech recognition. With the continuous iteration and updating of algorithms, the exploration of application scenarios has become increasingly important, and the engineering of machine learning applications has become a major research direction. Distributed machine learning, as one of the most important research branches, provides a suitable solution for machine learning model training in the context of big data. Distributed machine learning primarily targets the following application scenarios:

[0003] (1) When the original data sources are distributed across multiple different terminals, if the traditional model training solution is used to centralize all the original data in a unified data center for processing, a large amount of network bandwidth resources will be occupied during the data transmission process. This solution is time-consuming, consumes public resources, and is unnecessary. In this case, it is appropriate to use distributed machine learning for model training.

[0004] (2) When the owner of the original data source has data security requirements, transmitting the original data across the wide area network is no longer feasible. In this case, using distributed machine learning is the best solution, and this scenario is also the basis of federated learning.

[0005] Bottlenecks of distributed machine learning:

[0006] The biggest bottleneck of distributed machine learning is the transmission bandwidth between multiple servers. Since distributed machine learning has extremely high real-time requirements for communication between servers (us-level) and the amount of data updated during model training is huge (Tb / s-level), the network bandwidth requirements of distributed machine learning are generally between 100Mb / s and 10Gb / s.

[0007] Wide Area Network (WAN) Distributed Machine Learning Bottlenecks:

[0008] Distributed machine learning over a wide area network (WAN) is subject to bandwidth limitations similar to traditional distributed machine learning, as well as network latency limitations. Network latency in a WAN is primarily composed of physical latency, which is positively correlated with transmission distance. Compared to distributed machine learning over a local area network (LAN), bandwidth resources are more limited in a WAN. The average bandwidth in a LAN is 10x-100x the bandwidth in a WAN, and WAN bandwidth is closely related to geographic location. Currently, the typical service price for a 1Gb / s-10Gb / s intra-provincial dedicated line service in China is 50,000-200,000 yuan / month, while the cost of an inter-provincial dedicated line is typically 5x-10x the cost of an intra-provincial dedicated line. High network operating costs often limit the development of distributed machine learning over a WAN.

[0009] like Figure 1 As shown in the figure, it is a schematic diagram of the expansion of the local area network parameter server to the wide area network in the existing technology. The parameter server (PS) is one of the current mainstream high-performance distributed machine learning architectures. Its main idea is to split the model training data and the model itself and distribute them on multiple worker machines and PSs. By providing high computing power through multiple servers, the single model training process can be accelerated. Figure 1 As shown in the figure, after the PS model is extended to the cross-data center scenario (i.e., wide area network WAN), since each round of parameter update of the worker machine needs to be updated in real time in all PSs, the communication bandwidth and latency between WANs become the main factors limiting the performance of this architecture.

[0010] In summary, when deploying distributed machine learning in a wide area network, due to network bandwidth limitations, machine learning algorithms typically experience a 20X-50X performance degradation.

[0011] In view of this, how to overcome the defects of the existing technology and solve the above technical problems is a difficult problem to be solved in this technical field. [Summary of the invention]

[0012] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a method and device for updating parameters of a distributed learning model across data centers in a wide area network. The present invention mainly eliminates unimportant update parameters during model training by establishing a parameter update strategy. Since the parameters of each round of iteration need to be updated in each parameter server, i.e., multiple DCs, reducing the update parameters essentially reduces the communication volume between networks, thereby reducing the system's demand for network bandwidth.

[0013] The embodiment of the present invention adopts the following technical solutions:

[0014] In a first aspect, the present invention provides a method for updating parameters of a distributed learning model across data centers in a wide area network, comprising:

[0015] Split the model and training data and deploy them on N worker machines;

[0016] The same parameter importance estimator is set on each parameter server. The parameter importance estimator includes three parameters: model convergence fixed parameter T f , Network bandwidth threshold T b And the dynamic threshold value T of parameter importance d ;

[0017] Start distributed machine learning model training.

[0018] Furthermore, by setting T f Ensure model training convergence, specifically: set the minimum T f Value, when the workermachine update parameter Δx is greater than T f , allowing the parameter server to update the parameters.

[0019] Furthermore, the T f Manually set according to the model size and training data distribution. When the model is too large or the training data variance is too small, T f Set it too small. When the model is too small or the variance of the training data is too large, set T f The setting is too large.

[0020] Furthermore, by setting T b Ensure minimum communication loss between wide area networks. Specifically: Set the maximum T b When the data transmission rate between the worker machine and the parameter server is greater than T b When , the communication between the parameter server and other parameter servers in the wide area network is suspended.

[0021] Furthermore, the T b Manually set it according to the communication capacity between WANs in the environment. When the bandwidth between WANs is small, set a smaller T b When the bandwidth between WANs is large, set a larger T b .

[0022] Furthermore, by setting T d Filter unimportant parameters, specifically: set dynamic T d When the worker machine updates the parameter Δx, it is greater than T d , allowing the parameter server to update the parameters.

[0023] Furthermore, the T d The specific settings include:

[0024] Manually set the unit time t, and divide the total model training time T into n equal parts according to the unit time t, that is, the total training time T = {t1, t2, t3, ..., t n};

[0025] At time t1, the parameter server collects the updated parameters X1(i)={x 11 ,x 12 ,x 13 ,…,x 1i ,…,x 1n}, find the first moment of X1 Second moment Where P(x) is the value of x in X1 1i The probability density function of α is the restriction coefficient, α∈(0,1), and let the first-order moment average prediction value Second-order moment average prediction value

[0026] From time t2, at the end of the mth round of unit time window, the parameter server collects the updated parameters X of all worker machines in the data center. m (i) = {x m1 ,x m2 ,x m3 ,…,x mi ,…,x mn}, find X at this time m The first moment of and second-order moment renew Among them, β1 and β2 are weighted coefficients, β1,β2∈(0,1), then and Make corrections, Last Updated Where γ is the learning rate of the training model obtained from the training model, γ∈(10 -6 ,1).

[0027] Furthermore, when and If it is too small, take a smaller α. and When α is too large, take a larger α; when the model is too large or the amount of training data is too large, increase β1 and β2 to make it more inclined to the latest data update trend; when the model is too small or the amount of training data is too small, reduce β1 and β2 to make it more inclined to the historical data update trend; when the model is too large or the number of learning rounds is small, take a larger value of γ; when the model is small or after multiple learning rounds, take a smaller value of γ.

[0028] Furthermore, splitting and deploying the model and training data on N worker machines specifically includes:

[0029] Before model training begins, according to the PS architecture requirements, the model is split and deployed on N worker machines, and the training data is also divided into N parts and deployed on N worker machines.

[0030] On the other hand, the present invention provides a device for updating parameters of a distributed learning model in a wide area network across data centers, specifically comprising: at least one processor and a memory, at least one processor and the memory being connected via a data bus, the memory storing instructions that can be executed by at least one processor, and after the instructions are executed by the processor, being used to complete the method for updating parameters of a distributed learning model in a wide area network across data centers in the first aspect.

[0031] Compared with the existing technology, the beneficial effects of the present invention are: solving the problem of distributed machine learning training rate by introducing a parameter importance mechanism; solving the problem of distributed machine learning's excessive demand for network bandwidth by introducing a parameter importance mechanism; by using the parameter importance mechanism to eliminate updates of unimportant parameters, the demand for bandwidth of distributed machine learning between WANs can be reduced, the computing amount of the parameter server can be reduced, and the cost of obtaining computing resources within the data center is also much lower than that of inter-domain bandwidth resources.

Brief Description of the Drawings

[0032] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0033] Figure 1 A schematic diagram of a local area network parameter server provided as background technology of the present invention being extended to a wide area network;

[0034] Figure 2 A flow chart of a method for updating parameters of a distributed learning model across data centers in a wide area network provided in Example 1 of the present invention;

[0035] Figure 3 This is a flow chart of the internal settings of the parameter importance evaluator provided in Example 1 of the present invention;

[0036] Figure 4 A flow chart showing a specific determination of the Tf value provided in Example 1 of the present invention;

[0037] Figure 5 A flow chart showing a specific determination of the Td value provided in Example 1 of the present invention;

[0038] Figure 6 A flowchart for determining the Tb value provided in Example 1 of the present invention;

[0039] Figure 7 Schematic diagram of the cloud-network integration scenario provided in Example 2 of the present invention;

[0040] Figure 8 This is a diagram showing the architecture of a system for updating parameters of a distributed learning model across data centers over a wide area network, as provided in Example 3 of the present invention;

[0041] Figure 9 A schematic structural diagram of a wide area network cross-data center distributed learning model parameter updating device provided in Example 4 of the present invention. [Specific implementation method]

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] The present invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0044] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] Before describing the embodiments, it should be noted that the present invention is applicable to both BSP and SSP models. BSP (Bulk Synchronous Parallel) maintains a global update for all other worker machines after each worker machine updates its parameters. This means that all worker machines must observe any parameter update before a new iteration. SSP (Stale Synchronous Parallel) allows the fastest worker machine to lead the slowest worker machine by a certain number of iterations, maintaining multiple models with varying degrees of update within the system.

[0046] The distributed machine learning model used in this invention is parameter server (PS), which is the mainstream distributed machine learning architecture in the industry. Figure 1 As shown in the figure, the main idea is that each parameter server stores a part of the model parameters, and the worker machine is responsible for reading and updating the model parameters in the parameter server, and then the parameter server is responsible for synchronizing the data update.

[0047] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0048] Example 1:

[0049] like Figure 2 As shown, an embodiment of the present invention provides a method for updating parameters of a distributed learning model in a wide area network across data centers, and the specific steps are as follows.

[0050] Step 100: Split the model and training data and deploy them on N worker machines.

[0051] Step 200: Set the same parameter importance estimator on each parameter server. The parameter importance estimator includes three parameters: model convergence fixed parameter T f , Network bandwidth threshold T b And the dynamic threshold value T of parameter importance d .

[0052] Step 300: Start distributed machine learning model training.

[0053] Through the above steps, the present invention sets a parameter importance evaluator on each parameter server, establishes a parameter update strategy, and eliminates unimportant update parameters during model training. Since the parameters of each iteration need to be updated in each parameter server, reducing the update parameters essentially reduces the communication volume between networks, thereby reducing the system's demand for network bandwidth.

[0054] Specifically, in this preferred embodiment, step 100 (split the model and training data and deploy them on N worker machines) can be specifically expanded as follows: before model training begins, according to the PS architecture requirements, split the model and deploy it on N worker machines, and also divide the training data into N parts and deploy them on N worker machines.

[0055] like Figure 3 As shown, for step 200 of this preferred embodiment, the settings in the parameter importance evaluator may include:

[0056] Step 210: Set the model convergence fixed parameter T f .

[0057] Step 220: Set the parameter importance dynamic threshold T d .

[0058] Step 230: Set the network bandwidth threshold T b .

[0059] The embodiment of the present invention uses a parameter importance mechanism (i.e. setting a fixed parameter T for model convergence) f , Network bandwidth threshold T b And the dynamic threshold value T of parameter importance d ) is used to filter out non-important parameter updates. This is because during the training of mainstream machine learning models, more than 90% of parameter updates cause less than 1% change in the model parameters. Setting a parameter importance mechanism to limit updates to these non-important parameters will greatly reduce the amount of communication between parameter servers.

[0060] In addition, the parameter importance mechanism used in the embodiment of the present invention adopts a fixed parameter T for model convergence. f , parameter importance dynamic threshold T d , Network bandwidth threshold T b The combination of these three threshold values is achieved because:

[0061] (1) Based on the conclusion of a large amount of statistical data, we know that 90% of the parameters have less than 1% impact on the model. At this point, we can get the simplest parameter importance mechanism, which is to manually set the model convergence fixed parameter T f, used to ensure model training convergence.

[0062] like Figure 4 As shown, in this preferred embodiment, for setting T f The specific process of determining the value is as follows:

[0063] Step 211: Set minimum T f value.

[0064] Step 212: When the worker machine updates the parameter Δx and it is greater than T f When , the parameter server is allowed to update the parameters. The "updated parameter Δx" here can be understood as the absolute value ratio of the updated value to the original value.

[0065] For example, let T f = 0.01, which is equivalent to setting a hard threshold for parameter updates. Only parameter updates that affect the model by more than 0.01 are allowed. f The specific setting of T can be set manually according to the model size and training data distribution. When the model is too large or the variance of the training data is too small, T can be appropriately adjusted. f If the model is too small or the variance of the training data is too large, T f Setting a large threshold. When the model is too large or the variance of the training data is too small, the model will experience a large number of small changes in the later stages of training. At this time, the number of small changes is large. If a large hard threshold is set, it will affect the model quality. Therefore, it is possible to sacrifice the threshold limit effect in this case, that is, set a small threshold value to ensure that the model effect after training meets the expectations.

[0066] (2) However, since the training process of the machine learning model is dynamic, its parameter update state is also dynamic. Therefore, if the parameter importance mechanism is only determined by the model convergence fixed parameter T set in (1), f The effect will not be ideal, so a dynamic threshold is added, that is, the dynamic threshold value T of parameter importance. d , this dynamic threshold will be in line with the machine learning model training process, and maintain a large value in the early stage of model training, such as 0.05, 0.02, etc. (the specific value will be calculated according to the formula in the implementation method). Since the changes in model parameters are generally large at this time, T d Will effectively replace T f It plays the role of limiting the amount of communication, T d Soft threshold for model parameter update.

[0067] The dynamic threshold value Tx of parameter importance is used to filter unimportant parameters, such as Figure 5 As shown, in this preferred embodiment, for setting T d The specific process of determining the value is as follows:

[0068] Step 221: Set the dynamic Td value.

[0069] Step 222: When the worker machine updates the parameter Δx and it is greater than T d , allowing the parameter server to update the parameters.

[0070] (3) Network bandwidth threshold T b is used to ensure the minimum communication loss between wide area networks (WANs). It belongs to the environmental limit threshold and is formulated according to the actual inter-domain communication bandwidth to ensure that the inter-domain communication volume does not exceed the physical bandwidth. That is, when the instantaneous communication volume reaches the bandwidth limit, the network bandwidth threshold T is passed. b Suspending subsequent parameter updates will effectively ensure that all important parameters are not lost during inter-domain communication.

[0071] like Figure 6 As shown, in this preferred embodiment, for setting T b The specific process of determining the value is as follows:

[0072] Step 231: Set Maximum T b value.

[0073] Step 232: When the data transmission rate between the worker machine and the parameter server is greater than T b When , the communication between the parameter server and other parameter servers in the wide area network is suspended.

[0074] It should be noted that the T b It can be set manually according to the communication capacity between WANs in the environment. When the bandwidth between WANs is small, a smaller T can be set. b On the contrary, when the bandwidth between WANs is large, a larger T b .

[0075] From (1), (2), and (3), we can see that the model converges to a fixed parameter T f It can work alone but the effect is not good, so the dynamic threshold value T of parameter importance is introduced. d It will effectively improve the effect of limiting the amount of inter-domain communication in the early and middle stages of model training. Finally, due to the existence of the parameter importance mechanism, all inter-domain communications are important parameters. To ensure that all important parameters are not lost, a network bandwidth threshold value T is set. b Ensure communication security.

[0076] In addition, since the parameter importance mechanism needs to be used every time the parameters are updated, and there are a large number of parameter updates in the complete machine learning model training process, constructing the parameter importance mechanism in the lightest form will effectively reduce the demand for computing resources within the data center.

[0077] In summary, the model convergence fixed parameter Tf , parameter importance dynamic threshold T d , Network bandwidth threshold T b The parameter importance mechanism composed of these three parameters will be the balance point between light volume and full functionality.

[0078] In this preferred embodiment, T d The specific dynamic setting method is as follows:

[0079] Manually set the unit time t, and divide the total model training time T into n equal parts according to the unit time t, that is, the total training time T = {t1, t2, t3, ..., t n The unit time t can be set according to the system computing power and is generally set to 1ms. The setting of this unit time has no specific meaning and is only for the convenience of subsequent calculations.

[0080] At time t1, the parameter server collects the updated parameters X1(i)={x 11 ,x 12 ,x 13 ,…,x 1i ,…,x 1n}, find the first moment of X1 Second moment Where P(x) is the value of x in X1. 1i The probability density function of α is the restriction coefficient, α∈(0,1), and let the first-order moment average prediction value Second-order moment average prediction value In the above process, when and If it is too small, take a smaller α. and If it is too large, choose a larger α.

[0081] From time t2, at the end of the mth round of unit time window, the parameter server collects the updated parameters X of all worker machines in the data center. m (i) = {x m1 ,x m2 ,x m3 ,…,x mi ,…,x mn}, find X at this time m The first moment of and second-order moment renew Among them, β1 and β2 are weighted coefficients, β1,β2∈(0,1), then and Make corrections, In the above process, when the model is too large or the amount of training data is too large, β1 and β2 can be appropriately increased to make it more biased towards the latest data update trend. When the model is too small or the amount of training data is too small, β1 and β2 can be appropriately reduced to make it more biased towards the historical data update trend. The above steps all use the modified exponential moving weighted average method to formulate dynamic thresholds. Since the exponential moving weighted average method requires prediction of the observed values, this embodiment selects the first-order moment and second-order moment of each round of updated parameters as the observed values. The first-order moment is the sample expected value, and the second-order moment is the square of the expected difference between the sample variable and the expected value. The first-order moment and the second-order moment are used together as predicted values to better observe the distribution of the sequence data, which is conducive to the exponential moving weighted average method to better simulate the trend of the updated parameters.

[0082] Finally, update Among them, γ is the learning rate of the training model, which can be obtained from the training model. Generally speaking, γ∈(10 -6 1). When the model is too large or the number of learning epochs is small, γ can be set to a larger value. Conversely, when the model is small or after multiple learning epochs, γ can be appropriately reduced. When the model is too large or the number of learning epochs is small, increasing the learning rate helps the gradient to decrease quickly, which helps improve the convergence speed of the early model.

[0083] It should be noted that in the parameter calculation starting from time t2, the exponential moving weighted average mentioned is a commonly used sequence processing method, and its moving average formula at time t is V t =βV t-1 +(1-β)θ t ,t=1,2,3,…,n, where V t is the moving average forecast value at time t, θ t Where t is the true value at time t, and β is the weight. This formula uses the current actual value and the previous period (the time period of the average data is constrained by β) to smoothly modify the current value, generating a stable trend curve. Its physical meaning is that the smaller the β value, the lower the weight given to past measurements and the higher the weight given to current sample values. In this case, the timeliness of the moving average estimate is weaker, the predicted value is closer to the current sample value, and the stability of the prediction curve is worse. Conversely, the larger the β value, the more timeliness and stability of the prediction curve are.

[0084] The significance of the modified exponential moving weighted average is that when the exponential moving weighted average method is applied to the early stage of the sequence, there is often a large deviation, so it is necessary to introduce the deviation for correction. In the embodiment of the present invention, the original formula V t =βV t-1 +(1-β)θ t , corrected Modified exponential moving weighted average formula V't In the early stage of sequence prediction, it will be closer to the current true measurement value, reducing the error of the initial prediction. In the later stage of sequence prediction, as t increases, 1-β t Will approach 1, the modified V' t It is also close to the V before correction t .

[0085] The following is an explanation of how the parameter importance mechanism can ensure model convergence in the overall process of this embodiment: Assume that the distributed machine learning model uses stochastic gradient descent (SGD) and uses the parameter importance mechanism to screen parameters before inter-domain communication. Let f be the objective function, let is the model parameter during training, x * To optimize the model parameters when completed, the model learning process is to find the appropriate parameters to minimize f, that is, At this time, the model gradient descent formula is The gradient descent step length η t according to Decline, if and Then when t→∞, let at this time but Therefore, the objective function will converge within the exact time period T.

[0086] Through the above design, after completing the deployment of the model and parameter importance evaluator, you can start distributed machine learning model training. After adding the above parameter update mechanism, the demand for network communication capabilities for model training will be greatly reduced.

[0087] In summary, it can be seen from this preferred embodiment that the present invention solves the problem of distributed machine learning training rate by introducing a parameter importance mechanism; solves the problem of distributed machine learning having excessively high network bandwidth requirements by introducing a parameter importance mechanism; and by using the parameter importance mechanism to eliminate updates of unimportant parameters, the bandwidth requirements of distributed machine learning between WANs can be reduced, and the computational workload of the parameter server can be reduced. Compared with inter-domain bandwidth resources, the cost of obtaining computing resources within the data center is also much lower.

[0088] Example 2:

[0089] Based on the wide area network cross-data center distributed machine learning model parameter update method provided in Example 1, this Example 2 provides a parameter importance evaluator usage solution in a cloud-network fusion scenario based on the integration of computing, storage, and transmission resources.

[0090] In the cloud-network integration scenario, Figure 7 When two data centers are interconnected, the cross-domain unified data center network management, namely the Super Controller (SC), performs preliminary perception of the service traffic in the Data Center Interconnect (DCI) device and determines the service type. Then, a real-time evaluation of the current network status is performed, including but not limited to important indicators such as the computing power of the internal network of the data center (DCN), the storage capacity of the DCN, and the transmission capacity of the DCI, to determine whether the service will cause network congestion. If the judgment result is that the service traffic will affect the network service level agreement (SLA), the SC proposes to use the wide area network cross-data center distributed machine learning model parameter update method provided in Example 1 for the distributed machine learning service perceived in the DCN, or automatically runs the solution while ensuring the stability of the service to ensure the health of the network quality.

[0091] Specifically, when SC detects that the current network quality does not meet the SLA and needs to use this model parameter update method, it will first determine the model convergence fixed parameter T f and dynamic threshold T d , the initial fixed threshold is set to T f =0.01, dynamic threshold T d =0.005. Then determine the network bandwidth threshold value T according to the current network status b , T b That is, it is set to the maximum remaining bandwidth of the network at that time. Once the initial parameters are set, the distributed machine learning algorithm can be started under the parameter constraints.

[0092] During the training process, the dynamic threshold value T d It will keep changing dynamically with the above formula to ensure the screening of the importance of parameters in the early stage of model training. At the same time, during the process, the data center will monitor the real-time remaining bandwidth of the network and correct the network bandwidth threshold value T at this time. b .

[0093] Model convergence fixed parameter T f There are two setting schemes. When the system detects that the model size is small (for example, when the model parameter size is less than 1Mb), the initial value, T f = 0.01 to run the entire model training process. When the system detects that the model scale is large (for example, when the model parameter volume is greater than 1Mb), T can be appropriately reduced after each round of model training. f , that is, set (where t is the number of model iterations) to ensure that the model converges faster.

[0094] While the parameter importance mechanism is running, the data transmission quality between data center networks will also be monitored in real time. If it is detected that the network congestion has been alleviated, the mechanism can be terminated at any time during its operation and the normal training mode can be restored.

[0095] In the scenario of this embodiment 2, the wide area network cross-data center distributed machine learning model parameter update method provided in embodiment 1 can improve system performance by 2-10 times.

[0096] Example 3:

[0097] Based on the wide area network cross-data center distributed learning model parameter updating method provided in Example 1, this Example 3 provides a wide area network cross-data center distributed learning model parameter updating system corresponding to Example 1, such as Figure 8 As shown in the figure, the system architecture includes a parameter server, a worker machine, and a parameter importance evaluator.

[0098] In this preferred embodiment, the parameter server, i.e. Figure 8 Parameter servers (PSs) communicate with each other over a wide area network (WAN), and multiple worker machines communicate with the parameter servers to ensure that each round of parameter updates on the worker machines is updated in real time across all PSs. Furthermore, a parameter importance evaluator is deployed on each parameter server to filter out unimportant parameters and reduce system workload.

[0099] When training the model, first split the model and deploy it on N worker machines according to the PS architecture requirements, and then split the training data into N parts and deploy them on N worker machines. Then, set the three parameters needed in the parameter importance evaluator: the model convergence fixed parameter T f , parameter importance dynamic threshold T d , Network bandwidth threshold T b After the model and parameter importance estimator are deployed, you can start distributed machine learning model training.

[0100] Specifically, by setting T f Ensure model training convergence, that is, set the minimum T f When the worker machine updates the parameter Δx, it is greater than T f When PS is allowed to update parameters, T f It is the hard threshold value for updating model parameters, which can be set manually according to the model size and training data distribution. When the model is too large or the variance of the training data is too small, T can be appropriately adjusted. fIf the setting is too small, the T f The setting is too large.

[0101] By setting T b Ensure the minimum communication loss between WANs, that is, set the maximum T b When the data transmission rate between the worker machine and PS is greater than T b When the PS is suspended from communicating with other PSs in the WAN, T b It can be manually set according to the communication capacity between WANs in the environment. When the bandwidth between WANs is small, a smaller T can be set. b Otherwise, a larger T b .

[0102] By setting T d Filter unimportant parameters, that is, set dynamic T d When the worker machine updates the parameter Δx, it is greater than T d When PS is allowed to update parameters, T d Soft threshold for model parameter update.

[0103] Above T f 、T d 、T b The specific derivation formula, setting range, and setting reasons of the parameter setting are all described in detail in Example 1 and will not be repeated in this embodiment.

[0104] The system architecture provided in this embodiment can support model training of distributed machine learning models across data centers in a wide area network. By establishing a parameter update strategy, unimportant update parameters during model training are eliminated. Since the parameters of each iteration need to be updated in each parameter server, reducing the update parameters essentially reduces the communication volume between networks, thereby reducing the system's demand for network bandwidth.

[0105] Example 4:

[0106] Based on the wide area network cross-data center distributed learning model parameter updating method and system provided in the above embodiments 1 to 3, the present invention also provides a wide area network cross-data center distributed learning model parameter updating device that can be used to implement the above method and system, such as Figure 9 , is a schematic diagram of the device architecture of an embodiment of the present invention. The wide area network cross-data center distributed learning model parameter update device of this embodiment includes one or more processors 21 and a memory 22. Figure 9 A processor 21 is taken as an example.

[0107] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 9The bus connection is taken as an example.

[0108] Memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the wide area network cross-data center distributed learning model parameter update method and system in Examples 1 and 2. Processor 21 executes the various functional applications and data processing of the wide area network cross-data center distributed learning model parameter update device by running the non-volatile software programs, instructions, and modules stored in memory 22, thereby implementing the wide area network cross-data center distributed learning model parameter update method and system in Examples 1 to 3.

[0109] The memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21, and such remote memory may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0110] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the method and system for updating parameters of the wide area network cross-data center distributed learning model in the above-mentioned embodiments 1 to 3 are executed. For example, Figures 2 to 6 The steps shown.

[0111] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a disk or an optical disk, etc.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for updating parameters of a distributed learning model across data centers in a wide area network, characterized in that: include: Split the model and training data and deploy them on N worker machines; The same parameter importance evaluator is set on each parameter server. The parameter importance evaluator includes three parameters: model convergence fixed parameter , Network bandwidth threshold And the dynamic threshold of parameter importance At different times during training, the parameter server collects the updated parameters of all worker machines in the data center. Based on the historical updated parameters and the latest updated parameters, it dynamically weights the average predicted values of the first-order moments and the second-order moments and updates the learning rate of the model training. The weighting coefficient is used to adjust the reference ratio of the historical updated parameters to the latest updated parameters. Use modified exponential moving weighted average method to formulate dynamic threshold of parameter importance , increasing the weighting coefficient makes the model update more focused on the latest data update trend or decreasing the weighting coefficient makes the model update more focused on the historical data update trend; Start distributed machine learning model training.

2. The method for updating parameters of a distributed learning model across data centers in a wide area network according to claim 1, wherein: By setting Ensure model training convergence, specifically: set the minimum Value, when the worker machine updates the parameter Greater than , allowing the parameter server to update the parameters.

3. The method for updating parameters of a distributed learning model across data centers in a wide area network according to claim 2, wherein: described Manually set according to the model size and training data distribution. When the model is too large or the training data variance is too small, If the setting is too small, when the model is too small or the variance of the training data is too large, The setting is too large.

4. The method for updating parameters of a distributed learning model across data centers in a wide area network according to claim 1, wherein: By setting Ensure minimum communication loss between WANs. Specifically: Set the maximum When the data transmission rate between the worker machine and the parameter server is greater than When , the communication between the parameter server and other parameter servers in the wide area network is suspended.

5. The method for updating parameters of a distributed learning model across data centers in a wide area network according to claim 4, characterized in that: described Manually set according to the communication capacity between WANs in the environment. Set a smaller value when the bandwidth between WANs is small. , when the bandwidth between WANs is large, set a larger .

6. The method for updating parameters of a distributed learning model across data centers in a wide area network according to claim 1, wherein: By setting Filter unimportant parameters, specifically: set dynamic Value, when the worker machine updates the parameter Greater than , allowing the parameter server to update the parameters.

7. The method for updating parameters of a distributed learning model across data centers in a wide area network according to claim 6, wherein: described The specific settings include: Manually set the unit time t, and divide the total model training time T into n equal parts according to the unit time t, that is, the total training time ; At this moment, the parameter server collects the updated parameters of all worker machines in the data center ,beg The first moment of , second-order moment ,in for middle The probability density function of , is the restriction coefficient, , and let the first-order moment average prediction value , the second-order moment average prediction value ; from At the end of the mth round of unit time window, the parameter server collects the updated parameters of all worker machines in the data center. , ask for this time The first moment of and second-order moment ,renew 、 ,in, , then and Make corrections, 、 , last updated ,in, is the learning rate of the training model obtained from the training model, .

8. The method for updating parameters of a distributed learning model across data centers in a wide area network according to claim 7, wherein: when and If it is small, take the smaller one ,when and If it is too large, take the larger ; When the model is too large or the amount of training data is too large, increase and , making it more inclined to the latest data update trend. When the model is small or the amount of training data is small, reduce and , making it focus on the update trend of historical data; when the model is too large or the learning round is small, A larger value is used when the model is small or after multiple rounds of learning. The value is smaller.

9. The method for updating parameters of a distributed learning model across data centers in a wide area network according to any one of claims 1 to 8, wherein: Deploying the model and training data on N worker machines specifically includes: Before model training begins, according to the PS architecture requirements, the model is split and deployed on N worker machines, and the training data is also divided into N parts and deployed on N worker machines.

10. A device for updating parameters of a distributed learning model across data centers in a wide area network, characterized by: It includes at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the wide area network cross-data center distributed learning model parameter updating method described in any one of claims 1-9.

Images