Distributed model training optimization method and device, equipment, medium and program product

Abstract

The invention provides a distributed model training optimization method and device, equipment, a medium and a program product, and relates to the technical field of model training. The distributed model training optimization method comprises the following steps: selecting neighbor edge nodes in an edge node cluster to construct a distributed training topology; the method comprises the following steps: executing periodic training of a distributed model in a distributed training topology, completing a training period, calculating a local gradient of a model parameter obtained by training, receiving a neighbor compression gradient sent by a neighbor edge node, decompressing the neighbor compression gradient to obtain a neighbor gradient of the model parameter obtained by calculating the training of the neighbor edge node, and calculating the neighbor gradient of the model parameter obtained by calculating the training of the neighbor edge node. Performing aggregation operation on the local gradient and the neighbor gradient to obtain an aggregation gradient, performing momentum compensation on the aggregation gradient to obtain a compensation gradient, and updating model parameters based on the compensation gradient; and when detecting that the model parameters meet the convergence condition of the model, stopping periodic training. Through the technical scheme of the invention, the data volume of single communication can be reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of model training, and particularly relates to a distributed model training optimization method, a distributed model training optimization device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] The decentralized training method refers to a method of realizing model training through direct communication and autonomous cooperation between nodes without relying on a central node to aggregate parameters or coordinate updates. Although the decentralized architecture avoids the single point failure and communication bottleneck problem of the centralized architecture, the nodes in the decentralized architecture need to frequently exchange gradient information with adjacent nodes, which not only leads to an increase in gradient communication overhead, but also causes a time difference in gradient updates if the node computing capacity and network delay are not synchronized, thereby leading to slow model convergence speed or falling into a local optimum.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The purpose of the present disclosure is to provide a distributed model training optimization method, a training device, an electronic device, a storage medium and a computer program product, which at least partially overcome the problems of large gradient communication overhead and slow model convergence speed in the model training process in the related art.

[0005] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a distributed model training optimization method is provided, comprising: selecting neighbor edge nodes in an edge node cluster to construct a distributed training topology; performing periodic training of the distributed model in the distributed training topology, wherein one training period is completed, a local gradient of a model parameter obtained by training is calculated, neighbor compressed gradients sent by the neighbor edge nodes are received, the neighbor compressed gradients are decompressed to obtain neighbor gradients of the model parameter obtained by training of the neighbor edge nodes, an aggregation operation is performed on the local gradient and the neighbor gradient to obtain an aggregated gradient, momentum compensation is performed on the aggregated gradient to obtain a compensated gradient, and the model parameter is updated based on the compensated gradient; and detecting that the model parameter satisfies a convergence condition of the model, and stopping the periodic training.

[0007] In one embodiment of the present disclosure, further comprising: performing a gradient compression operation on the local gradient to obtain a local compressed gradient; and sending the local compressed gradient to at least one of the neighbor edge nodes based on asynchronous communication.

[0008] In one embodiment of the present disclosure, the gradient includes a weight gradient matrix and a bias gradient vector, and performing a gradient compression operation on the local gradient to obtain a local compressed gradient includes: calculating a number of elements to be reserved in the local weight gradient matrix based on a gradient compression ratio configured by a network bandwidth; arranging all elements in the local gradient in descending order based on absolute values to obtain an arrangement sequence; reserving gradient elements in a front row of the arrangement sequence in the number of elements to set other gradient elements to zero to generate a local sparse gradient; and performing quantization processing on the local sparse gradient to obtain the local compressed gradient.

[0009] In one embodiment of the present disclosure, sending the local compressed gradient to at least one of the neighbor edge nodes based on asynchronous communication includes: submitting an asynchronous task to a communication thread pool of the distributed training topology, the asynchronous task being used to traverse a neighbor edge node list of the distributed training topology to send the local compressed gradient to each of the neighbor edge nodes through the communication thread pool, the communication thread pool being a single-threaded executor.

[0010] In one embodiment of the present disclosure, decompressing the neighbor compressed gradient to obtain a neighbor gradient of a model parameter calculated by the neighbor edge node includes: extracting a neighbor sparse gradient, quantization metadata and original dimension information from the neighbor compressed gradient; initializing a full zero gradient matrix based on the original dimension information; performing inverse quantization processing on the neighbor sparse gradient based on the quantization metadata to obtain floating point values of non-zero gradient elements; and filling the floating point values to corresponding positions of the full zero gradient matrix based on index information of the neighbor sparse gradient to obtain the neighbor gradient.

[0011] In one embodiment of the present disclosure, compensating the aggregated gradient by momentum to obtain a compensated gradient includes: compensating the aggregated gradient by momentum to obtain a momentum buffer gradient; and compensating the momentum buffer gradient by error based on an error compensation mechanism to obtain the compensated gradient.

[0012] In one embodiment of the present disclosure, compensating the aggregated gradient by momentum to obtain a momentum buffer gradient includes: configuring a momentum coefficient based on a number of completed training cycles; and determining the momentum buffer gradient based on the momentum coefficient, a historical momentum buffer value and the aggregated gradient, wherein the historical momentum buffer value represents cumulative influence of an update direction obtained by the completed training cycles.

[0013] In an embodiment of the present disclosure, the error compensation mechanism is used to compensate the momentum buffered gradient to obtain the compensated gradient, including: obtaining a historical compensation buffer value obtained in a previous training cycle; and adding the historical compensation buffer value to the momentum buffered gradient to obtain the compensated gradient, wherein the compensated gradient is multiplied by a configured learning rate to obtain the historical compensation buffer value in the current training cycle.

[0014] In an embodiment of the present disclosure, the local gradient of the model parameter obtained by training is calculated, including: calculating an original gradient of the model parameter based on forward propagation and back propagation operations; and applying local momentum compensation to the original gradient to obtain the local gradient.

[0015] In an embodiment of the present disclosure, the local momentum compensation is applied to the original gradient to obtain the local gradient, including: determining the local gradient based on the momentum coefficient, the historical momentum buffer value and the original gradient.

[0016] In an embodiment of the present disclosure, the neighbor compressed gradient sent by the neighbor edge node is received, including: receiving the neighbor compressed gradient and putting the neighbor compressed gradient into a blocking queue; and taking out the neighbor compressed gradient from the blocking queue based on a non-blocking processing mode.

[0017] In an embodiment of the present disclosure, the neighbor edge node in the edge node cluster is selected to construct a distributed training topology, including: obtaining a neighbor edge node list of the edge node cluster; selecting a plurality of neighbor edge nodes satisfying distance requirements from the neighbor edge node list based on identification information and resource state information of each neighbor edge node in the neighbor edge node list, and establishing a communication connection with the plurality of neighbor edge nodes to construct the distributed training topology and the communication thread pool.

[0018] In an embodiment of the present disclosure, the model parameters include a weight matrix determined based on an input feature dimension and a hidden layer dimension of the model and a bias vector determined based on the hidden layer dimension, the compensation gradient includes a compensation weight gradient matrix corresponding to a dimension of the weight matrix and a compensation bias gradient vector corresponding to the bias vector, and updating the model parameters based on the compensation gradient includes: determining a corresponding first update value based on the compensation weight gradient matrix and a configured learning rate, wherein the first update value is a product of the learning rate and each element in the compensation weight gradient matrix; updating each weight value in the weight matrix based on the corresponding first update value to obtain an updated weight matrix; determining a corresponding second update value based on the compensation bias gradient vector and the learning rate, wherein the second update value is a product of the learning rate and each element in the compensation bias gradient vector; updating each bias value in the bias vector based on the corresponding second update value to obtain an updated bias vector, and obtaining an updated model parameter based on the updated weight matrix and the updated bias vector.

[0019] In an embodiment of the present disclosure, detecting that the model parameters satisfy a convergence condition of the model includes: collecting multi-dimensional weight parameters of all nodes in the distributed training topology based on the updated weight matrix; extracting weight values of the all nodes in any dimension from the multi-dimensional weight parameters and calculating an average value of the all nodes in the any dimension; calculating a variance of the any dimension based on the weight values of each of the nodes in the any dimension and the average value; determining an average variance of the all nodes based on variances of all the dimensions; and detecting that the average variance decreases to less than or equal to a variance threshold to determine that the convergence condition of the model is satisfied.

[0020] In an embodiment of the present disclosure, before performing periodic training of the distributed model in the distributed training topology, the method further includes: initializing a weight gradient matrix and a bias gradient vector based on an input dimension and an output dimension of the model, wherein the weight gradient matrix is initialized based on a random Gaussian distribution, and an initial value of the bias gradient vector is 0.

[0021] According to another aspect of the present disclosure, a distributed model training optimization apparatus is provided, comprising: a construction module, a user selects a neighbor edge node in an edge node cluster to construct a distributed training topology; a training module, configured to perform periodic training of the distributed model in the distributed training topology, wherein, after completing one training period, a local gradient of a model parameter obtained by training is calculated, a neighbor compressed gradient sent by the neighbor edge node is received, the neighbor compressed gradient is decompressed to obtain a neighbor gradient of the model parameter obtained by training of the neighbor edge node, an aggregation operation is performed on the local gradient and the neighbor gradient to obtain an aggregated gradient, momentum compensation is performed on the aggregated gradient to obtain a compensated gradient, and the model parameter is updated based on the compensated gradient; and a detection module, configured to detect that the model parameter satisfies a convergence condition of the model, and stop the periodic training.

[0022] According to still another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to perform the distributed model training optimization method of the first aspect described above via execution of the executable instructions.

[0023] According to yet another aspect of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, the computer program being executed by a processor to implement the distributed model training optimization method described above.

[0024] According to yet another aspect of the present disclosure, a computer program product is provided, having stored thereon a computer program, the computer program being executed by a processor to implement the distributed model training optimization method described above.

[0025] The distributed model training optimization scheme provided by the embodiments of the present disclosure, by dynamically constructing a local neighbor communication topology based on edge nodes, performs decentralized neighbor topology construction, and in each training period, after each edge node calculates a local gradient, the local gradient is compressed and sent to a neighbor, while receiving and decompressing a neighbor gradient, the neighbor gradient is aggregated with the local gradient, and the model parameter is updated after momentum compensation, and the training is terminated when the model convergence is determined. On the one hand, the decentralized neighbor topology, and through the gradient compression and decompression mechanism, the compressed gradient is transmitted between nodes instead of the complete gradient, which can significantly reduce the data volume of a single communication. On the other hand, the momentum mechanism accumulates the historical update direction, reduces the fluctuation influence of a single asynchronous update, and the periodic training and stopping the training based on the parameter convergence condition ensure that the nodes can still converge to the global optimum in dynamic updating, so as to prevent the model from falling into a local optimum or converging too slowly.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate implementations of the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is apparent that the drawing in the following description is only some embodiments of the present disclosure, and other drawings can be obtained by those of ordinary skill in the art without creative labor on the basis of these drawings.

[0028] Figure 1 A flow chart illustrating a distributed model training optimization method in an embodiment of the present disclosure is shown;

[0029] Figure 2 A flow chart illustrating another distributed model training optimization method in an embodiment of the present disclosure is shown;

[0030] Figure 3 A flow chart illustrating still another distributed model training optimization method in an embodiment of the present disclosure is shown;

[0031] Figure 4 A flow chart illustrating yet another distributed model training optimization method in an embodiment of the present disclosure is shown;

[0032] Figure 5 A flow chart illustrating still another distributed model training optimization method in an embodiment of the present disclosure is shown;

[0033] Figure 6 A schematic diagram of a distributed model training optimization apparatus in an embodiment of the present disclosure is shown;

[0034] Figure 7 A structural block diagram of a computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more implementations.

[0036] In addition, the accompanying drawings are merely schematic and are not intended to be drawn to scale. Identical reference numerals denote like or similar parts throughout the several views, so that repeated description is omitted. Some of the block diagrams shown in the drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] Currently, the distributed training methods mainly include parameter server architecture and AllReduce synchronous training. The parameter server architecture manages model parameters and aggregates gradients through one or a group of central nodes (parameter servers), and the AllReduce synchronous training is a distributed training mode without central nodes, which relies on collaborative communication between nodes to complete parameter update. The former relies on central nodes to aggregate gradients, has the risk of single point failure and communication bottleneck, requires all nodes to update synchronously, and has high network delay and poor fault tolerance.

[0038] Although the decentralized algorithm avoids the centralization problem, it still has defects such as static topology structure that cannot adapt to dynamic network environment, large gradient communication overhead, and gradient deviation caused by asynchronous update.

[0039] Therefore, there is an urgent need for a distributed model training optimization scheme that reduces communication overhead through dynamic topology and gradient compression, combines asynchronous update and local momentum compensation, and ensures efficient convergence of the model.

[0040] For ease of understanding, the following first explains several terms related to the present application.

[0041] Decentralized architecture: a distributed system design that does not rely on a central server or parameter server, with each node exchanging data directly through point-to-point communication, avoiding single point failure and communication bottleneck.

[0042] Distributed training: distributing the training task of a machine learning model to multiple computing nodes for parallel execution to speed up the training process and handle large-scale data.

[0043] Machine learning: a technology that enables a computer system to learn from data and improve performance through algorithms without explicit programming.

[0044] Neural network: a computational model composed of multiple layers of neurons, trained through forward propagation and backpropagation algorithms, used to solve classification, regression, and other tasks.

[0045] Dynamic topology: in distributed training, the connection relationship between nodes is dynamically adjusted according to network conditions and computing capacity to optimize communication efficiency.

[0046] Gradient compression: Reduce the amount of data transmitted by techniques such as sparsification and quantization, reducing communication overhead.

[0047] Asynchronous update: Nodes do not need to wait for synchronization signals from other nodes during training, and can independently calculate and update model parameters, improving resource utilization.

[0048] Local momentum compensation: In decentralized training, the momentum term is used to compensate for gradient deviation, ensuring model convergence stability.

[0049] In the following, the steps of the distributed model training optimization method in the example embodiment will be described in more detail in conjunction with the drawings and examples.

[0050] Figure 1 A flowchart of a distributed model training optimization method in an embodiment of the present disclosure is shown.

[0051] As Figure 1 shown, the distributed model training optimization method according to one embodiment of the present disclosure includes:

[0052] Step S102, select neighbor edge nodes in the edge node cluster to construct a distributed training topology.

[0053] In some embodiments, the initialization process of the edge node cluster involves network structure parameters including:

[0054] The dimension of the input feature is FEATURE_DIM=1000, i.e. each sample includes 1000 features.

[0055] The dimension of the neural network hidden layer is HIDDEN_DIM=512, including 512 neurons, forming a fully connected layer from 1000 to 512.

[0056] The optimizer parameters involved include:

[0057] The learning rate LEARNING_RATE=0.01 controls the step size of model parameter update, and the smaller the value, the more gentle the update.

[0058] The momentum coefficient MOMENTUM=0.9 is used to smooth the gradient update direction and reduce the impact of random fluctuations on training.

[0059] The communication optimization parameters involved include:

[0060] The gradient compression ratio COMPRESSION_RATIO=10 reduces the amount of data transmission between nodes by selecting key gradients.

[0061] In some embodiments, the k nodes closest to the current node can be selected from the edge node cluster based on node identification information (such as ID) or network topology distance (such as communication delay) to form a decentralized local communication topology, each node only establishes a connection with the neighbor edge nodes, rather than full-quantity node interconnection, thereby reducing the number of communication links.

[0062] In step S104, periodic training of the distributed model is performed in the distributed training topology, wherein one training period is completed, the local gradient of the model parameter obtained by training is calculated, the neighbor compressed gradient sent by the neighbor edge node is received, the neighbor compressed gradient is decompressed to obtain the neighbor gradient of the model parameter obtained by training of the neighbor edge node, the local gradient and the neighbor gradient are aggregated to obtain an aggregated gradient, the aggregated gradient is compensated by momentum to obtain a compensated gradient, and the model parameter is updated based on the compensated gradient.

[0063] In some embodiments, in each training period, the node first calculates the local gradient, compresses the local gradient by Top-K sparsification and quantization, and sends it to the neighbor, while receiving the compressed gradient of the neighbor, aggregates the decompressed local gradient and the neighbor gradient, such as averaging, and finally updates the local model parameter with the compensated gradient. The entire process realizes the decoupling of calculation and communication through asynchronous communication and blocking queue.

[0064] In step S106, it is detected that the model parameter satisfies the convergence condition of the model, and the periodic training is stopped.

[0065] In some embodiments, the model weight parameters of all nodes in the distributed topology are collected periodically, the weight matrix of each node is flattened into a one-dimensional array, the average value and variance of all nodes in each parameter dimension are calculated, and the average variance of all dimensions is calculated. When the average variance decreases to a preset threshold, such as close to 0, it indicates that the parameters of each node are highly consistent, and the model is determined to be converged.

[0066] In this embodiment, by dynamically constructing an edge node-based local neighbor communication topology, a decentralized neighbor topology is constructed. In each training period, after each edge node calculates the local gradient, it is compressed and sent to the neighbor, while receiving and decompressing the neighbor gradient, aggregating it with the local gradient, updating the model parameters after momentum compensation, and terminating the training when the model convergence is determined. On the one hand, the decentralized neighbor topology and the compressed gradient transmission between edge nodes instead of the complete gradient through the gradient compression and decompression mechanism can significantly reduce the data volume of a single communication in the edge business scenario. On the other hand, the momentum mechanism accumulates historical update directions, reduces the fluctuation of a single asynchronous update, and periodic training and parameter convergence condition-based stopping training ensure that the edge node can still converge to the global optimum in dynamic updating to prevent the model from falling into a local optimum or converging too slowly.

[0067] In some embodiments, the products applied to the edge node cluster include, but are not limited to, smart home device clusters, industrial Internet of Things edge gateways, vehicle-mounted intelligent terminal networks, and smart wearable device groups, etc. Through the gradient compression and neighbor node communication mechanism, the data transmission volume between edge nodes is reduced, which is conducive to preventing network congestion and ensuring the timeliness of smart device response. Through the aggregated gradient and error compensation mechanism, the communication overhead is reduced while the model accuracy is maintained.

[0068] In some embodiments, the smart home device cluster includes smart speakers, smart cameras, smart sensors (temperature and humidity, door and window sensors, etc.), and other network devices. These devices need to cooperatively train a user behavior prediction model to realize intelligent scene linkage.

[0069] In some embodiments, the industrial Internet of Things edge gateway is deployed on the edge computing device in the factory workshop, responsible for collecting industrial sensor data (such as device vibration, current and voltage data), and cooperatively training a device fault prediction model to support predictive maintenance.

[0070] In one embodiment of the present disclosure, the local gradient is subjected to a gradient compression operation to obtain a local compressed gradient, and the local compressed gradient is sent to at least one neighbor edge node based on asynchronous communication.

[0071] In some embodiments, in each training period, after the node completes local gradient calculation, the node first generates a local compressed gradient by compressing the gradient, and then submits the compressed gradient to an independent communication thread pool, such as commExecutor, and sends it to the neighbor edge node through an asynchronous task without waiting for a response. After the neighbor edge node receives the compressed gradient, it is stored in a blocking queue, such as gradQueue, and the main thread takes the gradient out of the queue for subsequent processing in a non-blocking manner.

[0072] In this embodiment, by compressing the local gradient and sending the compressed gradient to the neighbor edge node through an independent thread pool asynchronously, and by using a blocking queue to cache the gradient for non-blocking processing at the receiving end, parallelization of gradient transmission and model calculation is achieved, and through the compression scheme combined with asynchronous communication, the node reduces the network bandwidth pressure while improving the training throughput.

[0073] In one embodiment of the present disclosure, the gradient includes a weight gradient matrix and a bias gradient vector, and the gradient compression operation on the local gradient to obtain the local compressed gradient includes: calculating the number of elements to be retained in the local weight gradient matrix based on the gradient compression ratio configured by the network bandwidth; arranging all elements in the local gradient in descending order based on the absolute value to obtain an arrangement sequence; retaining gradient elements in the arrangement sequence in the selected top number of elements to set other gradient elements to zero to generate a local sparse gradient; and quantizing the local sparse gradient to obtain the local compressed gradient.

[0074] In some embodiments, the quantization processing refers to a process of mapping continuous range of floating point values to a limited number of discrete integer values, aiming to further reduce the storage bit number of gradient data.

[0075] In some embodiments, for the weight gradient matrix and the bias gradient vector in the gradient, the number of elements to be retained K is calculated according to a preset compression ratio, such as 10%, then all elements in the gradient matrix are traversed, sorted in descending order based on the absolute value, the top K elements are retained as important gradients, and the remaining elements are set to zero to form a sparse gradient matrix, and the retained non-zero elements are quantized, such as mapping 32-bit floating point numbers to 8-bit integers, to further reduce the data bit number, and through the mixed compression of sparsification and quantization, the data volume is greatly reduced while the key gradient information is retained, and the compressed gradient is stored and transmitted in the form of index-value pair, which is beneficial to improving the communication efficiency.

[0076] In this embodiment, by performing Top-K sparsification and quantization processing on the gradient matrix based on the configured ratio, the compressed gradient in the form of "index-value" pair is generated, important gradient information is selectively retained, the communication data volume is compressed while ensuring the correctness of the model update direction, the memory occupation and transmission delay are reduced through efficient sparse representation, and the quantization processing further reduces the storage bit number of each element, which is beneficial to alleviating the network bottleneck in distributed training and improving the system throughput.

[0077] In one embodiment of the present disclosure, the local compressed gradient is sent to at least one neighbor edge node based on asynchronous communication, including: submitting an asynchronous task to a communication thread pool of the distributed training topology, the asynchronous task being used to traverse a neighbor edge node list of the distributed training topology to send the local compressed gradient to each neighbor edge node through the communication thread pool, and the communication thread pool being a single-threaded executor.

[0078] In some embodiments, after generating a local compressed gradient, the node does not send it directly through the main thread. Instead, the gradient sending logic is encapsulated as an asynchronous task and submitted to an independent communication thread pool, namely a single-threaded executor. The single-threaded executor executes tasks sequentially, traverses the list of neighbor edge nodes, and sends the compressed gradient to each neighbor edge node in turn. During this process, the main thread can continue local model calculation without waiting for the sending to be completed, thus realizing parallel processing of calculation and communication. After the neighbor edge node receives the gradient, it is cached in a blocking queue for subsequent non-blocking processing.

[0079] In this embodiment, by combining a single-threaded communication thread pool and asynchronous task submission, the gradient sending process is decoupled from the local computing process. The single-threaded executor ensures the orderliness of gradient sending. At the same time, the asynchronous mechanism enables nodes to continuously perform model training calculations during network transmission, reducing the idle time of computing resources. While improving computing resource utilization, it helps alleviate the problem of reduced training efficiency caused by network delays.

[0080] like Figure 2 As shown, in one embodiment of the present disclosure, decompressing the neighbor compressed gradient to obtain the neighbor gradient of the model parameter obtained by the neighbor edge node calculation training includes:

[0081] Step S202 : extracting neighbor sparse gradients, quantization metadata, and original dimension information from the neighbor compressed gradients.

[0082] In some embodiments, the compressed gradient data sent from the neighbor edge node is parsed to obtain the neighbor sparse gradients, including the index and quantization value of the non-zero gradient elements, quantization metadata, such as the number of quantization bits, the maximum value range, and other parameters used for inverse quantization, and dimensional information, such as the number of rows and columns of the gradient matrix. The original dimension determines the structure of the gradient matrix, the metadata is used to restore the accuracy of the quantization value, and the sparse gradient contains the actual valid gradient data.

[0083] Step S204: Initialize an all-zero gradient matrix based on the original dimension information.

[0084] In some embodiments, an all-zero matrix of the same dimension is initialized based on the extracted original dimensional information. The zero matrix is ​​used to subsequently fill in non-zero gradient elements to ensure that the final restored gradient matrix is ​​consistent with the original gradient in shape and meets the requirements of the model calculation for gradient dimension.

[0085] Step S206 : performing inverse quantization processing on the neighbor sparse gradient based on the quantization metadata to obtain floating-point values ​​of non-zero gradient elements.

[0086] In some embodiments, dequantization is the inverse process of quantization, which restores discrete integer values to approximate floating-point values, relying on the metadata saved during quantization.

[0087] In step S208, based on the index information of the neighbor sparse gradient, the floating-point value is filled into the corresponding position of the full zero gradient matrix to obtain the neighbor gradient.

[0088] In some embodiments, the index of the non-zero element is extracted from the neighbor sparse gradient, such as (i, j) representing the i-th row and j-th column of the matrix. The floating-point value obtained by dequantization is filled into the initialized full zero matrix according to the index position to restore the complete neighbor gradient matrix, which not only preserves the effective information during sparse compression, but also ensures the integrity of the gradient matrix.

[0089] In this embodiment, by analyzing the original dimension, metadata and sparse gradient, combining full zero matrix initialization and index filling, the accuracy of decompression is ensured, the floating-point precision of the gradient is restored through dequantization, and the restored gradient can directly participate in local gradient aggregation, matching the sparsification and quantization operations during gradient compression, and ensuring the effective recovery of gradient information, providing a basis for accurate aggregation of neighbor gradient and local gradient, finally maintaining the parameter consistency of distributed training under low communication cost, and improving the stability of model convergence.

[0090] In one embodiment of the present disclosure, the aggregated gradient is compensated for momentum to obtain a compensated gradient, comprising: compensating the aggregated gradient for momentum to obtain a momentum buffer gradient; and compensating the momentum buffer gradient for error based on an error compensation mechanism to obtain the compensated gradient.

[0091] In some embodiments, through the momentum compensation mechanism, the current aggregated gradient is combined with the historically accumulated momentum buffer value, with a preset momentum coefficient, such as 0.9, to retain the inertia of the historical update direction, while incorporating new information of the current aggregated gradient to form a smooth momentum buffer gradient, reducing the impact of single gradient fluctuation on model update.

[0092] In some embodiments, for the information lost during gradient compression, an error buffer is maintained through an error compensation mechanism to add the accumulated error value from the previous round of training due to compression to the momentum buffer gradient to correct the information loss caused by compression, and finally obtain a compensated gradient that takes into account the historical update inertia and compression error correction.

[0093] In this embodiment, configuring momentum compensation utilizes the historical gradient accumulation effect to make the model update path more stable, and configuring error compensation dynamically tracks and corrects the information loss during compression to prevent the impact of error accumulation on model accuracy in long-term training, preserving the communication efficiency improvement brought by gradient compression, and solving the problems of slow convergence speed and accuracy decline caused by traditional compression methods.

[0094] In one embodiment of the present disclosure, momentum compensation is performed on the aggregated gradient to obtain a momentum-buffered gradient, including: configuring a momentum coefficient based on the number of completed training cycles; determining the momentum-buffered gradient based on the momentum coefficient, a historical momentum buffer value, and the aggregated gradient, wherein the historical momentum buffer value represents the cumulative influence of the update direction obtained in the completed training cycles.

[0095] In some embodiments, the historical momentum buffer value is a dynamic cumulative variable used to store historical information of the gradient update direction of the model in past training cycles. After each training cycle, the historical momentum buffer value is updated according to the following formula:

[0096] New momentum buffer value = momentum coefficient x old momentum buffer value + (1-momentum coefficient) x current aggregated gradient.

[0097] In some embodiments, the momentum coefficient is dynamically adjusted according to the number of completed training cycles. For example, a smaller coefficient is used at the beginning of training to allow rapid exploration, and the coefficient is increased at a later stage to stabilize the convergence direction. Then, by combining the coefficient, the historical momentum buffer value, which accumulates the gradient direction information of all previous training cycles, and the current aggregated gradient, a new momentum-buffered gradient is obtained by weighted calculation. The momentum coefficient controls the weight distribution of the historical update direction and the current gradient, and the number of training cycles serves as a basis for adjustment, so that the coefficient changes adaptively with the training process, ensuring that the model balances the exploration and convergence requirements at different stages.

[0098] In this embodiment, by dynamically adjusting the momentum coefficient according to the number of training cycles and combining the historical momentum buffer value and the current aggregated gradient to calculate a new momentum-buffered gradient, an adaptive gradient direction guide is formed, which is beneficial to prevent the problems of slow initial convergence or late oscillation caused by a fixed momentum coefficient, and also improves the optimization efficiency of the model in a complex optimization space.

[0099] In one embodiment of the present disclosure, error compensation is performed on the momentum-buffered gradient based on an error compensation mechanism to obtain a compensated gradient, including: obtaining a historical compensation buffer value obtained in the previous training cycle; superimposing the historical compensation buffer value on the momentum-buffered gradient to obtain the compensated gradient, wherein the compensated gradient is multiplied by a configured learning rate to obtain the historical compensation buffer value in the current training cycle.

[0100] In some embodiments, the historical compensation buffer value is a cumulative variable used to record and correct information loss in the gradient compression process. In each training cycle, this variable stores the cumulative error caused by gradient compression (such as sparsification or quantization) in all previous cycles.

[0101] In this embodiment, in the current cycle, the historical compensation buffer value obtained in the last training cycle is added to the current momentum buffer gradient to compensate for the gradient information lost in the previous compression operation; then, the compensated gradient (i.e., the compensation gradient) is multiplied by the configured learning rate, and the result is used as the new historical compensation buffer value for error correction in the next training cycle. This mechanism enables the historical compensation buffer value to continuously track and accumulate the error caused by gradient compression, thereby achieving dynamic correction of model parameter updates.

[0102] In one embodiment of the present disclosure, the local gradient of the trained model parameter is calculated, including: calculating the original gradient of the model parameter based on the forward propagation and the backward propagation operation; applying local momentum compensation to the original gradient to obtain the local gradient.

[0103] Wherein, the forward propagation refers to propagating input data and intermediate activation values, and the purpose is to obtain a prediction result, which is a forward calculation process from data to prediction; the backward propagation refers to propagating the gradient of the loss function on the parameter, and the purpose is to calculate the direction and amplitude of parameter update, which is a backward derivation process from error to parameter gradient.

[0104] In some embodiments, in the test environment, gradient data can also be generated in a simulated manner, specifically including: creating a Gradient instance with a dimension of HIDDEN_DIM X FEATURE_DIM, which is used to store the weight gradient matrix and the bias gradient vector; using a double loop to traverse each element of the weight gradient matrix and assigning a random Gaussian distribution value (mean value of 0 and standard deviation of 0.1) to it, which simulates the gradient distribution characteristics of the weight parameters in neural network training; assigning a random Gaussian distribution value (mean value of 0 and standard deviation of 0.01) to each element of the bias gradient vector, which reflects the characteristics that the bias parameters are usually more stable and have smaller update amplitude than the weight parameters in actual training; returning the Gradient object filled with random values for subsequent testing or verification.

[0105] In this embodiment, based on the training process of the neural network, the prediction value is generated by forward propagation, the error gradient is calculated by backward propagation, and a local momentum compensation mechanism is further introduced to integrate the historical gradient direction information into the current gradient calculation, which not only retains the response of the original gradient to the current sample, but also enhances the judgment ability of the global optimal direction through the historical accumulation effect.

[0106] In one embodiment of the present disclosure, the local momentum compensation is applied to the original gradient to obtain the local gradient, including: determining the local gradient based on the momentum coefficient, the historical momentum buffer value, and the original gradient.

[0107] In some embodiments, the momentum coefficient is used to control the weight distribution of the historical gradient and the current gradient, determining the degree of retention of historical information. The historical momentum buffer value stores the cumulative effect of the gradient in the previous training step, and records the historical direction of parameter update. The formula for calculating the local gradient is as follows:

[0108] Local gradient = momentum coefficient x historical momentum buffer value + (1-momentum coefficient) x original gradient

[0109] In this embodiment, in each training step, the weight distribution of the historical gradient and the current gradient is determined according to the preset momentum coefficient, and then the weighted sum of the historical momentum buffer value and the current original gradient is calculated to obtain the final local gradient. When the gradient directions of consecutive steps are consistent, the historical momentum buffer value will continuously accumulate and strengthen, accelerating the parameter update. When the gradient direction fluctuates, it is also beneficial to reduce the shock.

[0110] In one embodiment of the present disclosure, receiving the neighbor compressed gradient sent by the neighbor edge node comprises: receiving the neighbor compressed gradient and putting the neighbor compressed gradient into a blocking queue; and taking out the neighbor compressed gradient from the blocking queue based on a non-blocking processing mode.

[0111] In some embodiments, after the node receives the compressed gradient sent by the neighbor, it is immediately put into a blocking queue, such as Java's BlockingQueue. When the queue is full, the put operation will be blocked until the queue has space, ensuring that the memory usage is controllable.

[0112] In some embodiments, the main thread takes out the gradient from the queue in a non-blocking manner, such as the poll() method. If the queue is empty, it will return immediately, avoiding thread suspension and waiting. This design allows the main thread to continue executing other computing tasks (such as local model training) while waiting for the gradient, realizing the overlap of calculation and communication.

[0113] In some embodiments, the network IO and gradient processing logic are isolated by the queue, allowing data reception and model calculation to be performed in parallel. Even if the network fluctuation causes delay in the arrival of the gradient, the main thread can still continuously process the received gradient, improving the system throughput.

[0114] In this embodiment, the blocking queue is used to cache the gradient, solving the problem of mismatch between network delay and processing speed between nodes. Combined with the use of non-blocking processing mode, the main thread can continue to promote local training when there is no new gradient, realizing the overlap and parallelism of communication and calculation.

[0115] In one embodiment of the present disclosure, selecting neighbor edge nodes in the edge node cluster to build a distributed training topology comprises: obtaining a neighbor edge node list of the edge node cluster; selecting a plurality of neighbor edge nodes that meet distance requirements from the neighbor edge node list based on identification information and resource state information of each neighbor edge node in the neighbor edge node list; and establishing a communication connection with the plurality of neighbor edge nodes to build a distributed training topology and a communication thread pool.

[0116] The resource state information includes, for example, remaining power, CPU load, memory usage, real-time network bandwidth, and the like.

[0117] In some embodiments, an independent communication thread is allocated to each neighbor edge node to form a dedicated thread pool, parallelization of gradient transmission is achieved, and the size of the thread pool can be dynamically adjusted according to the computing power and network bandwidth of the edge node to prevent resource competition.

[0118] In this embodiment, by obtaining a neighbor edge node list of the edge node cluster, nodes that meet distance requirements are filtered according to neighbor edge node identification information, a communication connection is established, a distributed training topology and a communication thread pool are built, and an independent communication thread is allocated to each neighbor edge node to form a dedicated thread pool, which is conducive to reducing the delay caused by long-distance communication between nodes and achieving parallelization of gradient transmission and reasonable allocation of resources, thereby shortening the training time and improving the overall performance of model training.

[0119] As shown in FIG. 1, Figure 3 In one embodiment of the present disclosure, the model parameters include a weight matrix determined based on the input feature dimension and the hidden layer dimension of the model and a bias vector determined based on the hidden layer dimension, the compensation gradient includes a compensation weight gradient matrix corresponding to the dimension of the weight matrix and a compensation bias gradient vector corresponding to the bias vector, and updating the model parameters based on the compensation gradient comprises:

[0120] In step S302, a first update value corresponding to the compensation weight gradient matrix and the configured learning rate is determined, wherein the first update value is the product of the learning rate and each element in the compensation weight gradient matrix.

[0121] In step S304, each weight value in the weight matrix is updated based on the corresponding first update value to obtain an updated weight matrix.

[0122] In step S306, a second update value corresponding to the compensation bias gradient vector and the learning rate is determined, wherein the second update value is the product of the learning rate and each element in the compensation bias gradient vector.

[0123] Step S308, updating each bias value in the bias vector based on the corresponding second update value to obtain an updated bias vector, and obtaining the updated model parameters based on the updated weight matrix and the updated bias vector.

[0124] In this embodiment, the first update value is obtained by element-wise multiplication of the compensation weight gradient matrix and the learning rate, which is used to update the weight matrix, and the second update value is obtained by multiplication of the compensation bias gradient vector and the learning rate, which is used to update the bias vector, so as to finally obtain the updated model parameters, which not only retains the information integrity brought by the gradient compression and compensation mechanism, but also controls the update step through the learning rate, balancing the training stability and convergence speed.

[0125] As shown in FIG. 6, in one embodiment of the present disclosure, detecting that the model parameters satisfy the convergence condition of the model comprises: Figure 4

[0126] Step S402, collecting the multi-dimensional weight parameters of all nodes in the distributed training topology based on the updated weight matrix.

[0127] Step S404, extracting the weight values of all nodes in any dimension from the multi-dimensional weight parameters, and calculating the average value of all nodes in any dimension.

[0128] Step S406, calculating the variance of any dimension based on the weight value of each node in any dimension and the average value.

[0129] Step S408, determining the average variance of all nodes based on the variance of all dimensions.

[0130] Step S410, detecting that the average variance is reduced to less than or equal to the variance threshold, and determining that the convergence condition of the model is satisfied.

[0131] The variance threshold is set according to the edge node cluster size and the model accuracy requirement.

[0132] In some embodiments, when the distributed training converges, the model parameters of each node should tend to be consistent, which is manifested as the distribution discrete degree of the weight matrix in each dimension being reduced. By extracting the weight values of all nodes in each dimension, the average value and the variance are calculated to quantify the discrete degree of the parameter distribution. For example, in the i-th dimension, the weight value of node j is w j,i , the average value of all nodes in this dimension is μ i , and the variance is wherein N is the number of nodes.

[0133] ​In this embodiment, the multi-dimensional weight parameters of all nodes in the distributed training topology are collected, the weight value of each node in each dimension is extracted and the average value is calculated, and then the variance of each dimension is obtained, and then the average variance of all dimensions is calculated. Finally, whether the model converges is determined by detecting whether the average variance is reduced to a preset threshold. This realizes the use of parameter consistency between nodes as a convergence indicator, and reflects the degree of synchronization of the model parameters of each node in the distributed training by quantitatively analyzing the discrete degree of the weight distribution. Through multi-dimensional variance analysis, the convergence trend of the parameters between nodes in the distributed environment is effectively captured.

[0134] In one embodiment of the present disclosure, before performing periodic training of a distributed model in a distributed training topology, it also includes: initializing a weight gradient matrix and a bias gradient vector based on the input dimension and output dimension of the model, wherein the weight gradient matrix performs an initialization operation based on a random Gaussian distribution, and the initial value of the bias gradient vector is 0.

[0135] In this embodiment, the dimension of the weight gradient matrix is ​​[input dimension, output dimension], ensuring that the output of the correct dimension is obtained after the input feature is multiplied by the weight matrix. For example, if the input layer has m neurons and the hidden layer has n neurons, then the weight matrix W∈(m×n). The adaptation of Gaussian initialization and input dimension effectively alleviates the gradient vanishing problem in deep networks.

[0136] like Figure 5 As shown, according to another embodiment of the present disclosure, a distributed model training optimization method defines a main class named DECENTAlgorithm to implement a decentralized distributed training algorithm, including:

[0137] Step S502: Configure the configuration parameters related to the neural network.

[0138] Including but not limited to a feature dimension of 1000, a hidden layer dimension of 512, a learning rate of 0.01, a momentum coefficient of 0.9, and a gradient compression ratio of 10, that is, retaining 1 / 10 of the gradient elements.

[0139] Step S504: Initialize the weights and biases of the model according to the input dimension and the output dimension.

[0140] During initialization, a random Gaussian distribution is multiplied by 0.1 to generate the initial value, and the bias is initialized to 0.

[0141] Step S506: configure a gradient data structure Gradient class that stores the weight gradient matrix weightGrads and the bias gradient vector biasGrads.

[0142] Gradient class is used to store gradient information, including providing an add method to add the current gradient to the corresponding elements of another gradient (for aggregating multiple gradients), and providing a scale method to multiply all elements of the gradient by a coefficient (such as scaling the aggregated gradient).

[0143] Step S508, the compressed gradient is represented by the CompressedGradient class.

[0144] The class is used to store the compressed gradient, and implements a hybrid compression strategy, i.e. sparsification and quantization.

[0145] The specific steps of gradient compression include:

[0146] Receive the complete gradient and the compression ratio.

[0147] Calculate the total number of elements and the number of elements to be retained (total number of elements / compression ratio).

[0148] Use a priority queue (min heap) to filter out the Top-K gradient elements with the largest absolute value (K = number of retained elements): traverse all elements of the weight gradient, if the absolute value of the element is greater than the top element of the heap or the heap is not full, add it to the heap, and remove the top element (the smallest absolute value element) when the capacity is exceeded.

[0149] Store the indices (sparseWeightIndices, a two-dimensional array records [i, j] position) and values (quantizedWeightValues) of the filtered elements.

[0150] The bias gradient is not compressed and the original value (biasGrads) is directly stored.

[0151] In addition, the decompress method is used to decompress the compressed gradient into a complete gradient.

[0152] In some embodiments, the gradient element class (GradientElement class) is used to encapsulate the information of a single gradient element, including the row index i, column index j and value value of the element, and by implementing the comparison interface, the elements are sorted in ascending order of absolute value, which is used for filtering Top-K elements in the priority queue.

[0153] In some embodiments, the DECENTNode class is used to define a distributed node, which implements the Runnable interface and represents a node in distributed training, including node ID, neighbor edge node list, model parameters, gradient cache and other attributes.

[0154] The node attributes include: node ID (nodeId), neighbor edge node list (peers), local model (model, a NeuralLayer instance), local gradient (localGrad), momentum buffer (momentumBuffer), compensation buffer (compensationBuffer), communication thread pool (commExecutor), blocking queue receiving gradient (gradQueue), running state flag (running).

[0155] Step S510, set node ID, initialize model, initialize local gradient, momentum buffer, compensation buffer and neighbor list, create single-thread communication pool and blocking queue.

[0156] The model input dimension is initialized to 1000 and the output dimension is initialized to 512. The local gradient, momentum buffer and compensation buffer are initialized with dimensions matching the model. The neighbor list is initialized, and the single-thread communication pool and blocking queue are created.

[0157] Step S512, select 3 nodes closest to the current node ID from other nodes as neighbor edge nodes to join the peers list to build a distributed training topology.

[0158] In some embodiments, the run method of the Runnable interface is implemented to simulate the training loop of the node (the loop continues until running is false), which includes:

[0159] The computeMockGradient method is called to simulate the calculation of the local gradient.

[0160] The applyMomentumAndCompensation method is called to apply the momentum and compensation mechanism to the local gradient.

[0161] The CompressedGradient constructor method is called to compress the gradient (processed at a compression ratio of 10).

[0162] The sendGradientsToPeers method is called to asynchronously send the compressed gradient to all neighbor edge nodes.

[0163] The processReceivedGradients method is called to process the received neighbor gradient.

[0164] The updateModelParameters method is called to update the model parameters.

[0165] The batch processing interval is simulated, and a random sleep of 100-150 milliseconds is performed.

[0166] Step S514, calculate the gradient by forward / backward propagation.

[0167] In some implementations, in the test environment, a new gradient instance can also be generated, each element of the weight gradient is generated by a random Gaussian distribution (scaled by 0.1), and each element of the bias gradient is generated by a random Gaussian distribution (scaled by 0.01), to simulate the real gradient calculated by forward / backward propagation.

[0168] Step S516, apply the momentum mechanism and error compensation to the input gradient in turn.

[0169] In some embodiments, the momentum mechanism refers to traversing each element of the weight gradient, updating the momentum buffer, momentum buffer = 0.9 x historical momentum buffer + current gradient, and assigning the value of the momentum buffer back to the current gradient, and the processing logic of the bias gradient is the same.

[0170] In some embodiments, error compensation refers to adding the value of the historical compensation buffer to the current gradient for each element of the weight gradient, and then updating the compensation buffer, new compensation buffer = -learning rate x current gradient, and the processing logic of the bias gradient is the same.

[0171] Step S518, send the gradient to the neighbor edge node.

[0172] In some embodiments, an asynchronous task is submitted to the communication thread pool, all neighbor edge nodes are traversed, and the neighbor's receiveGradient method is called to send the compressed gradient (in practice, network communication, here the method is called directly to simulate).

[0173] Step S520, receive compressed gradients from other nodes and put them into the blocking queue gradQueue, and block if the queue is full.

[0174] Step S522, non-blockingly take out all received compressed gradients from gradQueue, convert them into a list, traverse each compressed gradient, call the decompress method to decompress into a complete gradient, and aggregate it with the local gradient.

[0175] Among them, the aggregation processing method includes adding and then multiplying by 0.5, that is, average processing.

[0176] Step S524, traverse the weights and biases of the model, and update the parameters according to the gradient descent rule.

[0177] Among them, the updated weight = weight - learning rate x weight gradient, and the updated bias = bias - learning rate x bias gradient.

[0178] Reset the local gradient to a new empty gradient instance.

[0179] Further, the weight matrix of the model is converted into a one-dimensional array (spliced by row) for subsequent calculation of the convergence between nodes.

[0180] In some embodiments, the distributed training cluster DECENTCluster class is used to manage all nodes of the distributed training and coordinate the training process.

[0181] In some embodiments, the number of nodes is received, the node list is initialized (DECENTNode instances are created according to the number), the connect method of each node is called to establish neighbor connections (each node is connected to three nearest nodes), and a fixed thread pool with the same number of nodes is initialized.

[0182] In some embodiments, all nodes are traversed, the nodes are submitted to the thread pool (the run method of the node is executed in parallel), and the specified training duration (seconds) is slept, and after the sleep ends, the shutdown method is called to close the cluster.

[0183] In some embodiments, all nodes are traversed, the shutdown method of each node is called, and then the node thread pool is closed.

[0184] In some embodiments, the flattened weights of all nodes are collected (through the getModelWeightsFlattened method).

[0185] In step S526, the average variance of all node weight parameters is calculated, the variance of the squared difference between each node value and the average value is calculated, and finally the average value of the variance of all parameter dimensions is calculated.

[0186] The smaller the value of the average variance, the closer the parameters of each node are, and the better the convergence.

[0187] In this embodiment, by configuring a decentralized distributed training framework, the node dynamic connection of neighbors, local gradient calculation, momentum and compensation optimization, gradient mixing compression (Top-K sparsification), inter-node gradient communication and aggregation, model parameter update, and convergence evaluation through parameter variance are realized based on the training framework.

[0188] It should be noted that the above-described figures are only schematic representations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to limit the purpose. It is easy to understand that the processes shown in the above-described figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0189] The distributed model training optimization device 600 according to the embodiments of the present disclosure will be described below with reference to Figure 6 . Figure 6The illustrated distributed model training optimization apparatus 600 is merely one example, and should not bring any limitation to the function and usage range of the embodiments of the present disclosure.

[0190] The distributed model training optimization apparatus 600 is in the form of a hardware module. The components of the distributed model training optimization apparatus 600 can include, but are not limited to: a construction module 602 for constructing a distributed training topology by a neighbor edge node in a set of edge nodes; a training module 604 for performing periodic training of a distributed model in the distributed training topology, wherein, after completing one training period, a local gradient of a model parameter obtained by training is calculated, neighbor compressed gradients sent by neighbor edge nodes are received, the neighbor compressed gradients are decompressed to obtain neighbor gradients of the model parameter obtained by training of the neighbor edge nodes, an aggregation operation is performed on the local gradient and the neighbor gradient to obtain an aggregated gradient, momentum compensation is performed on the aggregated gradient to obtain a compensated gradient, and the model parameter is updated based on the compensated gradient; and a detection module 606 for detecting that the model parameter satisfies a convergence condition of the model, and stopping the periodic training.

[0191] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied as a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0192] The electronic device 700 according to this embodiment of the present disclosure will be described below with reference to Figure 7 Figure 7 The illustrated electronic device 700 is merely one example, and should not bring any limitation to the function and usage range of the embodiments of the present disclosure.

[0193] As Figure 7 illustrated, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 can include, but are not limited to: the above-mentioned at least one processing unit 710, the above-mentioned at least one storage unit 720, and a bus 730 connecting different system components, including the storage unit 720 and the processing unit 710.

[0194] The storage unit stores program code which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification. For example, the processing unit 710 can perform the schemes as Figure 1 described.

[0195] ​The storage unit 720 can include a readable medium in the form of volatile storage such as random access memory (RAM) 7201 and / or cache 7202, and also can include a non-volatile storage such as read only memory (ROM) 7203.

[0196] The storage unit 720 also can include a program / utility 7204 having a set of program modules 7205 such as an operating system, one or more application programs, other program modules, and program data, each of which can give the electronic device 700 its functionality, at least in part. The program modules 7205 can include, but are not limited to, a network environment implementation, either alone or in some combination.

[0197] The bus 730 can represent one or more of several types of bus structures, including a storage bus or bus controller, a peripheral bus, an accelerated graphics port, an electronic device 700, or a local bus using any of a variety of bus architectures.

[0198] The electronic device 700 also can communicate with one or more external devices 770 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices such as printers, scanners, monitors, etc.; and / or various devices to enable a user to interact with the electronic device 700 in various ways (e.g., a digital camera, a digital video camera, a sound card, video card, etc.). Such interaction can occur through input / output (I / O) interface(s) 750. Still yet, the electronic device 700 can communicate with one or more networks, such as one or more local area networks (LANs), one or more wide area networks (WANs) and / or the Internet, through a network adapter 760. As depicted, the network adapter 760 communicates with the other components of the electronic device 700 through the bus 730. It should be appreciated that the network adapter 760 and / or the other hardware and / or software components depicted in FIG. 7 can be implemented with various hardware and / or software components, including but not limited to a microcode, a device driver, redundant processing unit, external disk drive array, RAID system, tape drive, and data backup storage system, etc.

[0199] From the above description of the embodiments, those skilled in the art will readily perceive that the example embodiments described herein can be implemented by software and / or by software in combination with the requisite hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the methods according to the embodiments of the present disclosure.

[0200] In exemplary embodiments of the present disclosure, a computer readable storage medium having stored thereon a program product capable of implementing the above-described methods of the present specification is also provided. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product including a program code that, when run on an electronic device, causes the electronic device to perform the steps described in the above "Exemplary Methods" section according to various exemplary embodiments of the present disclosure.

[0201] A program product for implementing the above-described methods according to embodiments of the present disclosure can take the form of a portable compact disc read-only memory (CD-ROM) and include a program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in the present document, a readable storage medium can be any tangible medium that includes or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0202] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0203] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the readable program code is embodied. Such propagated data signal can take multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a readable storage medium and that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0204] The program code included on the readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, and the like, or any suitable combination thereof.

[0205] Program code to implement an operation of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0206] It should be noted that, although several modules or units of the device for action execution are mentioned in the foregoing detailed description, such a division is not mandatory. Indeed, according to an embodiment of the present disclosure, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into a plurality of modules or units.

[0207] Moreover, although the various steps of the methods of the present disclosure are described in a particular order in the figures, this is not required or implied, nor is it required that all of the steps shown be performed in order to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into a single step, a single step can be broken into multiple steps, and / or the like.

[0208] From the above description of the embodiments, those skilled in the art will readily perceive that the example embodiments described herein can be implemented by software and / or by hardware and / or by a combination of software and hardware. The embodiments according to the present disclosure can be realized in the form of a software product including one or more instructions that are stored in a non-transitory storage medium (e.g., a CD-ROM, a USB memory, a hard disk, or the like) or a network, and are executable by a computing device (e.g., a personal computer, a server, a mobile terminal, or the like). The software product can be distributed over a network, computer-readable storage medium, or any other type of non-transitory media.

[0209] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A distributed model training optimization method, characterized in that: include: Select neighbor edge nodes in the edge node cluster to build a distributed training topology; Performing periodic training of the distributed model in the distributed training topology, wherein upon completing a training cycle, local gradients of the model parameters obtained through training are calculated, neighbor compressed gradients sent by the neighbor edge nodes are received, the neighbor compressed gradients are decompressed to obtain neighbor gradients of the model parameters obtained through training calculated by the neighbor edge nodes, the local gradients and the neighbor gradients are aggregated to obtain aggregated gradients, momentum compensation is performed on the aggregated gradients to obtain compensated gradients, and the model parameters are updated based on the compensated gradients; It is detected that the model parameters meet the convergence conditions of the model, and the periodic training is stopped.

2. The distributed model training optimization method according to claim 1, characterized in that: Also includes: Performing a gradient compression operation on the local gradient to obtain a local compressed gradient; The local compressed gradient is sent to at least one of the neighbor edge nodes based on asynchronous communication.

3. The distributed model training optimization method according to claim 2, characterized in that: The gradient includes a weight gradient matrix and a bias gradient vector, and a gradient compression operation is performed on the local gradient to obtain a local compressed gradient, including: Calculate the number of elements that need to be retained in the local weight gradient matrix based on the gradient compression ratio configured by the network bandwidth; Arrange all elements in the local gradient in descending order based on absolute values ​​to obtain an arrangement sequence; retaining the number of gradient elements of the elements in the first column of the permutation sequence, and setting the other gradient elements to zero to generate a local sparse gradient; The local sparse gradient is quantized to obtain the local compressed gradient.

4. The distributed model training optimization method according to claim 2, characterized in that: Sending the local compressed gradient to at least one of the neighboring edge nodes based on asynchronous communication includes: Submit an asynchronous task to the communication thread pool of the distributed training topology, wherein the asynchronous task is used to traverse the neighbor edge node list of the distributed training topology to send the local compressed gradient to each neighbor edge node through the communication thread pool, where the communication thread pool is a single-threaded executor.

5. The distributed model training optimization method according to claim 1, characterized in that: Decompressing the neighbor compressed gradient to obtain the neighbor gradient of the model parameter calculated and trained by the neighbor edge node includes: extracting neighbor sparse gradients, quantization metadata, and original dimension information from the neighbor compressed gradients; Initialize an all-zero gradient matrix based on the original dimension information; Performing inverse quantization processing on the neighbor sparse gradient based on the quantization metadata to obtain floating-point values ​​of non-zero gradient elements; Based on the index information of the neighbor sparse gradient, the floating point value is filled into the corresponding position of the all-zero gradient matrix to obtain the neighbor gradient.

6. The distributed model training optimization method according to claim 1, characterized in that: Performing momentum compensation on the aggregation gradient to obtain a compensation gradient comprises: performing momentum compensation on the polymerization gradient to obtain a momentum buffer gradient; Error compensation is performed on the momentum buffer gradient based on an error compensation mechanism to obtain the compensation gradient.

7. The distributed model training optimization method according to claim 6, characterized in that: Performing momentum compensation on the aggregation gradient to obtain a momentum buffer gradient comprises: configuring a momentum factor based on a number of completed training cycles; The momentum buffer gradient is determined based on the momentum coefficient, a historical momentum buffer value, and the aggregate gradient, wherein the historical momentum buffer value represents a cumulative impact of the update direction obtained from the completed training cycle.

8. The distributed model training optimization method according to claim 6, characterized in that: Performing error compensation on the momentum buffer gradient based on an error compensation mechanism to obtain the compensation gradient includes: Obtain the historical compensation buffer value obtained in the previous training cycle; The historical compensation buffer value is superimposed on the momentum buffer gradient to obtain the compensation gradient, The compensation gradient is multiplied by the configured learning rate to obtain the historical compensation buffer value in the current training cycle.

9. The distributed model training optimization method according to claim 7, characterized in that: Calculate the local gradients of the trained model parameters, including: Calculating raw gradients of the model parameters based on forward and backpropagation operations; A local momentum compensation is applied to the original gradient to obtain the local gradient.

10. The distributed model training optimization method according to claim 9, characterized in that: Applying local momentum compensation to the original gradient to obtain the local gradient includes: The local gradient is determined based on the momentum coefficient, the historical momentum buffer value, and the raw gradient.

11. The distributed model training optimization method according to claim 1, characterized in that: Receiving a neighbor compression gradient sent by the neighbor edge node includes: receiving the neighbor compressed gradient and placing the neighbor compressed gradient into a blocking queue; and The neighbor compressed gradient is retrieved from the blocking queue based on a non-blocking processing manner.

12. The distributed model training optimization method according to claim 4, characterized in that: Select neighbor edge nodes in the edge node cluster to build a distributed training topology, including: Obtaining a list of neighbor edge nodes of the edge node cluster; Based on the identification information and resource status information of each neighbor edge node in the neighbor edge node list, multiple neighbor edge nodes that meet the distance requirements are selected from the neighbor edge node list, and communication connections are established with the multiple neighbor edge nodes to construct the distributed training topology and the communication thread pool.

13. The distributed model training optimization method according to claim 1, characterized in that: The model parameters include a weight matrix determined based on the input feature dimension and hidden layer dimension of the model and a bias vector determined based on the hidden layer dimension, the compensation gradient includes a compensation weight gradient matrix corresponding to the weight matrix dimension and a compensation bias gradient vector corresponding to the bias vector, and updating the model parameters based on the compensation gradient includes: Determining a corresponding first update value based on the compensation weight gradient matrix and a configured learning rate, wherein the first update value is a product of the learning rate and each element in the compensation weight gradient matrix; updating each weight value in the weight matrix based on the corresponding first update value to obtain the updated weight matrix; Determining a corresponding second update value based on the compensation bias gradient vector and the learning rate, wherein the second update value is a product of the learning rate and each element in the compensation bias gradient vector; Each bias value in the bias vector is updated based on the corresponding second update value to obtain the updated bias vector, and the updated model parameters are obtained based on the updated weight matrix and the updated bias vector.

14. The distributed model training optimization method according to claim 13, characterized in that: Detecting that the model parameters meet the convergence conditions of the model includes: Collect multidimensional weight parameters of all nodes in the distributed training topology based on the updated weight matrix; Extracting the weight values ​​of all the nodes in any dimension from the multidimensional weight parameters, and calculating the average value of all the nodes in any dimension; Calculating the variance of any dimension based on the weight value of each node in any dimension and the average value; Determining an average variance of all nodes based on the variances of all the dimensions; It is detected that the average variance decreases to be less than or equal to a variance threshold, and it is determined that a convergence condition of the model is satisfied.

15. The distributed model training optimization method according to claim 1, characterized in that: Before performing periodic training of the distributed model in the distributed training topology, the method further includes: Initialize a weight gradient matrix and a bias gradient vector based on the input dimension and the output dimension of the model, wherein the weight gradient matrix performs an initialization operation based on a random Gaussian distribution, and the initial value of the bias gradient vector is 0.

16. A distributed model training optimization device, characterized in that: include: In the construction module, users select neighbor edge nodes in the edge node cluster to build a distributed training topology; a training module configured to perform periodic training of the distributed model in the distributed training topology, wherein upon completing a training cycle, local gradients of the model parameters obtained through training are calculated, compressed neighbor gradients sent by the neighbor edge nodes are received, the compressed neighbor gradients are decompressed to obtain neighbor gradients of the model parameters obtained through training calculated by the neighbor edge nodes, the local gradients and the neighbor gradients are aggregated to obtain aggregated gradients, momentum compensation is performed on the aggregated gradients to obtain compensated gradients, and the model parameters are updated based on the compensated gradients; The detection module is used to detect that the model parameters meet the convergence conditions of the model and stop the periodic training.

17. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the distributed model training optimization method according to any one of claims 1 to 15 by executing the executable instructions.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed model training optimization method according to any one of claims 1 to 15 is implemented.

19. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed model training optimization method according to any one of claims 1 to 15 is implemented.

Images