Distributed training method and device, equipment and storage medium

Abstract

The invention discloses a distributed training method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. Comprising the following steps: a computing node reads training data, generates a local gradient in combination with a current model parameter, and sends the local gradient and a timestamp during sending to a central aggregation server; the server receives the local gradients from the computing nodes and the timestamps during sending; under the condition that the number of the computational nodes receiving the local gradient is larger than a preset aggregation threshold value, the weight of each computational node is calculated, gradient aggregation is carried out, and a global gradient is obtained; calculating a target learning rate and a target model parameter based on the global gradient; respectively sending the target learning rate and the target model parameter to each computing node; and the node receives the target learning rate and the target model parameter for iterative updating. By adopting an asynchronous communication mechanism and considering the actual performance of each node, the distributed training method which can flexibly adjust each node, is higher in training efficiency and improves the resource utilization rate is provided.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a distributed training method, apparatus, device and storage medium. Background Art

[0002] For artificial intelligence (AI) training, in traditional synchronous distributed training methods, training tasks are assigned to multiple nodes. Each node is equipped with corresponding computing resources (such as a GPU) and is responsible for performing forward and backward propagation calculations on the assigned portion of data to generate local gradients. During each iteration, all nodes must complete their own local gradient calculations and then send the local gradients to the central server through a synchronous communication mechanism. After receiving the local gradients of all nodes, the central server aggregates the gradients to obtain the global gradient, and then distributes the global gradient to each node. Each node updates the local model parameters based on the global gradient before entering the next iteration.

[0003] During this process, computation and communication between nodes are strictly synchronized to ensure that model parameters remain consistent after each iteration. This also results in gradient synchronization requiring the slowest node to complete computation before it can proceed. This causes fast nodes to remain idle for extended periods, significantly reducing overall computing resource utilization. This also slows down iteration speed and reduces efficiency. Furthermore, gradient aggregation fails to account for performance differences between nodes, lacking the flexibility to adjust performance for each node.

[0004] Therefore, how to propose an artificial intelligence training method that can flexibly adjust each node, improve training efficiency, and improve resource utilization has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present application provides a distributed training method, apparatus, device and storage medium, the main purpose of which is to propose a distributed AI training method with asynchronous communication to improve flexibility, training efficiency and resource utilization.

[0006] In a first aspect, the present application provides a distributed training method, comprising:

[0007] Read the training data and generate local gradients based on the current model parameters;

[0008] Sending the local gradient and the timestamp of sending to a central aggregation server;

[0009] Receive the target learning rate and target model parameters sent by the central aggregation server, and perform iterative update based on the target learning rate and the target model parameters.

[0010] Optionally, the reading of training data and generating local gradients in combination with current model parameters includes: standardizing the training data and dividing the data blocks according to a preset batch size; calculating the parameter gradients of each layer through forward propagation and back propagation based on the data blocks and the current model parameters; and integrating the parameter gradients of each layer to obtain the local gradient.

[0011] Optionally, after receiving the target learning rate and target model parameters sent by the central aggregation server and performing iterative updates based on the target learning rate and the target model parameters, the method further includes: calculating the model accuracy after the number of iterative updates reaches a first preset number; and stopping training the model when the model accuracy has not been continuously improved or after the number of iterative updates reaches a second preset number.

[0012] In a second aspect, the present application provides a distributed training method, comprising:

[0013] Receive local gradients from each computing node and the timestamp when they were sent;

[0014] When the number of computing nodes receiving local gradients is greater than the preset aggregation threshold, the weights of each computing node are calculated and gradient aggregation is performed to obtain the global gradient;

[0015] Calculating a target learning rate and target model parameters based on the global gradient;

[0016] The target learning rate and the target model parameters are sent to each computing node respectively, so that each computing node performs iterative update according to the target learning rate and the target model parameters.

[0017] Optionally, when the number of computing nodes receiving local gradients is greater than a preset aggregation threshold, the weight of each computing node is calculated and gradient aggregation is performed to obtain a global gradient, including: when the number of computing nodes receiving local gradients is greater than the preset aggregation threshold, obtaining an evaluation index for each computing node; the evaluation index is used to characterize the performance of the computing node; based on the evaluation index of each computing node, the weight of each computing node is calculated; and gradient aggregation is performed according to the local gradients sent by each computing node and the weight of each computing node to obtain the global gradient.

[0018] Optionally, calculating the target learning rate and target model parameters based on the global gradient includes: obtaining the global gradient for a preset time period or a preset number of times, and the global gradient is based on the calculated variance; adjusting the initial learning rate according to the comparison result of the variance and a preset variance threshold to obtain the target learning rate; and determining the target model parameters based on the global gradient and the target learning rate.

[0019] In a third aspect, the present application provides a distributed training device, comprising:

[0020] A generation unit is configured to read the training data and generate local gradients based on the current model parameters;

[0021] a sending unit, configured to send the local gradient and a timestamp of sending to a central aggregation server;

[0022] An updating unit is configured to receive a target learning rate and target model parameters sent by the central aggregation server, and perform iterative updating based on the target learning rate and the target model parameters.

[0023] In a fourth aspect, the present application provides a distributed training device, comprising:

[0024] A receiving unit, configured to receive local gradients and a timestamp of sending from each computing node;

[0025] an aggregation unit configured to calculate the weight of each computing node and perform gradient aggregation to obtain a global gradient when the number of computing nodes receiving the local gradient is greater than a preset aggregation threshold;

[0026] A computing unit configured to calculate a target learning rate and target model parameters based on the global gradient;

[0027] The distribution unit is configured to send the target learning rate and the target model parameters to each computing node respectively, so that each computing node performs iterative update according to the target learning rate and the target model parameters.

[0028] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed training method described in the first aspect or the second aspect.

[0029] In a sixth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the distributed training method described in the first aspect or the second aspect.

[0030] By means of the above technical solution, the present application provides a distributed training method, apparatus, device and storage medium, in which the computing node first reads the training data and generates a local gradient in combination with the current model parameters, and sends the local gradient and the timestamp when sending to the central aggregation server; the server receives the local gradient and the timestamp when sending from each computing node; when the number of computing nodes receiving the local gradient is greater than the preset aggregation threshold, the weight of each computing node is calculated and the gradient aggregation is performed to obtain the global gradient; based on the global gradient, the target learning rate and target model parameters are calculated; the target learning rate and target model parameters are sent to each computing node respectively; the node receives the target learning rate and target model parameters for iterative update. Compared with the related art, first of all, the present application no longer sends local gradients to the server through a synchronous communication mechanism, but adopts an asynchronous communication method for nodes to send local gradients at any time, that is, different computing nodes do not have to send data at the same time; gradient aggregation can be performed when the number of computing nodes whose local gradients are received by the server is greater than a preset aggregation threshold, which solves the problem of low training efficiency and low resource utilization caused by synchronous communication in the existing method; in addition, gradient aggregation is performed based on the weight of each node, considering the performance of each node, and adjusting the model parameters and the learning rate of each node, so that the actual situation of each node can be dynamically adjusted during the training process, thereby improving the flexibility of training.

[0031] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 A schematic diagram of the interactive process of a distributed training method provided in an embodiment of the present application is shown;

[0034] Figure 2 The following is a schematic diagram showing the overall process of a distributed training method provided by an embodiment of the present application;

[0035] Figure 3 A schematic diagram of a flow chart of a calculation unit calculating a local gradient according to an embodiment of the present application is shown;

[0036] Figure 4 A schematic diagram of a process for performing double verification by a central aggregation server provided in an embodiment of the present application is shown;

[0037] Figure 5 A schematic diagram of a process for calculating weights and gradient aggregation by a central aggregation server provided in an embodiment of the present application is shown;

[0038] Figure 6 A schematic structural diagram of a distributed training device provided in an embodiment of the present application is shown;

[0039] Figure 7 A structural diagram of another distributed training device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0041] Based on the aforementioned background technology, current AI training typically uses a synchronous communication mechanism. That is, during each iteration, all nodes must complete their local gradient calculations and then send the local gradients to a central server via the synchronous communication mechanism. The central server aggregates the gradients and then distributes them to each node to update the local model parameters. This has the following drawbacks:

[0042] (1) Serious waste of computing resources: Since all nodes are required to wait for the slowest node to complete calculation before gradient synchronization in each iteration, the nodes with fast computing speed are idle for a long time, which greatly reduces the utilization of overall computing resources.

[0043] (2) Low training efficiency: Limited by the computing progress of the slowest node, the iteration speed of the entire training process is slowed down, which significantly increases the training time. This inefficiency problem is particularly prominent in large-scale distributed training scenarios.

[0044] (3) Lack of flexibility: Fixed weights are used for gradient aggregation without considering the differences in computing power and communication performance of different nodes, resulting in unsatisfactory aggregation effects. At the same time, the learning rate is fixed and cannot be dynamically adjusted according to the actual situation during training, which affects training efficiency.

[0045] To solve at least one of the above technical problems, this embodiment proposes a distributed training method, which can be applied to a distributed training device or electronic device. The device or electronic device can be installed or integrated into an AI training system, which includes a central aggregation server and multiple computing nodes, and can execute any of the distributed training methods mentioned below during operation.

[0046] In one feasible implementation, the above-mentioned AI training system includes a GPU node module and a central aggregation server module. As a feasible implementation, the following configuration may be specifically adopted:

[0047] GPU node module, each GPU node serves as a computing node for distributed training, also known as a computing terminal. Each computing node contains a computing submodule, a communication submodule, a local storage submodule, and a status monitoring submodule. Specifically:

[0048] Computing submodule: Equipped with a deep learning framework (such as TensorFlow), after receiving training data from the local storage submodule, it loads the current local model parameters to perform forward propagation calculations, generates prediction values ​​and loss function values, and then calculates the local gradients of each model parameter through a backpropagation algorithm (such as the chain rule), and supports batch processing.

[0049] Communication submodule: It has a built-in asynchronous communication protocol stack, establishes a connection with the central aggregation server, sends local gradients and timestamps in real time, listens to the server's parameter update push, and triggers local model updates after receiving global model parameters.

[0050] Local storage submodule: Partitioned storage of allocated training data (which can be divided by data category or time series), current local model parameter snapshots, and historical gradient records (retaining the calculation results of the most recent preset number of times), supporting data compression and fast reading.

[0051] Status monitoring submodule: collects real-time node GPU utilization (refreshed once per second), computing time (single batch forward + back propagation time), network sending / receiving rate and other indicators, and sends them to the dynamic weight adjustment submodule of the central aggregation server at regular intervals.

[0052] The central aggregation server module in the AI ​​training system serves as the global coordination center and includes a receiving and recording submodule, a dynamic weight adjustment submodule, a gradient aggregation submodule, an adaptive learning rate adjustment submodule, a parameter distribution submodule, a global storage submodule, and a cluster management submodule. Specifically:

[0053] Receiving and recording submodule: Deploys a multi-threaded listening service to receive local gradient data from each computing node in parallel, parses the data packet to obtain the gradient value, timestamp, and node ID, and stores it in the gradient information table (a structured database table containing the following fields: sequence number, node ID, timestamp, gradient vector, and receiving status). It can also mark abnormal data packets (such as gradient dimension mismatch and timestamp duplication) and trigger a retransmission mechanism.

[0054] Dynamic weight adjustment submodule: This module connects to the status monitoring submodule of the computing nodes, stores the historical computing power (average TFLOPS in the past 10 minutes) and communication delay (average round-trip time of the past 10 transmissions) of each node, calculates the dynamic weight periodically according to the preset algorithm (every time N / 2 gradients are received, N is the total number of nodes), generates a weight table, and pushes it to the gradient aggregation submodule.

[0055] Gradient aggregation submodule: Reads valid gradient data from the receiving and recording submodule, sorts them by timestamp, and calls the dynamic weight table to perform weighted summation on the gradient vector to generate a global gradient. At the same time, it calculates the aggregation error (the average deviation between the gradient of each node and the global gradient). When the aggregation error exceeds the preset aggregation threshold, it triggers weight recalculation.

[0056] Adaptive learning rate adjustment submodule: obtains the historical global gradient sequence from the gradient aggregation submodule, calculates statistical features such as the gradient change rate and variance, combines the model verification accuracy of the global storage submodule, generates a new learning rate according to the adjustment strategy, and passes it to the parameter distribution submodule.

[0057] Parameter distribution submodule: Receives updated global model parameters, uses incremental transmission (only sends the changed parameter blocks), pushes them to all computing nodes through a broadcast protocol (such as UDP multicast), records the reception confirmation status of each node, and performs unicast retransmission for nodes that have timed out and have not confirmed.

[0058] Global storage submodule: Saves the complete historical version of global model parameters (named by iteration number), dynamic weight records of all nodes, gradient aggregation logs, and validation set performance indicators (accuracy, loss value), and supports data backup and backtracking query.

[0059] Cluster management submodule: monitors the online status of each computing node. When a node is offline (no response for three consecutive heartbeats), it automatically assigns its training tasks to other nodes and adjusts the node set for dynamic weight calculation.

[0060] In combination with the above system configuration, the distributed training method of the asynchronous communication mechanism proposed in this embodiment is described below. Figure 1 FIG. 1 shows a schematic diagram of the interactive process of the distributed training method proposed in this embodiment, which includes:

[0061] S101, the computing node reads the training data and generates a local gradient based on the current model parameters;

[0062] Each compute node reads training data from the local storage submodule and generates local gradients based on the current model parameters. The current model parameters refer to the local model parameters before the current iteration, and these parameters may be adjusted after each model iteration. The training data can also be pre-processed and divided into batches of data blocks.

[0063] S102, the computing node sends the local gradient and timestamp to the central aggregation server;

[0064] After calculating the local gradient, the communication submodule obtains the system timestamp and packages the data to send to the central aggregation server. After sending, the computation submodule loads the next batch of data blocks and continues the computation, achieving parallel communication and computation.

[0065] S103, the central aggregation server receives the local gradients and the timestamp of sending from each computing node;

[0066] S104: When the number of computing nodes receiving local gradients is greater than a preset aggregation threshold, the central aggregation server calculates the weight of each computing node and performs gradient aggregation to obtain a global gradient.

[0067] After receiving the local gradient, the central aggregation server will calculate the number of nodes that have received the local gradient. If the number of computing nodes that have received the local gradient is greater than the preset aggregation threshold, it will calculate the weight of each computing node and perform gradient aggregation. Figure 1 Only two computing nodes are shown as examples, and the remaining nodes are omitted. For example, if there are 8 computing nodes, the preset aggregation threshold K can be set to 6 (75% of the total). When local gradients are received from more than 6 computing nodes, the weights of each node can be further calculated and gradient aggregation can be performed.

[0068] S105, the central aggregation server calculates the target learning rate and target model parameters based on the global gradient;

[0069] The central aggregation server then calculates the target learning rate and target model parameters based on the global gradient. The target learning rate is used to adjust the learning efficiency of the computing node, and the target model parameters are used to update the local model parameters of each node after this iteration.

[0070] S106, the central aggregation server sends the target learning rate and target model parameters to each computing node respectively;

[0071] S107, the computing node performs iterative updates based on the target learning rate and target model parameters.

[0072] In this embodiment, first, local gradients are no longer sent to the server through a synchronous communication mechanism. Instead, nodes use asynchronous communication to send local gradients at any time. That is, different computing nodes do not need to send data at the same time. Gradient aggregation can be performed when the number of computing nodes whose local gradients are received by the server is greater than a preset aggregation threshold. This solves the problems of low training efficiency and low resource utilization caused by synchronous communication in existing methods. In addition, gradient aggregation is performed based on the weight of each node, considering the performance of each node, and adjusting the model parameters and the learning rate of each node. This allows dynamic adjustment to the actual situation of each node during training, thereby improving training flexibility.

[0073] As shown in 2, a schematic diagram of the overall process of a distributed training method provided by an embodiment of the present application is shown. For the initialization step, a specific application configuration is given here for easy understanding. System initialization and cluster configuration include: starting the cluster management submodule of the central aggregation server, and reading the configuration file to set key parameters. In the embodiment of the present application, the number of GPU nodes N is set to 8, the current model parameters are θ0, and the He normal distribution is initialized (such as the convolutional layer parameters of ResNet50), the total number of training iterations T = 10000, the batch size B = 64, the dynamic weight adjustment coefficient α = 0.6 (the optimal value determined by comparative experiments), the initial learning rate = 0.01, the gradient aggregation threshold K = 6 (75% of the total number of nodes), the aggregation error threshold σ = 0.2, the gradient variance threshold σ 2 high=0.01、σ 2 low = 0.001. Pre-training preparation steps: The global storage submodule creates a version repository, labels θ0 as version v0, and stores it. The cluster management submodule sends initialization instructions to the eight GPU nodes. Upon receiving these instructions, the local storage submodule loads the assigned ImageNet training subset (125,000 images per node). The communication submodule establishes a persistent TCP connection with the server (port 5000). The status monitoring submodule initiates metric collection (sampling frequency 1Hz). The computation submodule loads θ0 into GPU memory. These parameters will be used one by one in the subsequent steps. The following section provides a detailed explanation of each step in the training method to facilitate understanding of the overall solution.

[0074] Optionally, the training data is read and combined with the current model parameters to generate local gradients, including: normalizing the training data and dividing the data into data blocks according to a preset batch size; calculating the parameter gradients of each layer through forward propagation and backpropagation based on the data blocks and the current model parameters; and integrating the parameter gradients of each layer to obtain the local gradient.

[0075] In this embodiment, before calculating the local gradient, each computing node can divide the training data into data blocks according to the preset batch size, so that the data processing process of each computing node is more refined, so that even the computing nodes with slower calculation progress will not be unable to send data for a long time due to excessive data volume. First, the local storage submodule of each computing node performs preprocessing on the training data: for example, the image training data is resized to 224×224 pixels, standardized to the interval [-1, 1], and divided into data blocks according to the preset batch size B=64 (125,000 / 64≈1953 batches per node). Each data block contains image data and corresponding labels (such as One-Hot encoding of "cat" and "dog"). The computing submodule reads the first data block (batch number 1) from the local storage submodule and prepares to perform calculations. The calculation process is as follows: Figure 3 As shown in the figure, the local gradient is first calculated and the calculation time is recorded. Then the communication submodule obtains the corresponding timestamp and sends it to the central aggregation server. After sending, the next batch of calculations is prepared to avoid the situation where some computing nodes do not send data for a long time.

[0076] In a feasible implementation, after the central aggregation server receives the local gradients and the timestamps when they are sent from each computing node, it further includes: performing dimension verification on the local gradients sent by each computing node and performing timestamp verification on the timestamps.

[0077] like Figure 4 As shown, specifically, the dimension check includes checking the local gradient G of the i-th node i The parameter dimensions of the node are checked to see if they are consistent with the dimensions of the current model parameters. For example, if the gradient sent by node 3 is 1000 less in dimension, it is marked as "invalid" and a retransmission instruction is sent (error code 101). Timestamp verification involves querying the node's last valid timestamp (initially 0). If the timestamp is greater than the previous timestamp, the node passes; otherwise, it is marked as "invalid" (error code 102). Local gradients that pass the verification are stored in the gradient information table for use in weight calculation and gradient aggregation.

[0078] Optionally, when the number of computing nodes receiving local gradients is greater than a preset aggregation threshold, the weight of each computing node is calculated and gradient aggregation is performed to obtain a global gradient, including: when the number of computing nodes receiving local gradients is greater than a preset aggregation threshold, obtaining an evaluation index of each computing node; the evaluation index is used to characterize the performance of the computing node; based on the evaluation index of each computing node, the weight of each computing node is calculated; and gradient aggregation is performed according to the local gradients sent by each computing node and the weight of each computing node to obtain a global gradient.

[0079] In this embodiment, after receiving the local gradient, the central aggregation server calculates the number of nodes that receive the local gradient. When the number of computing nodes that receive the local gradient is greater than a preset aggregation threshold, the central aggregation server calculates the weight of each computing node and performs gradient aggregation.

[0080] Next, we will explain how to use weights to reflect the performance of each node and perform gradient aggregation. Figure 5 The dynamic weight submodule reads the metrics of each node several times (for example, 10 times) from the database. The metrics include GPU computing power p and average round-trip latency t. Then, the numerator a1 of computing node 1 is obtained through formula 1, and the weight w1 corresponding to computing node 1 is obtained through formula 2:

[0081] a=(adjustment coefficient 0.6*p / (t+1e -6 )(Formula 1)

[0082]

[0083] The following is a calculation example to illustrate the above calculation process. For example, if the computing power of computing node 1 is p = 10 TFLOPS (GPU computing power unit) and the average round-trip delay is t = 20ms (average round-trip delay), then the numerator a1 of node 1 is 0.6x10 / 20+1e -6 =0.3; the computing power parameters and latency parameters of the remaining nodes are omitted. Only the numerators of the six computing nodes are calculated according to the above method, and the results are 0.3, 0.192, 0.25, 0.21, 0.18, and 0.15 respectively. The sum of the numerators is 0.3+0.192+0.25+0.21+0.18+0.15=1.282, and the weight w1=0.3 / 1.282≈0.234. The weight calculation method for the remaining nodes is similar and will not be calculated here. However, the sum of the weights of the six computing nodes is 1. Therefore, the weight of each node reflects the computing performance of the computing node, making it easier to adjust the model performance based on the computing performance of each node.

[0084] Furthermore, a weighted summation is performed based on the weight of each node, and the global gradient Gglobal = 0.234 (weight w1) × G1 + 0.150 × G2 + 0.195 × G3 + 0.164 × G4 + 0.140 × G5 + 0.117 × G6, where G1 to G6 are local gradients.

[0085] Optionally, after obtaining the global gradient, the method further includes: calculating an aggregation error, and triggering weight recalculation when the aggregation error is greater than a preset error threshold.

[0086] In this embodiment, the preset error threshold is set to 0.2, and the aggregation error σ is calculated as mean(|Gi -Gglobal|), if σ=0.15<0.2, the aggregation is valid; if σ=0.22 for a batch, the weight recalculation is triggered, for example, recalculating the last five indicators, thereby improving the accuracy of the weight calculation.

[0087] Optionally, based on the global gradient, the target learning rate and target model parameters are calculated, including: obtaining the global gradient for a preset time period or a preset number of times, and the global gradient is based on the calculated variance; according to the comparison result of the variance and a preset variance threshold, the initial learning rate is adjusted to obtain the target learning rate; based on the global gradient and the target learning rate, the target model parameters are determined.

[0088] In this embodiment, the process of calculating the target learning rate and target model parameters is introduced by way of example. The initial learning rate is 0.01, and the variance threshold σ is preset. 2 high=0.01、σ 2 low=0.001, first the adaptive learning rate adjustment submodule reads the most recent 10 global gradients [G g lobal 1 ,...,G g lobal 10 ], and calculate the variance σ 2 =var(G g lobal 1. ..G g lobal 10 ). Compare the variance with the preset variance threshold, adjust the initial learning rate, and get the target learning rate. For example, if the current σ is calculated 2 =0.012>0.01(σ 2 high), then the target learning rate η=0.01(initial learning rate)×1.1=0.011; if the subsequent σ 2 =0.0008<0.001(σ 2 low), then the target learning rate η = 0.011 × 0.9 = 0.0099. 1.1 and 0.9 are adjustment coefficients that can be set based on actual application. Dynamically adjusting the learning rate improves model adjustment flexibility.

[0089] Furthermore, the central server calculates the target model parameters θnew = θold - η × Gglobal (θold is the current model parameter) based on the global gradient Gglobal and the target learning rate η, thereby obtaining the target model parameters for subsequent iteration of the current model parameters.

[0090] Optionally, after calculating the target learning rate and the target model parameters, the method further includes: comparing the target model parameters with the current model parameters to extract a difference parameter block; and sending the difference parameter block to each computing node via incremental transmission.

[0091] In this embodiment, the parameter distribution submodule compares the target model parameters θnew with the current model parameters θold, extracts the difference parameter blocks (such as only the Conv2d layer and the fully connected layer parameters change, accounting for 30% of the total parameters), and uses the algorithm to compress and send it through UDP multicast, thereby greatly reducing the transmission pressure. The data contains the version number v1 and the parameter checksum. After receiving the data, each node communication submodule will verify the integrity of the compressed package, and update the local model parameters to version v1 after decompression. The local storage submodule records the version number (completes one iteration). For nodes that have not received the data packet (such as node 5 timed out for 3s), the server retransmits via TCP unicast to ensure that all 8 nodes are synchronized to v1.

[0092] Optionally, after receiving the target learning rate and target model parameters sent by the central aggregation server and performing iterative updates based on the target learning rate and target model parameters, the method also includes: calculating the model accuracy after the number of iterative updates reaches a first preset number; if the model accuracy has not been continuously improved, or after the number of iterative updates reaches a second preset number, stopping training the model.

[0093] In this embodiment, the iteration termination condition can be set. For example, the first preset number of times is 100 times, and the second preset number of times is 10,000 times. Then, every time the model completes 100 iterations, the global storage submodule evaluates the accuracy of the model on the validation set. If the accuracy of the model does not improve for several consecutive times (for example, 5 consecutive times), it is considered that the training accuracy has reached the maximum. Alternatively, after the model completes 10,000 iterations, it is considered that the training accuracy has reached the maximum, at which point the final model parameters are saved and training ends.

[0094] Further, as Figures 1 to 5 The specific implementation of the method shown in this embodiment provides a distributed training device, such as Figure 6 As shown, the device includes: a generating unit 601, a sending unit 602 and an updating unit 603.

[0095] A generating unit 601 is configured to read the training data and generate a local gradient in combination with the current model parameters;

[0096] A sending unit 602 is configured to send the local gradient and a timestamp of sending to a central aggregation server;

[0097] The updating unit 603 is configured to receive the target learning rate and target model parameters sent by the central aggregation server, and perform iterative update based on the target learning rate and the target model parameters.

[0098] In a specific application scenario, the generation unit 601 is further configured to normalize the training data and divide the data blocks according to a preset batch size; based on the data blocks and the current model parameters, calculate the parameter gradients of each layer through forward propagation and back propagation; and integrate the parameter gradients of each layer to obtain the local gradient.

[0099] In a specific application scenario, the update unit 603 is further configured to calculate the model accuracy after the number of iterative updates reaches a first preset number; and stop training the model when the model accuracy has not been continuously improved or when the number of iterative updates reaches a second preset number.

[0100] Further, as Figures 1 to 5 The specific implementation of the method shown in this embodiment provides a distributed training device, such as Figure 7 As shown, the apparatus includes: a receiving unit 701 , an aggregation unit 702 , a calculation unit 703 and a distribution unit 704 .

[0101] A receiving unit 701 is configured to receive local gradients and a timestamp of transmission from each computing node;

[0102] Aggregation unit 702, configured to calculate the weight of each computing node and perform gradient aggregation to obtain a global gradient when the number of computing nodes receiving the local gradient is greater than a preset aggregation threshold;

[0103] A calculation unit 703 is configured to calculate a target learning rate and target model parameters based on the global gradient;

[0104] The distribution unit 704 is configured to send the target learning rate and the target model parameters to each computing node respectively, so that each computing node performs iterative update according to the target learning rate and the target model parameters.

[0105] In a specific application scenario, the aggregation unit 702 is further configured to obtain an evaluation index of each computing node when the number of computing nodes receiving local gradients is greater than the preset aggregation threshold; the evaluation index is used to characterize the performance of the computing node; based on the evaluation index of each computing node, the weight of each computing node is calculated; and gradient aggregation is performed according to the local gradients sent by each computing node and the weight of each computing node to obtain the global gradient.

[0106] In a specific application scenario, the computing unit 703 is further configured to obtain a global gradient for a preset time period or a preset number of times, and the global gradient is based on a calculated variance; based on a comparison result of the variance and a preset variance threshold, the initial learning rate is adjusted to obtain the target learning rate; based on the global gradient and the target learning rate, the target model parameters are determined.

[0107] It should be noted that for other corresponding descriptions of the functional units involved in the distributed training device provided in this embodiment, please refer to Figures 1 to 5 The corresponding description in will not be repeated here.

[0108] Based on the above Figures 1 to 5 The method shown in FIG. 1 is a method for performing the above-mentioned steps. Accordingly, this embodiment further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program can realize the above-mentioned steps. Figures 1 to 5 The method shown.

[0109] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0110] Based on the above Figures 1 to 5 The method shown, and Figure 6 and Figure 7 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides an electronic device, which can be configured on a computer terminal side, etc. The device includes a memory, a processor and a computer program stored on the memory, and the processor executes the computer program to achieve the above-mentioned Figures 1 to 5 The method shown.

[0111] Optionally, the physical device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a Wi-Fi module, and the like. The user interface may include a display, an input unit such as a keyboard, and the like. The optional user interface may also include a USB interface, a card reader interface, and the like. The network interface may optionally include a standard wired interface, a wireless interface (such as a Wi-Fi interface), and the like.

[0112] Those skilled in the art will understand that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0113] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device, supporting the execution of information processing programs and other software and / or programs. The network communication module is used to enable communication between components within the storage medium, as well as with other hardware and software within the physical information processing device.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the solution of this embodiment, compared with the related art, firstly, a threat intelligence feature library containing a matching library for judging whether threat intelligence exists and an offline library for judging the type of intelligence is created, and the threat intelligence feature library is used to detect the existence of threat intelligence and its type during the detection process, thereby getting rid of the dependence on the detection system and being able to independently detect threat intelligence.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0116] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A distributed training method, characterized in that: include: Read the training data and generate local gradients based on the current model parameters; Sending the local gradient and the timestamp of sending to a central aggregation server; Receive the target learning rate and target model parameters sent by the central aggregation server, and perform iterative update based on the target learning rate and the target model parameters.

2. The method according to claim 1, characterized in that The reading of training data and generating local gradients in combination with current model parameters include: Standardize the training data and divide the data into blocks according to a preset batch size; Based on the data block and the current model parameters, calculating the parameter gradients of each layer through forward propagation and backpropagation; The local gradient is obtained by integrating the parameter gradients of each layer.

3. The method according to claim 2, characterized in that After receiving the target learning rate and target model parameters sent by the central aggregation server and performing iterative updates based on the target learning rate and the target model parameters, the method further includes: After the number of iterative updates reaches a first preset number, calculating the model accuracy; When the accuracy of the model does not continue to improve, or when the number of iterative updates reaches a second preset number, training of the model is stopped.

4. A distributed training method, characterized in that: include: Receive local gradients from each computing node and the timestamp when they were sent; When the number of computing nodes receiving local gradients is greater than the preset aggregation threshold, the weights of each computing node are calculated and gradient aggregation is performed to obtain the global gradient; Calculating a target learning rate and target model parameters based on the global gradient; The target learning rate and the target model parameters are sent to each computing node respectively, so that each computing node performs iterative update according to the target learning rate and the target model parameters.

5. The method according to claim 4, characterized in that When the number of computing nodes receiving local gradients is greater than a preset aggregation threshold, the weight of each computing node is calculated and gradient aggregation is performed to obtain a global gradient, including: When the number of computing nodes receiving the local gradient is greater than the preset aggregation threshold, obtaining an evaluation index of each computing node; the evaluation index is used to characterize the performance of the computing node; Calculate the weight of each computing node based on the evaluation index of each computing node; Gradient aggregation is performed based on the local gradients sent by each computing node and the weights of each computing node to obtain the global gradient.

6. The method according to claim 5, characterized in that The calculating of the target learning rate and the target model parameters based on the global gradient includes: Obtaining a global gradient for a preset time period or a preset number of times, wherein the global gradient is based on a calculated variance; Adjusting the initial learning rate according to a comparison result between the variance and a preset variance threshold to obtain the target learning rate; The target model parameters are determined based on the global gradient and the target learning rate.

7. A distributed training device, characterized in that: include: A generation unit is configured to read the training data and generate local gradients based on the current model parameters; a sending unit, configured to send the local gradient and a timestamp of sending to a central aggregation server; An updating unit is configured to receive a target learning rate and target model parameters sent by the central aggregation server, and perform iterative updating based on the target learning rate and the target model parameters.

8. A distributed training device, characterized in that: include: A receiving unit, configured to receive local gradients and a timestamp of sending from each computing node; an aggregation unit configured to calculate the weight of each computing node and perform gradient aggregation to obtain a global gradient when the number of computing nodes receiving the local gradient is greater than a preset aggregation threshold; A computing unit configured to calculate a target learning rate and target model parameters based on the global gradient; The distribution unit is configured to send the target learning rate and the target model parameters to each computing node respectively, so that each computing node performs iterative update according to the target learning rate and the target model parameters.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images