Distributed training method and apparatus, device, and storage medium

Abstract

The application discloses a distributed training method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: a computing node reads training data and generates a local gradient in combination with current model parameters, and sends the local gradient and a time stamp at the time of sending to a central aggregation server; the server receives the local gradient and the time stamp at the time of sending from each computing node; in the case that the number of computing nodes receiving the local gradient is greater than a preset aggregation threshold, the weight of each computing node is calculated and gradient aggregation is performed to obtain a global gradient; based on the global gradient, a target learning rate and target model parameters are calculated; the target learning rate and the target model parameters are sent to each computing node respectively; and the nodes receive the target learning rate and the target model parameters for iterative updating. By adopting an asynchronous communication mechanism and considering the actual performance of each node, a distributed training method which can flexibly adjust each node, has higher training efficiency and improves resource utilization is provided.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a distributed training method, apparatus, device, and storage medium. Background Technology

[0002] In traditional synchronous distributed training methods for artificial intelligence (AI), the training task is assigned to multiple nodes, each equipped with corresponding computing resources (such as a GPU). Each node is responsible for performing forward and backward propagation calculations on its assigned portion of the data, generating local gradients. During each iteration, all nodes must complete their respective local gradient calculations and then send these gradients to the central server via a synchronous communication mechanism. Upon receiving the local gradients from all nodes, the central server aggregates the gradients to obtain the global gradient, which is then distributed back to each node. Each node updates its local model parameters based on the global gradient before proceeding to the next iteration.

[0003] In this process, the computation and communication of each node are strictly synchronized to ensure that the model parameters remain consistent after each iteration. This also means that gradient synchronization can only be performed after the slowest node has completed its computation in each iteration, resulting in faster nodes being idle for extended periods, significantly reducing the overall utilization of computing resources; at the same time, the iteration speed is also slowed down, leading to low efficiency. On the other hand, the performance differences between different nodes are not considered during gradient aggregation, lacking the flexibility to adjust the performance of each node.

[0004] Therefore, how to propose an artificial intelligence training method that can flexibly adjust each node, improve training efficiency, and enhance resource utilization has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, this application provides a distributed training method, apparatus, device and storage medium, the main purpose of which is to propose an asynchronous communication distributed AI training method to improve flexibility, training efficiency and resource utilization.

[0006] Firstly, this application provides a distributed training method, including:

[0007] Read the training data and generate local gradients based on the current model parameters;

[0008] The local gradient and the timestamp at the time of transmission are sent to the central aggregation server;

[0009] The system receives the target learning rate and target model parameters sent by the central aggregation server, and iteratively updates the data based on the target learning rate and target model parameters.

[0010] Optionally, the step of reading training data and generating local gradients in conjunction with current model parameters includes: standardizing the training data and dividing it into data blocks according to a preset batch size; calculating the gradients of parameters at each layer through forward and backward propagation based on the data blocks and the current model parameters; and integrating the gradients of parameters at each layer to obtain the local gradients.

[0011] Optionally, after receiving the target learning rate and target model parameters sent by the central aggregation server, and performing iterative updates based on the target learning rate and target model parameters, the method further includes: calculating the model accuracy after the number of iterative updates reaches a first preset number; stopping training the model if the model accuracy does not improve continuously, or after the number of iterative updates reaches a second preset number.

[0012] Secondly, this application provides a distributed training method, including:

[0013] Receive local gradients from each computing node and the timestamp of the transmission.

[0014] If the number of computation nodes receiving local gradients is greater than a preset aggregation threshold, the weights of each computation node are calculated and gradients are aggregated to obtain the global gradient.

[0015] Based on the global gradient, calculate the target learning rate and target model parameters;

[0016] The target learning rate and the target model parameters are sent to each computing node so that each computing node can iteratively update the model based on the target learning rate and the target model parameters.

[0017] Optionally, the step of calculating the weights of each computing node and performing gradient aggregation to obtain the global gradient when the number of computing nodes receiving local gradients is greater than a preset aggregation threshold includes: obtaining the evaluation index of each computing node when the number of computing nodes receiving local gradients is greater than the preset aggregation threshold; the evaluation index is used to characterize the performance of the computing node; calculating the weight of each computing node based on the evaluation index of each computing node; and performing gradient aggregation according to the local gradients sent by each computing node and the weights of each computing node to obtain the global gradient.

[0018] Optionally, the step of calculating the target learning rate and target model parameters based on the global gradient includes: obtaining the global gradient over a preset time period or a preset number of times, and calculating the variance based on the global gradient; adjusting the initial learning rate according to the comparison result between the variance and a preset variance threshold to obtain the target learning rate; and determining the target model parameters based on the global gradient and the target learning rate.

[0019] Thirdly, this application provides a distributed training device, comprising:

[0020] The generation unit is configured to read training data and generate local gradients based on the current model parameters;

[0021] The sending unit is configured to send the local gradient and the timestamp at the time of transmission to the central aggregation server;

[0022] The update unit is configured to receive the target learning rate and target model parameters sent by the central aggregation server, and to perform iterative updates based on the target learning rate and the target model parameters.

[0023] Fourthly, this application provides a distributed training device, comprising:

[0024] The receiving unit is configured to receive local gradients from each computing node and timestamps at the time of transmission.

[0025] The aggregation unit is configured to calculate the weights of each computing node and perform gradient aggregation to obtain the global gradient when the number of computing nodes receiving local gradients is greater than a preset aggregation threshold.

[0026] The computing unit is configured to calculate the target learning rate and target model parameters based on the global gradient;

[0027] The distribution unit is configured to send the target learning rate and the target model parameters to each computing node, so that each computing node can iteratively update according to the target learning rate and the target model parameters.

[0028] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed training method described in the first or second aspect.

[0029] In a sixth aspect, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the distributed training method described in the first or second aspect.

[0030] Using the above technical solution, this application provides a distributed training method, apparatus, device, and storage medium. First, computing nodes read training data and generate local gradients by combining them with current model parameters. The local gradients and their timestamps are then sent to a central aggregation server. The server receives the local gradients and their timestamps from each computing node. If the number of computing nodes receiving local gradients exceeds a preset aggregation threshold, the weights of each computing node are calculated, and gradient aggregation is performed to obtain the global gradient. Based on the global gradient, the target learning rate and target model parameters are calculated. The target learning rate and target model parameters are then sent to each computing node. The nodes receive the target learning rate and target model parameters and iteratively update them. Compared with related technologies, this application firstly eliminates the need for synchronous communication mechanisms to send local gradients to the server. Instead, it uses asynchronous communication to allow nodes to send local gradients at any time, meaning different computational nodes do not need to send data simultaneously. Gradient aggregation can be performed when the number of computational nodes receiving local gradients on the server exceeds a preset aggregation threshold, thus solving the problems of low training efficiency and low resource utilization caused by synchronous communication in existing methods. Furthermore, gradient aggregation is performed based on the weights of each node, taking into account the performance of each node and adjusting the model parameters and the learning rate of each node. This allows for dynamic adjustments to the actual situation of each node during training, improving the flexibility of training.

[0031] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0032] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This illustration shows an interactive flow diagram of a distributed training method provided in an embodiment of this application;

[0034] Figure 2 This paper presents a schematic diagram of the overall process of a distributed training method provided in an embodiment of this application.

[0035] Figure 3 This illustration shows a flowchart of a computing unit calculating local gradients according to an embodiment of this application;

[0036] Figure 4 This illustration shows a flowchart of a central aggregation server performing dual verification according to an embodiment of this application;

[0037] Figure 5 This illustration shows a flowchart of a central aggregation server calculating weights and gradient aggregation according to an embodiment of this application;

[0038] Figure 6 This paper shows a schematic diagram of the structure of a distributed training device provided in an embodiment of this application;

[0039] Figure 7 A schematic diagram of another distributed training device provided in an embodiment of this application is shown. Detailed Implementation

[0040] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0041] As mentioned in the background, current AI training typically employs a synchronous communication mechanism. This means that in each iteration, all nodes must complete their own local gradient calculations and then send these local gradients to the central server via synchronous communication. The central server then aggregates the gradients and distributes them back to each node to update its local model parameters. This approach has the following drawbacks:

[0042] (1) Serious waste of computing resources: Because all nodes are required to wait for the slowest node to complete the calculation before gradient synchronization can be performed in each iteration, the fast nodes are idle for a long time, which greatly reduces the overall utilization rate of computing resources.

[0043] (2) Low training efficiency: Due to the limited computation progress of the slowest node, the iteration speed of the entire training process is slowed down, which significantly increases the training time. This inefficiency is even more prominent in large-scale distributed training scenarios.

[0044] (3) Lack of flexibility: Gradient aggregation is performed using fixed weights, without taking into account the differences in computing power and communication performance of different nodes, resulting in unsatisfactory aggregation effect; at the same time, the learning rate is fixed and cannot be dynamically adjusted according to the actual situation during the training process, which affects training efficiency.

[0045] To address at least one of the aforementioned technical problems, this embodiment proposes a distributed training method that can be applied to a distributed training device or electronic device. This device or electronic device can be installed or integrated into an AI training system, which includes a central aggregation server and multiple computing nodes. During operation, it can execute any of the distributed training methods mentioned below.

[0046] In one feasible implementation, the AI ​​training system includes a GPU node module and a central aggregation server module. As a possible implementation, the following configuration can be adopted:

[0047] The GPU node module, where each GPU node serves as a computing node for distributed training, also known as a computing terminal, comprises a computing submodule, a communication submodule, a local storage submodule, and a status monitoring submodule. Specifically:

[0048] The computation submodule is equipped with a deep learning framework (such as TensorFlow). After receiving the training data from the local storage submodule, it loads the current local model parameters, performs forward propagation calculations, generates predicted values ​​and loss function values, and then calculates the local gradients of each model parameter through a backpropagation algorithm (such as the chain rule). It also supports batch processing calculations.

[0049] Communication submodule: It has a built-in asynchronous communication protocol stack, establishes a connection with the central aggregation server, sends local gradients and timestamps in real time, listens for parameter update pushes from the server, and triggers local model updates after receiving global model parameters.

[0050] Local storage submodule: Partitioned storage of allocated training data (which can be divided by data category or time series), current local model parameter snapshots, and historical gradient records (retaining the calculation results of the most recent preset number of times), supporting data compression and fast reading.

[0051] Status monitoring submodule: Real-time collection of node GPU utilization (refreshed once per second), computation time (single batch forward + backward propagation time), network send / receive rate and other indicators, and periodically send them to the dynamic weight adjustment submodule of the central aggregation server.

[0052] The central aggregation server module in the AI ​​training system, acting as the global coordination center, includes submodules for receiving and recording data, dynamic weight adjustment, gradient aggregation, adaptive learning rate adjustment, parameter distribution, global storage, and cluster management. Specifically:

[0053] The receiving and recording submodule deploys a multi-threaded listening service to receive local gradient data from each computing node in parallel. It parses data packets to obtain gradient values, timestamps, and node IDs, and stores them in a gradient information table (a structured database table containing fields: sequence number, node ID, timestamp, gradient vector, and receiving status). It can also mark abnormal data packets (such as gradient dimension mismatch or duplicate timestamps) and trigger a retransmission mechanism.

[0054] Dynamic weight adjustment submodule: Connects to the status monitoring submodule of computing nodes, stores the historical computing power (average TFLOPS in the last 10 minutes) and communication latency (average round-trip time of the last 10 transmissions) of each node, calculates dynamic weights periodically according to a preset algorithm (every time N / 2 gradients are received, where N is the total number of nodes), generates a weight table and pushes it to the gradient aggregation submodule.

[0055] Gradient aggregation submodule: Reads valid gradient data from the receiving and recording submodule, sorts it by timestamp, calls the dynamic weight table to perform weighted summation of the gradient vectors to generate the global gradient, and calculates the aggregation error (the average deviation between the gradient of each node and the global gradient). When the aggregation error exceeds the preset aggregation threshold, the weights are recalculated.

[0056] Adaptive Learning Rate Adjustment Submodule: Obtains historical global gradient sequences from the gradient aggregation submodule, calculates statistical features such as gradient change rate and variance, combines the model validation accuracy from the global storage submodule, generates a new learning rate according to the adjustment strategy, and passes it to the parameter distribution submodule.

[0057] The parameter distribution submodule receives the updated global model parameters, uses incremental transmission (only sends the changed parameter blocks), pushes them to all computing nodes via a broadcast protocol (such as UDP multicast), records the reception confirmation status of each node, and performs unicast retransmission for nodes that have not confirmed the data after timeout.

[0058] The global storage submodule saves complete historical versions of global model parameters (named by iteration number), dynamic weight records of all nodes, gradient aggregation logs, and validation set performance metrics (accuracy, loss value), and supports data backup and backtracking queries.

[0059] Cluster Management Submodule: Monitors the online status of each computing node. When a node goes offline (no heartbeat response for 3 consecutive times), it automatically assigns its training tasks to other nodes and adjusts the set of nodes for dynamic weight calculation.

[0060] Based on the above system configuration, the distributed training method using the asynchronous communication mechanism proposed in this embodiment will be described next, such as... Figure 1 The diagram illustrates the interactive flow of the distributed training method proposed in this embodiment. The method includes:

[0061] S101, the computing node reads the training data and generates local gradients by combining the current model parameters;

[0062] Each computing node reads training data from its local storage submodule and then uses this data to generate local gradients based on the current model parameters. The current model parameters refer to the local model parameters before the current iteration; these parameters may be adjusted after each iteration. Furthermore, the training data can be preprocessed and divided into batches of data blocks.

[0063] S102, the computing node sends the local gradient and timestamp to the central aggregation server;

[0064] After calculating the local gradient, the communication submodule obtains the system timestamp and packages the data to send to the central aggregation server. After sending, the computation submodule loads the next batch of data blocks and continues computation, achieving parallel communication and computation.

[0065] S103, the central aggregation server receives local gradients from each computing node and the timestamps at the time of transmission;

[0066] S104, when the number of computing nodes receiving local gradients is greater than the preset aggregation threshold, the central aggregation server calculates the weight of each computing node and performs gradient aggregation to obtain the global gradient.

[0067] After receiving local gradients, the central aggregation server calculates the number of nodes that received local gradients. If the number of nodes receiving local gradients exceeds a preset aggregation threshold, it calculates the weights of each node and performs gradient aggregation. Figure 1 Only two computation nodes are shown as examples; the rest are omitted. For instance, if there are 8 computation nodes, the preset aggregation threshold K can be set to 6 (75% of the total). When local gradients are received from more than 6 computation nodes, the weights of each node can be further calculated and gradient aggregation can be performed.

[0068] S105, the central aggregation server calculates the target learning rate and target model parameters based on the global gradient;

[0069] The central aggregation server then calculates the target learning rate and target model parameters based on the global gradient. The target learning rate is used to adjust the learning efficiency of the computing nodes, and the target model parameters are used to update the local model parameters of each node after this iteration.

[0070] S106, The central aggregation server sends the target learning rate and target model parameters to each computing node respectively;

[0071] S107, the computing nodes are iteratively updated based on the target learning rate and target model parameters.

[0072] In this embodiment, instead of sending local gradients to the server via synchronous communication, asynchronous communication is used to send local gradients at any time, meaning different computing nodes do not need to send data simultaneously. Gradient aggregation can be performed when the number of computing nodes receiving local gradients on the server exceeds a preset aggregation threshold, solving the problems of low training efficiency and low resource utilization caused by synchronous communication in existing methods. Furthermore, gradient aggregation is performed based on the weights of each node, taking into account the performance of each node, and adjusting the model parameters and the learning rate of each node. This allows for dynamic adjustments to the actual situation of each node during training, improving the flexibility of training.

[0073] Referring to Figure 2, a schematic diagram of the overall process of a distributed training method provided in this application embodiment is shown. For the initialization step, a specific application configuration is given here for understanding. System initialization and cluster configuration include: starting the cluster management submodule of the central aggregation server, reading the configuration file and setting key parameters. In this application embodiment, the number of GPU nodes is set to N = 8, the current model parameters are θ0, initialization is performed using a He normal distribution (such as the convolutional layer parameters of ResNet50), the total number of training iterations is T = 10000, the batch size is B = 64, the dynamic weight adjustment coefficient α = 0.6 (the optimal value determined through comparative experiments), the initial learning rate is 0.01, the gradient aggregation threshold K = 6 (75% of the total number of nodes), the aggregation error threshold σ = 0.2, and the gradient variance threshold σ 2 high = 0.01, σ 2 low = 0.001. Pre-training preparation steps: The global storage submodule creates a repository, marks θ0 as version v0, and stores it; the cluster management submodule sends initialization commands to the 8 GPU nodes. After receiving the commands, the local storage submodule loads the allocated ImageNet training subset (125,000 images per node), the communication submodule establishes a long TCP connection with the server (port 5000), the status monitoring submodule starts metric acquisition (sampling frequency 1Hz), and the computation submodule loads θ0 into GPU memory. The above parameters will be used one by one in subsequent steps. The following sections will explain each step of the training method in detail to facilitate understanding of the overall scheme.

[0074] Optionally, reading training data and generating local gradients in conjunction with current model parameters includes: standardizing the training data and dividing it into data blocks according to a preset batch size; calculating the gradients of parameters at each layer through forward and backward propagation based on the data blocks and current model parameters; and integrating the gradients of parameters at each layer to obtain local gradients.

[0075] In this embodiment, before calculating the local gradient, each computing node can divide the training data into data blocks according to a preset batch size. This makes the data processing process of each computing node more refined, ensuring that even computing nodes with slower computation progress will not experience prolonged delays in data output due to excessive data volume. First, the local storage submodule of each computing node performs preprocessing on the training data: for example, resizing the image training data to 224×224 pixels, standardizing it to the [-1, 1] range, and dividing it into data blocks according to a preset batch size B = 64 (125,000 / 64 ≈ 1953 batches per node). Each data block contains image data and corresponding labels (such as One-Hot encoding for "cat" and "dog"). The computing submodule reads the first data block (batch number 1) from the local storage submodule, ready to perform the computation. The computation process is as follows: Figure 3 As shown, the local gradient is first calculated and the calculation time is recorded. Then, the communication submodule obtains the corresponding timestamp and sends it to the central aggregation server. After sending, the next batch of calculations is prepared, thus avoiding the situation where some computing nodes have not sent data for a long time.

[0076] In one feasible implementation, after the central aggregation server receives the local gradients and timestamps from each computing node, the method further includes: performing dimension verification on the local gradients sent by each computing node and performing timestamp verification on the timestamps.

[0077] like Figure 4 As shown, specifically, dimensionality verification includes checking the local gradient G of the i-th node. i The system checks whether the dimension of the parameters matches the dimension of the current model parameters. For example, if the gradient sent by node 3 has 1000 fewer dimensions, it is marked as "invalid" and a retransmission command is sent (error code 101). Timestamp verification includes querying the last valid timestamp of the node (initially 0). If the timestamp is greater than the previous timestamp, it passes; otherwise, it is marked as "invalid" (error code 102). Local gradients that pass verification are stored in the gradient information table for weight calculation and gradient aggregation.

[0078] Optionally, if the number of computing nodes receiving local gradients is greater than a preset aggregation threshold, the weights of each computing node are calculated and gradient aggregation is performed to obtain the global gradient. This includes: obtaining the evaluation metrics of each computing node when the number of computing nodes receiving local gradients is greater than the preset aggregation threshold; the evaluation metrics are used to characterize the performance of the computing nodes; calculating the weights of each computing node based on the evaluation metrics of each computing node; and performing gradient aggregation based on the local gradients sent by each computing node and the weights of each computing node to obtain the global gradient.

[0079] In this embodiment, after receiving the local gradient, the central aggregation server calculates the number of nodes that have received the local gradient. If the number of computing nodes that have received the local gradient is greater than a preset aggregation threshold, the weight of each computing node is calculated and gradient aggregation is performed.

[0080] The following section explains how weights are used to reflect the performance of each node and how gradient aggregation is performed. Figure 5 The dynamic weighting submodule reads metrics for each node from the database a certain number of times (e.g., 10 times). These metrics include GPU computing power p and average round-trip latency t. Then, it obtains the numerator a1 for computing node 1 using Formula 1, and the corresponding weight w1 for computing node 1 using Formula 2.

[0081] a = (adjustment coefficient 0.6 * p / (t + 1e)) -6 (Formula 1)

[0082]

[0083] The following calculation example illustrates the above calculation process. For instance, if the computing power of node 1 is p = 10 TFLOPS (GPU computing power unit) and the average round-trip time is t = 20 ms (average round-trip latency), then the numerator a1 of node 1 is 0.6 x 10 / 20 + 1e -6 =0.3; the computing power and latency parameters of the remaining nodes are omitted. Only the numerators of the six computing nodes are given. Calculated in the above manner, they are 0.3, 0.192, 0.25, 0.21, 0.18, and 0.15 respectively. The sum of the numerators is 0.3 + 0.192 + 0.25 + 0.21 + 0.18 + 0.15 = 1.282. Therefore, the weight w1 = 0.3 / 1.282 ≈ 0.234. The weight calculation method for the remaining nodes is similar and will not be calculated here, but the sum of the weights of the six computing nodes is 1. Thus, the weights of each node reflect the computing performance of the computing nodes, which facilitates subsequent adjustments to the model performance based on the computing performance of each node.

[0084] Furthermore, a weighted sum is performed based on the weights of each node, and the global gradient Gglobal = 0.234(weight w1)×G1 + 0.150×G2 + 0.195×G3 + 0.164×G4 + 0.140×G5 + 0.117×G6, where G1 to G6 are local gradients.

[0085] Optionally, after obtaining the global gradient, the method further includes: calculating the aggregation error, and triggering weight recalculation if the aggregation error is greater than a preset error threshold.

[0086] In this embodiment, a preset error threshold of 0.2 is used to calculate the aggregation error σ = mean(|Gi -Gglobal|), if σ=0.15<0.2, the aggregation is valid; if σ=0.22 for a certain batch, the weight recalculation is triggered, for example, the most recent 5 indicators are recalculated, thereby improving the accuracy of the weight calculation.

[0087] Optionally, the target learning rate and target model parameters are calculated based on the global gradient, including: obtaining the global gradient over a preset time period or a preset number of times, and calculating the variance based on the global gradient; adjusting the initial learning rate according to the comparison result between the variance and a preset variance threshold to obtain the target learning rate; and determining the target model parameters based on the global gradient and the target learning rate.

[0088] In this embodiment, the process of calculating the target learning rate and target model parameters is described by way of example. The initial learning rate is set to 0.01, and the variance threshold σ is preset. 2 high = 0.01, σ 2 When low = 0.001, the adaptive learning rate adjustment submodule first reads the last 10 global gradients [G] from the global storage submodule. g lobal 1 ,...,G g lobal 10 ] and calculate the variance σ 2 =var(G g lobal 1. ..G g lobal 10 The variance is compared with a preset variance threshold, and the initial learning rate is adjusted to obtain the target learning rate. For example, if the current σ is calculated... 2 =0.012>0.01(σ 2 If the initial learning rate is high, then the target learning rate η = 0.01 (initial learning rate) × 1.1 = 0.011; if the subsequent σ 2 =0.0008<0.001(σ 2 If the learning rate is low, then the target learning rate η = 0.011 × 0.9 = 0.0099. Here, 1.1 and 0.9 are adjustment coefficients that can be set according to the actual application. Dynamically adjusting the learning rate improves the flexibility of model adjustment.

[0089] Furthermore, the central server calculates the target model parameters θnew = θold - η × Gglobal (θold is the current model parameter) based on the global gradient Gglobal and the target learning rate η, thereby obtaining the target model parameters, which are used for subsequent iterations on the current model parameters.

[0090] Optionally, after calculating the target learning rate and target model parameters, the method further includes: comparing the target model parameters with the current model parameters and extracting the difference parameter blocks; and sending the difference parameter blocks to each computing node via incremental transmission.

[0091] In this embodiment, the parameter distribution submodule compares the target model parameter θnew with the current model parameter θold, extracts the difference parameter blocks (e.g., only the Conv2d layer and fully connected layer parameters have changed, accounting for 30% of the total parameters), compresses them using an algorithm, and sends them via UDP multicast, thus greatly reducing the transmission pressure. The data includes the version number v1 and the parameter checksum. After receiving the data, each node's communication submodule verifies the integrity of the compressed package, decompresses it, and updates its local model parameters to version v1. The local storage submodule records the version number (completing one iteration). For nodes that do not receive the data packet (e.g., node 5 times out by 3 seconds), the server retransmits it via TCP unicast to ensure that all 8 nodes are synchronized to v1.

[0092] Optionally, after receiving the target learning rate and target model parameters sent by the central aggregation server, and performing iterative updates based on the target learning rate and target model parameters, the method further includes: calculating the model accuracy after the number of iteration updates reaches a first preset number; stopping model training if the model accuracy does not improve continuously, or after the number of iteration updates reaches a second preset number.

[0093] In this embodiment, the iteration termination condition can be set. For example, the first preset number of iterations is 100, and the second preset number of iterations is 10,000. Then, after the model completes 100 iterations, the global storage submodule evaluates the accuracy of the model on the validation set. If the accuracy of the model does not improve for several consecutive iterations (e.g., 5 consecutive iterations), it is considered that the training accuracy has reached its maximum. Alternatively, after the model completes 10,000 iterations, it is considered that the training accuracy has reached its maximum. At this point, the final model parameters are saved and training ends.

[0094] Furthermore, as Figures 1 to 5 The specific implementation of the method shown in this embodiment provides a distributed training device, such as... Figure 6 As shown, the device includes: a generation unit 601, a transmission unit 602, and an update unit 603.

[0095] The generation unit 601 is configured to read training data and generate local gradients in combination with the current model parameters;

[0096] The sending unit 602 is configured to send the local gradient and the timestamp at the time of transmission to the central aggregation server;

[0097] The update unit 603 is configured to receive the target learning rate and target model parameters sent by the central aggregation server, and to perform iterative updates based on the target learning rate and the target model parameters.

[0098] In a specific application scenario, the generation unit 601 is further configured to standardize the training data and divide it into data blocks according to a preset batch size; calculate the gradient of each layer parameter through forward propagation and backward propagation based on the data blocks and the current model parameters; and integrate the gradients of each layer parameter to obtain the local gradient.

[0099] In specific application scenarios, the update unit 603 is further configured to calculate the model accuracy after the number of iterations reaches a first preset number; and to stop training the model if the model accuracy does not improve continuously, or after the number of iterations reaches a second preset number.

[0100] Furthermore, as Figures 1 to 5 The specific implementation of the method shown in this embodiment provides a distributed training device, such as... Figure 7 As shown, the device includes: a receiving unit 701, an aggregation unit 702, a calculation unit 703, and a distribution unit 704.

[0101] The receiving unit 701 is configured to receive local gradients from each computing node and timestamps at the time of transmission.

[0102] The aggregation unit 702 is configured to calculate the weights of each computing node and perform gradient aggregation to obtain the global gradient when the number of computing nodes receiving local gradients is greater than a preset aggregation threshold.

[0103] The computing unit 703 is configured to calculate the target learning rate and target model parameters based on the global gradient;

[0104] The distribution unit 704 is configured to send the target learning rate and the target model parameters to each computing node, so that each computing node can perform iterative updates based on the target learning rate and the target model parameters.

[0105] In a specific application scenario, the aggregation unit 702 is further configured to obtain the evaluation index of each computing node when the number of computing nodes receiving local gradients is greater than the preset aggregation threshold; the evaluation index is used to characterize the performance of the computing node; the weight of each computing node is calculated based on the evaluation index of each computing node; and gradient aggregation is performed according to the local gradient sent by each computing node and the weight of each computing node to obtain the global gradient.

[0106] In specific application scenarios, the computing unit 703 is further configured to acquire the global gradient over a preset time period or a preset number of times, and the global gradient is based on the calculated variance; adjust the initial learning rate according to the comparison result of the variance and the preset variance threshold to obtain the target learning rate; and determine the target model parameters based on the global gradient and the target learning rate.

[0107] It should be noted that other corresponding descriptions of the functional units involved in the distributed training device provided in this embodiment can be found in [reference needed]. Figures 1 to 5 The corresponding descriptions in [the document] will not be repeated here.

[0108] Based on the above, Figures 1 to 5 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figures 1 to 5 The method shown.

[0109] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0110] Based on the above, Figures 1 to 5 The method shown, and Figure 6 and Figure 7 To achieve the above objectives, this application also provides an electronic device, which can be configured on a computer side, etc. This device includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to achieve the above-described virtual device embodiments. Figures 1 to 5 The method shown.

[0111] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0112] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0113] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented using hardware. By applying the solution of this embodiment, compared with related technologies, firstly, a threat intelligence feature library is created containing a matching library for determining the existence of threat intelligence and an offline library for determining the type of intelligence. During the detection process, the threat intelligence feature library is used to detect the existence and type of threat intelligence, thereby eliminating the dependence on the detection system and enabling autonomous detection of threat intelligence.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A distributed training method, comprising: The method comprises the following steps: reading training data and generating a local gradient in combination with current model parameters; sending the local gradient and a time stamp at the time of sending to a central aggregation server, so that the central aggregation server obtains evaluation indexes of each computing node when the number of computing nodes receiving the local gradient is greater than a preset aggregation threshold; the evaluation indexes are used to represent the performance of the computing nodes; weights of the computing nodes are calculated based on the evaluation indexes of the computing nodes; performing gradient aggregation based on the local gradient sent by each computing node and the weights of the computing nodes to obtain a global gradient; calculating an aggregation error, and triggering weight recalculation when the aggregation error is greater than a preset error threshold; a target learning rate and a target model parameter are calculated based on the global gradient; receiving the target learning rate and the target model parameter sent by the central aggregation server, and performing iterative updating based on the target learning rate and the target model parameter.

2. The method of claim 1, wherein, The reading training data and generating a local gradient in combination with current model parameters comprises the following steps: standardizing the training data and dividing data blocks according to a preset batch processing size; calculating the gradient of each layer parameter based on the data blocks and the current model parameters through forward propagation and backward propagation; integrating the gradient of each layer parameter to obtain the local gradient.

3. The method of claim 2, wherein, After the receiving the target learning rate and the target model parameter sent by the central aggregation server, and performing iterative updating based on the target learning rate and the target model parameter, the method further comprises the following steps: calculating a model accuracy rate after the number of iterative updating reaches a first preset number; stopping training of the model when the model accuracy rate does not continuously improve or when the number of iterative updating reaches a second preset number.

4. A distributed training method, comprising: The method comprises the following steps: receiving the local gradient and the time stamp at the time of sending from each computing node; obtaining evaluation indexes of each computing node when the number of computing nodes receiving the local gradient is greater than a preset aggregation threshold; the evaluation indexes are used to represent the performance of the computing nodes; calculating weights of the computing nodes based on the evaluation indexes of the computing nodes; performing gradient aggregation based on the local gradient sent by each computing node and the weights of the computing nodes to obtain a global gradient; calculating an aggregation error, and triggering weight recalculation when the aggregation error is greater than a preset error threshold; calculating a target learning rate and a target model parameter based on the global gradient; sending the target learning rate and the target model parameter to each computing node respectively, so that each computing node performs iterative updating based on the target learning rate and the target model parameter.

5. The method of claim 4, wherein, The calculating a target learning rate and a target model parameter based on the global gradient comprises the following steps: obtaining global gradients in a preset time period or a preset number of times, and calculating a variance based on the global gradients; adjusting an initial learning rate based on a comparison result of the variance and a preset variance threshold to obtain the target learning rate; determining the target model parameter based on the global gradient and the target learning rate.

6. A distributed training apparatus, characterized by, The method comprises the following steps: a generating unit configured to read training data and generate a local gradient in combination with current model parameters; a sending unit configured to send the local gradient and a time stamp at the time of sending to a central aggregation server, so that the central aggregation server acquires an evaluation index of each computing node in a case where a number of computing nodes that receive the local gradient is greater than a preset aggregation threshold; the evaluation index is used to represent performance of the computing node; and weights of the computing nodes are calculated based on the evaluation index of each computing node; gradient aggregation is performed according to the local gradient sent by each computing node and the weight of each computing node, to obtain a global gradient; an aggregation error is calculated, and weight recalculation is triggered in a case where the aggregation error is greater than a preset error threshold; a target learning rate and a target model parameter are calculated based on the global gradient; 7. A distributed training apparatus, characterized by, a receiving unit configured to receive the local gradient and a time stamp at the time of sending from each computing node; an aggregation unit configured to acquire an evaluation index of each computing node in a case where a number of computing nodes that receive the local gradient is greater than a preset aggregation threshold; the evaluation index is used to represent performance of the computing node; and weights of the computing nodes are calculated based on the evaluation index of each computing node; gradient aggregation is performed according to the local gradient sent by each computing node and the weight of each computing node, to obtain a global gradient; an aggregation error is calculated, and weight recalculation is triggered in a case where the aggregation error is greater than a preset error threshold; a calculation unit configured to calculate a target learning rate and a target model parameter based on the global gradient; a distribution unit configured to send the target learning rate and the target model parameter to each computing node respectively, so that each computing node performs iterative updating according to the target learning rate and the target model parameter. The processor executes the computer program to implement the method in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-7. The computer program is executed by the processor to implement the method in any one of claims 1 to 5.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images