A distributed monitoring system and method based on diffusion model

Abstract

The present invention discloses a distributed monitoring system and method based on a diffusion model. The system includes an elastic fault-tolerant training module to obtain training status; a training information visualization module to obtain training data; a GPU performance monitoring module to obtain GPU information; and a forward propagation process monitoring module to obtain model parameters. The present invention adopts the above-mentioned distributed monitoring system and method based on a diffusion model, and by applying an advanced diffusion model to distributed training monitoring, the efficiency and stability of distributed training are effectively improved, the optimal configuration of resources and automated fault prevention are achieved, and problems such as node failure and low training efficiency are solved; system monitoring is achieved by initializing the monitoring module and calling the monitoring function in the training script, and storing and visualizing data through different data formats and tools; the diffusion model not only enhances the elasticity and fault tolerance of the system, but also improves the real-time and accuracy of the monitoring data.

Description

Technical Field

[0001] The present invention relates to the technical field of large-scale neural network model distributed training monitoring, and in particular to a distributed monitoring system and method based on a diffusion model. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning models are becoming more and more complex, requiring a large amount of computing resources for effective training. Especially for large-scale data sets and complex neural network models, a single computing unit (such as a single GPU) is often unable to meet the training requirements. Therefore, distributed training has become one of the key technologies to solve such problems.

[0003] During distributed training, training tasks are assigned to multiple computing nodes for parallel execution to accelerate the training process. However, distributed training systems face many challenges, such as node failures, network latency, uneven resource allocation, and low training efficiency. Current distributed training monitoring tools often lack real-time monitoring and adaptive optimization capabilities and cannot effectively solve the above problems.

[0004] Therefore, the present invention proposes a distributed monitoring system and method based on a diffusion model to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed monitoring system and method based on a diffusion model. By applying an advanced diffusion model to distributed training monitoring, the efficiency and stability of distributed training are effectively improved, the optimal configuration of resources and automated fault prevention are achieved, and problems such as node failures and low training efficiency are solved; system monitoring is achieved by initializing a monitoring module and calling a monitoring function in a training script, and storing and visualizing data through different data formats and tools; the diffusion model not only enhances the elasticity and fault tolerance of the system, but also improves the real-time and accuracy of monitoring data.

[0006] To achieve the above object, the present invention provides a distributed monitoring system and method based on a diffusion model, characterized in that the system comprises: an elastic fault-tolerant training module, a training information visualization module, a GPU performance monitoring module and a forward propagation process monitoring module;

[0007] The specific method includes the following steps:

[0008] S1, based on the denoising diffusion probability model, extract samples from the data distribution of random noise application reverse process;

[0009] S2, using KL divergence to achieve the optimization goal of DDPM to make the generated images conform to the distribution of training data;

[0010] S3, performing noise processing on the optimization target in step S2 by adopting a heavy parameter method;

[0011] S4, during training, calculate the sampling noise estimate, minimize the denoising diffusion objective function, and calculate the standard noise target;

[0012] S5. During inference, reconstruct the noisy samples by iteratively applying the learned denoiser.

[0013] S6. The compression and amplification operations are used to compress the images between different layers. The model complexity is analyzed by writing corresponding analysis codes during the model forward propagation process monitoring.

[0014] S7. Through the training information visualization module, the results of using DDPM for classification tasks are displayed in real time.

[0015] Preferably, the elastic fault-tolerant training module uses a centralized control mechanism so that new nodes can be dynamically added or exited at any time point, and the nodes synchronize their states through a checkpoint mechanism to ensure the consistency and integrity of the training data; based on the dynamic adjustment strategy of the diffusion model, when a node fails or is added, the state of the existing node can be quickly diffused to the new node or the restarted node. Based on Pytorch Elastic fault-tolerant training, nodes are dynamically added or removed in distributed training to achieve automatic restart of training tasks and continue training from the most recent checkpoint to cope with the dynamic addition and exit of computing nodes, ensure the continuity and stability of training, and maintain seamless continuation of the training process.

[0016] Preferably, the training information visualization module has real-time data analysis and graphic display functions, automatically adjusts the granularity and style of data display according to user configuration, and improves the transparency and monitorability of the training process; based on Tensorboard or WandB tools, it monitors and visualizes the training indicators of each GPU in real time, monitors the training status of each node in real time, and presents it through a visual interface, so that users can understand the training process in a timely manner.

[0017] Preferably, the GPU performance monitoring module includes an early warning system, which automatically triggers an alarm and recommends optimization measures when an abnormal GPU performance indicator is detected, to help users quickly respond to potential problems; the module not only records the real-time usage of the GPU, but also classifies and counts the various computing tasks performed by the GPU, analyzes their specific impact on training efficiency, and provides data support for system optimization; the resource requirements between nodes are predicted and adjusted through a diffusion model, to optimize GPU utilization and reduce energy consumption.

[0018] Preferably, the forward propagation process monitoring module includes the function of dynamically tracking the amount of computation of each layer of the model, using layered data collection technology to perform fine-grained monitoring of the calculation of each layer, accurately diagnosing computing bottlenecks and providing targeted optimization suggestions, and using a custom algorithm to calculate and optimize the model's computing path in real time, thereby reducing the overall training time and improving efficiency; by registering pre-hooks and post-hooks at the model layer, monitoring the amount of computation and performance of each layer in the forward propagation process, and real-time statistics and optimization of the model's computing efficiency, so as to optimize the model structure and parameter settings.

[0019] Preferably, in step S1, based on DDPM, samples are extracted from the data distribution of random noise applying the reverse process, and the specific process is as follows:

[0020] In the forward process starting from the conditional distribution d0∶=d, Gaussian noise is gradually added to t∈{1,…,T} to obtain the noisy sample d t , as shown below:

[0021]

[0022] Among them, ∈~N(0,I), And {β1, ..., β T} is the variance table of the T-step process;

[0023] In the reverse process, the conditional denoising model ∈ parameterized by the learned parameters θ θ (·) Gradually remove d t The noise in the t-1 .

[0024] Preferably, in step S2, the optimization goal in DDPM is to make the generated image conform to the distribution of training data as much as possible, and KL divergence is used to achieve this goal. The objective function is as follows:

[0025] argmin θ KL(P data ||P θ ) (2)

[0026] Among them, P data is the distribution of training data images, P θ is the probability distribution of the pictures generated by the model;

[0027] The objective function is changed, and the transformed objective function is as follows:

[0028]

[0029] To maximize the above objective function, push one time_step to all time_steps according to the Markov property, and optimize the objective as follows:

[0030]

[0031] Preferably, in step S3, the expression on the right side of formula (4) is obtained by adding noise using a heavy parameter method:

[0032]

[0033] Among them, in q(d t-1 |d t ,d0) distribution, the variance is a constant that is only related to the hyperparameters; p θ (d t-1 |d t ) is related to the model.

[0034] Preferably, during training in step S4, the parameter θ is updated by obtaining data pairs (x, d) from the training set, denoising d with sampling noise ∈ of a random time step t, and calculating the sampling noise estimate as shown below:

[0035]

[0036] Minimize the denoising diffusion objective function, then the standard noise objective As shown below:

[0037]

[0038] Therefore, the present invention adopts the above-mentioned distributed monitoring system and method based on the diffusion model, and effectively improves the efficiency and stability of distributed training by applying the advanced diffusion model to distributed training monitoring, realizes the optimal configuration of resources and automatic fault prevention, and solves the problems of node failure and low training efficiency; the system is monitored by initializing the monitoring module and calling the monitoring function in the training script, and storing and visualizing the data through different data formats and tools; the diffusion model not only enhances the elasticity and fault tolerance of the system, but also improves the real-time and accuracy of the monitoring data.

[0039] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the composition of the distributed monitoring system based on the diffusion model of the present invention;

[0041] Figure 2It is a flow chart of the distributed monitoring method based on the diffusion model of the present invention;

[0042] Figure 3 This is the workflow of the elastic fault-tolerant training module of the present invention;

[0043] Figure 4 It is a flow chart of the training information visualization algorithm of the present invention;

[0044] Figure 5 is a flow chart of GPU performance monitoring of the present invention;

[0045] Figure 6 It is the details of the forward propagation process monitoring of the present invention;

[0046] Figure 7 It is an architecture diagram of the distributed training monitoring system based on the diffusion model of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0048] Example

[0049] like Figure 1 and Figure 2 As shown, the present invention provides a distributed monitoring system based on a diffusion model, including an elastic fault-tolerant training module, a training information visualization module, a GPU performance monitoring module and a forward propagation process monitoring module.

[0050] A distributed monitoring method based on a diffusion model specifically includes the following steps:

[0051] S1, based on the denoising diffusion probability model, extract samples from the data distribution of random noise application reverse process;

[0052] S2, using KL divergence to achieve the optimization goal of DDPM to make the generated images conform to the distribution of training data;

[0053] S3, performing noise processing on the optimization target in step S2 by adopting a heavy parameter method;

[0054] S4, during training, calculate the sampling noise estimate, minimize the denoising diffusion objective function, and calculate the standard noise target;

[0055] S5. During inference, reconstruct the noisy samples by iteratively applying the learned denoiser.

[0056] S6, the compression (Conv) and amplification (ConvT) operations are used to compress the images between different layers, and the model complexity is analyzed by writing corresponding analysis codes during the model forward propagation process monitoring;

[0057] S7. Through the training information visualization module, the results of using DDPM for classification tasks are displayed in real time.

[0058] like Figure 3 As shown in the figure, the elastic fault-tolerant training module uses a centralized control mechanism (Agent and Rendezvous protocol) to allow new nodes to dynamically join or exit at any time point. The nodes synchronize their states through the checkpoint mechanism to ensure the consistency and integrity of the training data. The dynamic adjustment strategy based on the diffusion model can quickly diffuse the state of the existing nodes to the new nodes or restart the nodes when the nodes fail or join. Based on Pytorch Elastic fault-tolerant training, nodes are dynamically added or removed in distributed training, and the training tasks are automatically restarted and training is continued from the most recent checkpoint to cope with the dynamic addition and exit of computing nodes, ensure the continuity and stability of training, and keep the training process seamless.

[0059] like Figure 4 As shown in the figure, the training information visualization module integrates real-time data analysis and graphic display functions, and can automatically adjust the granularity and style of data display according to user configuration, improving the transparency and monitorability of the training process. Using Tensorboard or WandB as a tool, real-time monitoring and visualization of the training indicators of each GPU, such as loss value, accuracy, and training cycle time. Real-time monitoring of the training status of each node, including key indicators such as loss value, accuracy, and time consumption, and presenting them through a visual interface, allows users to understand the training process in a timely manner.

[0060] like Figure 5 As shown in the figure, the GPU performance monitoring module includes an early warning system. When an abnormal GPU performance indicator is detected, an alarm is automatically triggered and optimization measures are recommended to help users quickly respond to potential problems. This module not only records the real-time usage of the GPU, but also classifies and counts various computing tasks performed by the GPU, analyzes their specific impact on training efficiency, and provides data support for system optimization. The CUDA Profiling Tools Interface (CUPTI) monitors GPU utilization, the ratio of calculations to communication, and the number and duration of calls to operations to help identify and resolve performance bottlenecks. The diffusion model is used to predict and adjust resource requirements between nodes to optimize GPU utilization and reduce energy consumption.

[0061] like Figure 6As shown in the figure, the forward propagation process monitoring module includes the function of dynamically tracking the amount of computation of each layer of the model. It uses layered data collection technology to perform fine-grained monitoring of the computation of each layer, accurately diagnose computational bottlenecks and provide targeted optimization suggestions. It uses a custom algorithm to calculate and optimize the model's computational path in real time, thereby reducing the overall training time and improving efficiency. By registering pre-hooks and post-hooks at the model layer, the floating-point operations (FLOPs) and multiplication-addition operations (MACs) of each layer in the forward propagation process are monitored, and the computational efficiency of the model is counted and optimized in real time, so as to optimize the model structure and parameter settings.

[0062] The entire monitoring system includes an adaptive learning module, which automatically adjusts monitoring parameters and performance optimization strategies based on historical training data and user feedback to adapt to different training environments and model requirements. It adopts a modular design, and each monitoring module can operate independently or work with other modules to facilitate rapid deployment and adjustment according to specific training needs. The entire system is designed with full consideration of user interaction and operational convenience, providing a graphical user interface (GUI) and a command line interface (CLI), so that users can choose different operating methods according to their personal preferences and professional level.

[0063] S1. The present invention designs a powerful generative model, namely the Denoising Diffusion Probistic Models (DDPM), such as Figure 7 As shown, based on DDPM, the diffusion process gradually degrades the image with Gaussian noise, thus drawing samples from the data distribution of random noise applying the reverse process.

[0064] In the forward process starting from the conditional distribution d0∶=d, Gaussian noise is gradually added to t∈{1,…,T} to obtain the noisy sample d t , as shown below:

[0065]

[0066] Among them, ∈~N(0,I), And {β1, ..., β T} is the variance table of the T-step process.

[0067] In the reverse process, the conditional denoising model ∈ parameterized by the learned parameters θ θ (·) Gradually remove d t The noise in the t-1 .

[0068] S2. The optimization goal in DDPM is to make the generated image as consistent as possible with the distribution of training data. KL divergence is used to achieve this goal. The objective function is as follows:

[0069] argmin θ KL(P data ||P θ ) (2)

[0070] Among them, P data is the (probability) distribution of training data (which can also be understood as the real world) images, P θ is the (probability) distribution of images generated by the model.

[0071] The objective function is changed, and the transformed objective function is as follows:

[0072]

[0073] To maximize the above objective function, the essence is to maximize each item in the continuous multiplication. Use log to represent this continuous multiplication item, further decompose and push one time_step to all time_steps according to the Markov property, then optimize the target as follows:

[0074]

[0075] S3. The expression on the right side of the equation is called Evidence Lower Bound (ELBO), which is obtained by adding noise using the heavy parameter method:

[0076]

[0077] Among them, in q(d t-1 |d t ,d0) distribution, the variance is a constant that is only related to the hyperparameters; p θ (d t-1 |d t ) The subscript indicates that the term is relevant to the model.

[0078] In order to make the distribution of p and q close, p needs to learn the mean and variance of q. Since the variance is a constant, in DDPM, it is assumed to be fixed and no longer needs to be learned separately. Therefore, we only need to learn the mean of q and let the model predict the noise ∈ θ By making it close to ∈, we can achieve the goal of generating image distributions that are as similar as possible to the real image distributions.

[0079] S4. During training, the parameter θ is updated by obtaining data pairs (x, d) from the training set, denoising d with sampling noise ∈ of random time step t, and calculating the sampling noise estimate as follows:

[0080]

[0081] Minimize the denoising diffusion objective function, then the standard noise objective As shown below:

[0082]

[0083] S5. At inference time, learn the denoiser ∈ by iterative application θ (d t ,x,t), from the normally distributed variable d t Start reconstructing d0∶=d.

[0084] In DDPM, UNet is used as the core model for predicting noise. It is generally divided into two parts: Encoder and Decoder. In the Encoder part, the UNet model will gradually compress the size of the image; in the Decoder part, it will gradually restore the size of the image. At the same time, a "residual connection" is used between the Encoder and Deocder to ensure that the Decoder part does not lose the information of the previous steps when reasoning and restoring the image information.

[0085] S6. The compression (Conv) and magnification (ConvT) operations are used to compress images between different layers. The model layer operations used, such as convolutional layers and tensor operations, will be monitored during the forward propagation process of the model by writing corresponding analysis codes to analyze the model complexity.

[0086] S7. The results of using DDPM for classification tasks can be displayed in real time in the training information visualization module.

[0087] Therefore, the present invention adopts the above-mentioned distributed monitoring system and method based on the diffusion model, and effectively improves the efficiency and stability of distributed training by applying the advanced diffusion model to distributed training monitoring, realizes the optimal configuration of resources and automatic fault prevention, and solves the problems of node failure and low training efficiency; the system is monitored by initializing the monitoring module and calling the monitoring function in the training script, and storing and visualizing the data through different data formats and tools; the diffusion model not only enhances the elasticity and fault tolerance of the system, but also improves the real-time and accuracy of the monitoring data.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A distributed monitoring method based on a diffusion model, characterized in that: The method is applied to a distributed monitoring system based on a diffusion model, which includes: an elastic fault-tolerant training module, a training information visualization module, a GPU performance monitoring module and a forward propagation process monitoring module; The elastic fault-tolerant training module uses a centralized control mechanism to allow new nodes to dynamically join or leave at any time. The nodes synchronize their states through a checkpoint mechanism to ensure the consistency and integrity of the training data. The dynamic adjustment strategy based on the diffusion model can quickly spread the state of existing nodes to new nodes or restart nodes when a node fails or joins. Based on Pytorch Elastic fault-tolerant training, nodes are dynamically added or removed in distributed training to achieve automatic restart of training tasks and continue training from the most recent checkpoint to cope with the dynamic addition and exit of computing nodes, ensure the continuity and stability of training, and maintain seamless continuation of the training process. The training information visualization module has real-time data analysis and graphic display functions, automatically adjusting the granularity and style of data display according to user configuration, improving the transparency and monitorability of the training process; based on Tensorboard or WandB tools, it monitors and visualizes the training indicators of each GPU in real time, monitors the training status of each node in real time, and presents it through a visual interface, so that users can understand the training process in a timely manner; The GPU performance monitoring module includes an early warning system. When abnormal GPU performance indicators are detected, an alarm is automatically triggered and optimization measures are recommended to help users quickly respond to potential problems. This module not only records the real-time usage of the GPU, but also classifies and counts various computing tasks performed by the GPU, analyzes their specific impact on training efficiency, and provides data support for system optimization. The module predicts and adjusts resource requirements between nodes through a diffusion model to optimize GPU utilization and reduce energy consumption. The forward propagation process monitoring module includes the function of dynamically tracking the amount of calculation of each layer of the diffusion model. It uses layered data collection technology to perform fine-grained monitoring of the calculation of each layer, accurately diagnose the calculation bottleneck and provide targeted optimization suggestions. It uses a custom algorithm to calculate and optimize the calculation path of the model in real time, thereby reducing the overall training time and improving efficiency. By registering pre-hook and post-hook at the diffusion model layer, it monitors the amount of calculation and performance of each layer in the forward propagation process, and counts and optimizes the calculation efficiency of the model in real time, so as to optimize the diffusion model structure and parameter settings. A distributed monitoring method of a distributed monitoring system based on a diffusion model includes the following steps: S1, based on the denoising diffusion probability model, extract samples from the data distribution of random noise application reverse process; S2, using KL divergence to achieve the optimization goal of DDPM to make the generated images conform to the distribution of training data; S3, performing noise processing on the optimization target in step S2 by adopting a heavy parameter method; S4, when training the denoising diffusion probability model, calculating the sampling noise estimate, minimizing the denoising diffusion objective function, and calculating the standard noise target; S5, during the inference of the denoising diffusion probability model, reconstructing the noisy samples by iteratively applying the learned denoiser; S6. Use compression and amplification operations to compress images between different layers, and write corresponding analysis codes to analyze the model complexity during the model forward propagation process monitoring; S7. Through the training information visualization module, the results of using DDPM for classification tasks are displayed in real time.

2. According to claim 1, a distributed monitoring method based on a diffusion model is characterized in that: In step S1, based on the DDPM diffusion model, samples are extracted from the data distribution of random noise applying the reverse process, and the specific process is as follows: From the conditional distribution In the forward process starting at Gaussian noise is gradually added to obtain noisy samples , as shown below: （1）； in, N , ,and is the variance table of the T-step process; In the reverse process, the learned parameters are used Parameterized Conditional Denoising Model Gradually remove The noise in .

3. The distributed monitoring method based on the diffusion model according to claim 1 is characterized in that: In step S2, the optimization goal in DDPM is to make the generated image as consistent as possible with the distribution of training data. KL divergence is used to achieve this goal. The objective function is as follows: （2）； in, is the distribution of training data images, is the probability distribution of the pictures generated by the model; The objective function is changed, and the transformed objective function is as follows: （3）； Among them, to maximize the above objective function, according to the Markov property, one time_step is pushed to all time_steps, and the optimization target is as follows: （4）。 4. The distributed monitoring method based on the diffusion model according to claim 3 is characterized in that: In step S3, the expression on the right side of formula (4) is obtained by adding noise using the heavy parameter method: （5）； Among them, In the distribution, the variance is a constant that is only related to the hyperparameters; Related to the model.

5. The distributed monitoring method based on the diffusion model according to claim 1 is characterized in that: During training in step S4, the parameters By obtaining data pairs from the training set To update, use the sampling noise of random time step t right Perform denoising and calculate the sampling noise estimate as follows: （6）； Minimize the denoising diffusion objective function, then the standard noise objective , as shown below: （7）。

Images