Distributed system performance evaluation method oriented to deep learning load in CENI network

Abstract

The invention discloses a distributed system performance evaluation method for a deep learning load in a CENI network, and provides a customized deep learning load configuration method which can be configured by a user, and supports the user to carry out targeted adjustment and optimization on test load configuration according to a system use scene. A deep learning load deployment and test framework is realized, and efficient compression and automatic distribution of deep learning test mirror images and adaptive generation of gradual change loads and sudden loads are supported; on the basis of the basic monitoring indexes of the system, comprehensive evaluation is carried out on the distributed system from multiple dimensions, and the performance of the system facing set deep learning tasks and loads is obtained. According to the method, distributed system performance evaluation which is highly suitable for a deep learning calculation scene can be realized from three aspects of customized deep learning test load setting, deep learning load deployment and test and distributed system performance comprehensive evaluation.

Description

Technical Field

[0001] The present invention belongs to the field of cloud computing and edge computing, and specifically relates to a distributed system performance evaluation method for deep learning loads in a CENI network. Background Art

[0002] With the continuous advancement of information technology, distributed systems have become an important part of modern computing architecture. By distributing computing tasks to multiple nodes for processing, distributed systems not only improve computing efficiency and system scalability, but also enhance the system's fault tolerance and reliability. Today, distributed systems are widely used in cloud computing, the Internet of Things, big data processing, edge computing and other fields. These systems involve a large number of computing nodes and complex resource scheduling, which puts higher requirements on system performance and resource management.

[0003] The Future Network Experimental Facility CENI is my country's first major national scientific and technological infrastructure in the field of communications and information. It can provide users with low-latency, low-jitter, high-throughput, and highly reliable network transmission services. The CENI network deploys high-performance network nodes and optical fiber backbone links across the country, has flexible resource scheduling and advanced network virtualization capabilities, and can efficiently carry a variety of innovative application loads including artificial intelligence, deep learning, cloud computing, and big data analysis. Based on the CENI network, it is possible to build a cross-regional, high-throughput distributed computing system, eliminate the network transmission bottleneck between remote computing nodes, significantly improve the collaborative computing efficiency of distributed applications, and provide key network support for the connection, operation, and expansion of remote computing nodes.

[0004] Performance testing and load testing are indispensable in the development and operation of distributed systems. However, existing load testing technologies have certain limitations, especially when faced with modern complex distributed systems. Traditional load evaluation methods usually evaluate the performance of the system by simulating a large number of user requests, but these methods often fail to truly reflect the actual performance of the system under high concurrency, complex data transmission, and computing requirements. For example, in scenarios where large-scale data needs to be processed and complex calculations need to be performed, traditional load testing often fails to provide sufficient test depth and accuracy, resulting in the inability to accurately identify possible bottlenecks in the system.

[0005] With the rapid development of deep learning technology, deep learning models have demonstrated powerful application capabilities in many fields, from image recognition, speech processing to natural language processing and recommendation systems, deep learning has become an important tool for solving complex problems. The training and reasoning process of deep learning models usually requires a large amount of computing resources. These computing tasks usually have the characteristics of high concurrency, high data transmission and complex calculations, which puts high demands on the computing and communication performance of the system.

[0006] In distributed systems such as cloud computing and edge computing, containerization technology has become a common deployment method. Container technology allows applications to be packaged together with the environment they depend on, ensuring that they can run consistently in any environment. Container images usually contain the application itself and all necessary dependencies and library files, which makes container images an ideal way to deploy and run applications across environments. However, as the scale of distributed systems continues to expand, the distribution problem of container images has gradually emerged. Especially in edge computing and large-scale distributed systems, the transmission and distribution of container images are often limited by network bandwidth and storage space, resulting in slow distribution speed and affecting the efficiency of system deployment. How to improve the distribution efficiency of container images, especially image distribution between edge nodes, has become an important technical challenge in distributed systems.

[0007] In addition, the changes in load in distributed systems are usually unpredictable, especially in high-load and high-concurrency situations. How to effectively deal with fluctuations in system load has become the key to system expansion. Traditional expansion mechanisms mainly include active expansion and responsive expansion. Active expansion relies on load prediction, predicting load changes based on historical data or trends, and increasing computing resources in advance. Although this method can ensure that the system prepares resources before the load increases, its disadvantage is that inaccurate predictions may lead to waste of resources. Responsive expansion starts the expansion operation when the load actually reaches a certain threshold. Due to the long reaction time of expansion, the system may not respond to load changes in time, and performance may fluctuate. Summary of the invention

[0008] The purpose of the present invention is to propose a distributed system performance evaluation method for deep learning load in CENI network, customize deep learning test according to user usage scenarios, provide efficient compression and automatic distribution of test images, and adaptive generation capabilities of gradual load and sudden load, and on the basis of system basic monitoring indicators, comprehensively evaluate the distributed system from multiple dimensions such as task completion, resource allocation, load balancing, etc.

[0009] The technical solution to achieve the purpose of the present invention is:

[0010] A distributed system performance evaluation method for deep learning load in a CENI network includes the following steps:

[0011] S1: Set the deep learning model file, parse the deep learning model file, extract the deep learning model name and version information, select the corresponding deep learning model on the cloud, and build an image based on the extracted deep learning model name and version information;

[0012] S2: Dynamically delete and hierarchically compress the image, and distribute the compressed image;

[0013] S3: Generate corresponding loads according to different evaluation requirements;

[0014] S4: Collect indicator data after load generation and conduct a comprehensive evaluation of distributed system performance based on the indicator data.

[0015] Preferably, setting the deep learning model file specifically includes setting the model name, model version, running rounds, number of samples processed at a single time, sample data set, load type, cycle time, initial load quantity, load increment quantity, burst load quantity, burst load occurrence time, total test time, and number of nodes.

[0016] Preferably, the types of loads include periodic loads, gradual loads and burst loads; the periodic load refers to a load that repeatedly runs the same number of loads at specified time intervals; the gradual load refers to a load that repeatedly runs a load with a gradually increasing amount of data at specified time intervals; the burst load refers to a load in which the number of samples and the increase in the number of processing times within a limited time interval are both greater than a set threshold.

[0017] Preferably, the specific method for dynamically deleting and hierarchically compressing the image is:

[0018] Start a temporary container for the image and monitor the file access during the temporary container operation in real time; after the temporary container ends, package the accessed files into the new image;

[0019] Merge all image layers in the new image, and use the final merged file view as the only image layer of the new image;

[0020] The merged new image is divided into logical blocks that can be downloaded in parallel and distributed to multiple points synchronously.

[0021] Preferably, the specific method of dividing the merged new image into logical blocks that can be downloaded in parallel and performing multi-point synchronous distribution is:

[0022] Pre-caching image layers or data blocks on distributed system nodes;

[0023] When there is a high-speed interconnection link between the nodes of the distributed system, the nodes that have completed the download can share the image blocks with other nodes.

[0024] Preferably, according to different evaluation requirements, the specific method of generating corresponding loads is:

[0025] Periodic load generation: Set a timer in the distributed system, start a new Pod with a specified number of loads, and start a container in the new Pod. The container runs the test load of the specified deep learning model. Whenever a new time period arrives, the load that has not been completed in the previous period will not stop running, but continue to execute. The scheduler distributes the newly started Pod with a specified number of loads to the distributed system nodes. When the distributed system node task is completed, the Pod is destroyed, releasing the occupied computing resources, and the subsequent test load uses the released computing resources.

[0026] Gradual load generation: Set a timer in the distributed system to start a new Pod with a specified initial load number, and start a container in the new Pod, in which the test load of the specified deep learning model is running; at every specified time period, the number of test loads of the newly started deep learning model gradually increases, and the load number increases by the specified incremental load number compared to the previous time period; at the same time, the scheduler selects nodes to run the newly added deep learning load tasks based on the node resource occupancy;

[0027] Burst load generation: Set a timer in the distributed system, start a new Pod with a specified initial load, and start a container in the new Pod, in which the test load of the specified deep learning model is running. At specified time intervals, initiate a new Pod with a specified initial load; when the evaluation time reaches the specified burst load occurrence time, initiate a new Pod with a specified burst load at one time.

[0028] Compared with the prior art, the present invention has the following significant advantages:

[0029] (1) The deep learning load deployment and testing framework designed by the present invention can support the efficient compression and automatic distribution of deep learning test images, as well as the adaptive generation of gradual loads and sudden loads. It breaks through the limitations of the difficulty of test program deployment and the single form of test requests faced by traditional distributed system testing, and has a high degree of originality.

[0030] (2) The comprehensive evaluation index of distributed system performance proposed in the present invention can comprehensively evaluate the distributed system from multiple dimensions such as task completion, resource allocation, and load balancing on the basis of the basic monitoring indicators of the system. It solves the limitation of a single evaluation index in judging the performance of cloud-edge hybrid distributed computing systems and is highly innovative.

[0031] (3) The user-configurable customized deep learning load generation method adopted by the present invention supports users to make targeted adjustments and optimizations to the test load configuration according to the system usage scenarios, significantly improving the matching degree between the test environment and the generation environment, and providing a method for combining user-defined configuration with system default configuration, thereby simplifying the difficulty of user configuration and having high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of the distributed system performance evaluation method for deep learning load in the CENI network of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0034] like Figure 1 As shown, as an embodiment, a distributed system performance evaluation method for deep learning load in a CENI network includes the following steps:

[0035] Step S1, customized deep learning test load settings:

[0036] The user sets the deep learning model file according to the load requirements, including the required model name, model version, run rounds, number of samples processed at a time, sample data set, load type, cycle time, initial load number, load increment number, burst load number, burst load occurrence time, total test time, and number of nodes. A distributed system node refers to an independent computing unit or device in a distributed system. Each node can run independently and communicate and collaborate with other nodes through the network to complete system tasks together.

[0037] In a further embodiment, the load type is determined according to the test content, specifically:

[0038] In order to test the stability performance of the system in a regular scenario, a periodic load is selected;

[0039] To test the system's elastic expansion and contraction capabilities, select a gradual load change.

[0040] In order to test the system's response efficiency and fault recovery level when facing instantaneous high concurrency or resource preemption, burst load is selected.

[0041] In a further embodiment, the load types include periodic load, gradually changing load and sudden load.

[0042] Periodic load: Repeatedly runs the same amount of load at specified time intervals;

[0043] Gradual load: Repeatedly run a load with gradually increasing data volume at specified time intervals;

[0044] Burst load: A load that runs with a sharp increase in the number of samples and processing times in a short time interval.

[0045] Parse the deep learning model file, extract the deep learning model name and version information, select the corresponding deep learning model in the cloud, and build an image based on the extracted deep learning model name and version information. The load setting file is described in JSON format.

[0046] S2: Efficient compression and fast distribution of load images;

[0047] In traditional image building, the complete Linux distribution and all dependencies are often packaged together, resulting in a large image file size and time-consuming distribution and deployment. To effectively solve this problem, the present invention proposes a triple reduction and transmission solution that integrates "dynamic analysis and pruning + image multi-layer merging + distributed fast distribution" for large loads such as deep learning models. It can not only accurately eliminate useless image dependencies, but also realize fast image distribution to distributed computing nodes, significantly improving the overall transmission and startup efficiency of the system.

[0048] First, the present invention accurately identifies the actual necessary files through the "dynamic analysis and tailoring" method. Specifically, a temporary container is started and the access to files during the operation of the temporary container is monitored in real time; after the temporary container ends, only the accessed files are packaged into the new image, thereby removing redundant content that has never been called or is only used for testing. Compared with traditional solutions that rely on static lists or manually configured black and white lists, dynamic analysis can more accurately capture the library files, model parameters and data dependencies required for deep learning reasoning or training, thereby improving the accuracy of image tailoring.

[0049] Secondly, the present invention implements container multi-layer storage and write-time copy mechanism through the "mirror multi-layer merging" method, merging new mirror layers into a unique layer, thereby further cleaning up redundant information caused by modifying or deleting lower-layer files. Specifically, since the container's modification of lower-layer files does not actually change the underlying files, but generates incremental layers or placeholders to mark deletions, multi-layer stacking will generate a large amount of duplicate data. By merging them into a single layer, redundant files can be effectively eliminated and the final consistent file view can be retained, greatly reducing the image size.

[0050] Finally, in order to further shorten the distribution time of images in a distributed environment, the present invention also proposes a "distributed fast distribution" method for multiple nodes. This method can divide the merged new image into logical blocks that can be downloaded in parallel, and supports multi-point synchronization in the distribution network. Specifically, it includes the following two aspects: one is layered or block pre-pull, that is, pre-caching some commonly used image layers or key data blocks on the distributed system nodes to achieve "local pull to start" fast parallel deployment; the second is peer-to-peer transmission (P2P) or hierarchical relay: when there is a high-speed interconnection link between the nodes of the distributed system, the nodes that have completed the download can share the image blocks with other nodes, reducing the dependence on the central image warehouse and reducing bandwidth and request pressure.

[0051] Through the above three-fold solution, the present invention not only significantly compresses the file size in the image building stage, but also realizes distribution acceleration by utilizing multi-point parallel and peer-to-peer transmission in the distribution link, greatly reducing the resource usage and time overhead in the distributed deployment process.

[0052] Step S3, customized generation of multiple types of evaluation loads;

[0053] In the actual operation of distributed systems, they often face diverse task scenarios such as real-time reasoning and batch training at the same time. The differentiation of task fields will lead to significant differences in load scale, concurrency characteristics and resource requirements. Therefore, customized generation of multiple types of evaluation loads is particularly critical for the comprehensive evaluation of distributed systems.

[0054] Through user-customizable and flexible configuration of deep learning models, load generation time and quantity, it can be closer to real business needs, so as to comprehensively evaluate the adaptability of the system's scheduling algorithms, resource management strategies and fault-tolerance mechanisms under different load forms. Among them, periodic load can test the stability of the system in conventional scenarios, gradual load measures its elastic expansion and reduction capabilities, and burst load is used to evaluate the system's response efficiency and fault recovery level when facing instantaneous high concurrency or resource preemption. With the combination of multiple types of loads and monitoring feedback, it can provide a reliable basis for system optimization and upgrade, and comprehensively improve the stability and high availability of distributed environments.

[0055] (1) Periodic load generation

[0056] The characteristic of periodic load is that in the distributed system being evaluated, the same amount of load is repeatedly initiated according to a specified time period. The specified cycle time (cycle_time), initial load quantity (initial_load) and test load (model_name, model_version) are set by the system evaluation user in step S1.

[0057] Specifically, at the beginning of the evaluation, a timer is set in the distributed system to start a new Pod with a specified number of loads, and a container is started in the new Pod, in which the test load of the specified deep learning model is running. Whenever a new time period is reached, on the one hand, the load that has not been completed in the previous period will not stop running, but will continue to execute. On the other hand, the scheduler will distribute the newly started Pod with a specified number of loads to the appropriate distributed system nodes; when the distributed system node task is completed, the Pod will be destroyed, releasing the occupied computing resources, and subsequent test loads will use the released computing resources. Among them, Pod is the smallest deployable computing unit in the distributed container management platform Kubernetes, which contains one or more containers that share network and storage resources. Through such a load generation mechanism, the performance of the distributed system under periodic stable load conditions can be evaluated.

[0058] (2) Gradual load generation

[0059] The characteristic of the gradual load is that in the distributed system being evaluated, the number of deep learning computing loads initiated to the system gradually increases at specified time intervals. The specified cycle time (cycle_time), initial load quantity (initial_load), incremental load quantity (load_increment) and test load (model_name, model_version) are set by the system evaluation user in step S1.

[0060] Specifically, at the beginning of the evaluation, a timer is set in the distributed system to start a new Pod with a specified initial load, and a container is started in the new Pod, in which the test load of the specified deep learning model is running. After that, the number of newly started deep learning loads will gradually increase at specified time periods, and the number of loads will increase by the specified incremental load number compared to the previous time period; at the same time, the system scheduler will select a suitable node to run the newly added deep learning load tasks based on the node resource occupancy. Through such a load generation mechanism, the performance of the distributed system can be evaluated under the condition of gradually increasing load pressure.

[0061] (3) Burst load generation

[0062] The characteristic of burst load is that, in the distributed system being evaluated, based on the periodic load, a specified number of deep learning computing loads are launched to the system at a specified time point. The cycle time (cycle_time), initial load quantity (initial_load), burst load quantity (burst_load), burst load occurrence time (burst_time) and test load (model_name, model_version) are set by the system evaluation user through step S1.

[0063] Specifically, at the beginning of the evaluation, a timer is set in the distributed system to start a new Pod with a specified initial load, and a container is started in the new Pod, in which the test load of the specified deep learning model is run. After that, a new Pod with a specified initial load will be launched to the system at a specified time interval. On this basis, when the system evaluation time reaches the specified burst load occurrence time, a new Pod with a specified burst load will be launched to the system at one time. Through such a load generation mechanism, the performance of the distributed system in the face of a large number of burst loads can be evaluated.

[0064] Step S4: Comprehensive evaluation of distributed system performance

[0065] Step S401: Obtaining basic indicators of the distributed system

[0066] This step defines and designs methods for obtaining seven basic indicators of distributed systems, including task response time, completion time, node resource utilization, number of node tasks, number of successfully run tasks, total number of tasks, and total number of system nodes, to achieve all-round monitoring of task and node performance during distributed system evaluation, and provide support for subsequent system comprehensive indicator analysis and optimization suggestion generation. The specific basic indicator symbols, basic indicator meanings, and acquisition methods are as follows:

[0067]

[0068] Step S402: Distributed system comprehensive index analysis and optimization suggestion generation

[0069] The purpose of the comprehensive index analysis and optimization suggestion generation of the distributed system is to fully understand the system operation status and locate potential bottlenecks. This step is based on the distributed system foundation obtained in step S401, and further calculates the seven comprehensive indicators of the distributed system, namely, the system response time variance, task completion time variance, node resource utilization balance, load task completion rate, load task completion consistency, cluster load balancing index, and load task balancing index, to comprehensively evaluate the system performance and stability. Based on the analysis results, a flexible and efficient optimization plan is proposed for possible communication delays, node performance differences, uneven resources, and failure risks. This process helps system analysts quickly locate and repair deep-seated problems in distributed systems, and provides a scientific basis for system expansion, upgrades, and overall architecture design. Through continuous monitoring and feedback loops, distributed systems can continue to evolve, maintain high availability and scalability, and achieve stable, efficient, and sustainable service capabilities.

[0070] System response time variance :

[0071]

[0072] in, is the number of tasks, represents the response time of the i-th task, is the mean of the average response time of all tasks. Determine whether the system's task distribution and node response are stable under different load requirements. More than 50ms 2 , it means that the response time fluctuates greatly and there may be a problem of unstable communication delay. The system optimization suggestions generated for this problem are to reduce the fluctuation of communication delay by optimizing network topology, introducing load balancing, reducing cross-node communication, and setting up cache and retry mechanisms to improve response stability.

[0073] Task completion time variance :

[0074]

[0075] in, is the number of tasks, is the time required to complete the i-th task, is the average time required to complete all tasks. More than 20s 2 , there may be large differences in node performance and unstable computing performance. The system optimization suggestions generated for this problem are to dynamically allocate tasks and redundant key tasks through adaptive load balancing mechanism and real-time monitoring of node performance; quickly migrate computing to nodes with surplus resources to ensure overall stability.

[0076] Node resource utilization balance B:

[0077]

[0078] in, is the number of nodes, is the resource utilization of the jth node, is the mean of all node resource utilization. If B is lower than 0.8, it means that there is a large gap in node resource utilization and the task distribution is not balanced. The system optimization suggestion generated for this problem is to adopt a resource-aware scheduling strategy, monitor the real-time load of the node to migrate and balance the task, and perform moderate redundant parallel execution to reduce the overload of individual nodes.

[0079] Load task completion rate CTR:

[0080]

[0081] Where T s Indicates the number of tasks that get results, T t is the total number of tasks, if If it is less than 90%, there may be task configuration errors, insufficient resources, or system failures. The system optimization suggestions generated for this problem are to strengthen configuration verification and automated testing, dynamically expand capacity to cope with insufficient resources, establish monitoring and failover mechanisms, and promptly discover and restore abnormal nodes.

[0082] Load task completion consistency CI:

[0083]

[0084] in, is the number of tasks, is the time required to complete the i-th task, It is the average time required to complete all tasks. When the system completes tasks with the same data set and model information but different nodes, the load execution time is used as a parameter to judge the stability of the system. If the CI is lower than 0.8, it means that the consistency of task completion is poor, and there may be problems such as improper dynamic distribution of resources and disordered task execution order. The system optimization suggestions generated for this problem are to use a centralized scheduler to uniformly distribute tasks, monitor the load of each node and adjust it dynamically; use priority or FIFO queues to ensure execution order consistency and reduce waiting. Through automated resource elastic expansion and contraction and node health detection, ensure reasonable resource allocation, avoid task preemption and disordered order, and improve overall execution consistency.

[0085] Cluster load balancing index (CL):

[0086]

[0087] in, is the number of nodes, is the number of tasks for the jth node, is the average number of tasks on all nodes. This is used to measure the uniformity of the load in the distributed cluster. If the result is greater than N / 10, it means that there is an uneven distribution of tasks among the nodes. The system optimization suggestion generated for this problem is to use the monitoring system to obtain the node load in real time, dynamically schedule and migrate tasks through the load balancing algorithm, avoid overloading of individual nodes, and improve resource utilization.

[0088] Load task balance index TLBI

[0089]

[0090] in, is the number of nodes, is the number of tasks for the jth node, is the average number of tasks for all nodes. If it is less than 0.8 or greater than 1.2, it means that some nodes may be overloaded or underloaded. The system optimization suggestion generated for this problem is to monitor the node load in real time and apply a dynamic scheduling algorithm to reasonably migrate tasks between overloaded nodes and underloaded nodes to maintain the overall load balance of the system.

Claims

1. A distributed system performance evaluation method for deep learning load in CENI network, characterized in that: The following steps are involved: S1: Set the deep learning model file, parse the deep learning model file, extract the deep learning model name and version information, select the corresponding deep learning model on the cloud, and build an image based on the extracted deep learning model name and version information; S2: Dynamically delete and hierarchically compress the image, and distribute the compressed image; S3: Generate corresponding loads according to different evaluation requirements; S4: Collect indicator data after load generation and conduct a comprehensive evaluation of distributed system performance based on the indicator data.

2. The method for evaluating the performance of a distributed system for deep learning loads in a CENI network according to claim 1, characterized in that: Setting up the deep learning model file specifically includes setting the model name, model version, run rounds, number of samples processed at a single time, sample data set, load type, cycle time, initial load quantity, load increment quantity, burst load quantity, burst load occurrence time, total test time, and number of distributed system nodes.

3. The distributed system performance evaluation method for deep learning load in CENI network according to claim 1 is characterized in that: The types of loads include periodic loads, gradual loads and sudden loads; the periodic load refers to a load that repeatedly runs the same amount of loads at specified time intervals; the gradual load refers to a load that repeatedly runs a load with a gradually increasing amount of data at specified time intervals; the sudden load refers to a load that runs a sample quantity and a processing number that increases by more than a set threshold within a limited time interval.

4. The distributed system performance evaluation method for deep learning load in CENI network according to claim 1, characterized in that: The specific method for dynamic deletion and hierarchical compression of images is as follows: Start a temporary container for the image and monitor the file access during the temporary container operation in real time; after the temporary container ends, package the accessed files into the new image; Merge all image layers in the new image, and use the final merged file view as the only image layer of the new image; The merged new image is divided into logical blocks that can be downloaded in parallel and distributed to multiple points synchronously.

5. The method for evaluating the performance of a distributed system for deep learning load in a CENI network according to claim 4, characterized in that: The specific method of dividing the merged new image into logical blocks that can be downloaded in parallel and distributing them to multiple points synchronously is as follows: Pre-caching image layers or data blocks on distributed system nodes; When there is a high-speed interconnection link between the distributed system nodes, the distributed system nodes that have completed downloading are allowed to share the image blocks with other distributed system nodes.

6. The distributed system performance evaluation method for deep learning load in CENI network according to claim 1, characterized in that: According to different evaluation requirements, the specific method of generating corresponding loads is as follows: Periodic load generation: Set a timer in the distributed system, start a new Pod with a specified number of loads, and start a container in the new Pod. The container runs the test load of the specified deep learning model. Whenever a new time period arrives, the load that has not been completed in the previous period will not stop running, but continue to execute. The scheduler distributes the newly started Pod with a specified number of loads to the distributed system nodes. When the distributed system node task is completed, the Pod is destroyed, releasing the occupied computing resources, and the subsequent test load uses the released computing resources. Gradual load generation: Set a timer in the distributed system, start a new Pod with a specified initial load, and start a container in the new Pod, in which the test load of the specified deep learning model is running; at every specified time period, the number of test loads of the newly started deep learning model gradually increases, and the load number increases by the specified incremental load number compared to the previous time period; at the same time, the scheduler selects the corresponding distributed system node to run the newly added deep learning load task according to the resource occupancy of the distributed system node; Burst load generation: Set a timer in the distributed system to start a new Pod with a specified initial load, and start a container in the new Pod, in which the test load of the specified deep learning model is run; initiate a new Pod with a specified initial load at specified time intervals; when the evaluation time reaches the specified burst load occurrence time, initiate a new Pod with a specified burst load at one time.

7. The method for evaluating the performance of a distributed system for deep learning load in a CENI network according to claim 1, characterized in that: Comprehensively evaluate the performance of distributed systems based on indicator data, including: Determining System Response Time Variance : ； in, is the number of tasks, represents the response time of the i-th task, is the mean of the average response time of all tasks; according to Determine whether the system's task distribution and node response are stable under different load requirements. If the first threshold is exceeded, it indicates that there is a problem of unstable communication delay; Calculate the variance of task completion time : ； in, is the number of tasks, is the time required to complete the i-th task, is the mean time required to complete all tasks, if If it is greater than the second threshold, there will be a problem of large node performance differences and unstable computing power performance; Compute node resource utilization balance B: ； in, is the number of nodes, is the resource utilization of the jth node, is the mean of all node resource utilization. If B is lower than the third threshold, it means that there is a large gap in node resource utilization and the task distribution is not balanced. Load task completion rate CTR: ； Among them, T s Indicates the number of tasks that get results, T t is the total number of tasks, if If it is less than the fourth threshold, there is a problem of task configuration error, insufficient resources, or system failure; Calculate the load task completion consistency CI: ； in, is the number of tasks, is the time required to complete the i-th task, It is the average time required to complete all tasks. When completing tasks with the same data set and model information but different nodes, the load execution time is used as a parameter to judge the stability of the system. If CI is lower than the fifth threshold, it means that there is improper dynamic distribution of resources and disordered task execution order. Calculate the cluster load balancing index: ； in, is the number of nodes, is the number of tasks for the jth node, is the average number of tasks on all nodes, which is used to measure the uniformity of the load in the distributed cluster. If the result is greater than N / 10, it means that there is an uneven distribution of tasks among the nodes. Calculate the load task balance index TLBI ； in, is the number of nodes, is the number of tasks for the jth node, is the mean number of tasks for all nodes, if If it is less than the sixth threshold or greater than the seventh threshold, it means that some nodes are overloaded or underloaded, and the sixth threshold is less than the seventh threshold.

Images