Cluster performance prediction method, apparatus and computing device

Abstract

Provided in the present application is a cluster performance prediction method, the method comprising: acquiring information input by a user, the information comprising detailed information of a large-scale cluster to undergo prediction and a large task to be executed on the large-scale cluster; constructing a small-scale cluster and constructing a small task running on the small-scale cluster, wherein the number of devices in the small-scale cluster is less than the number of devices in the large-scale cluster, the interconnection mode of the devices in the small-scale cluster is the same as the interconnection mode of the devices in the large-scale cluster, and the load of each device in the small-scale cluster when running the small task is the same as the load of each device in the large-scale cluster when running the large task; when the small-scale cluster executes the small task, collecting performance data of the small-scale cluster; and using the performance data of the small-scale cluster as performance data of the large-scale cluster when executing the large task. The method can reduce the cost of performance prediction for large-scale clusters, and improve the prediction efficiency and accuracy.

Description

Cluster performance prediction methods, devices, and computing equipment

[0001] This application claims priority to Chinese Patent Application No. 202410818051.9, filed on June 21, 2024, entitled "Method, Apparatus and Computing Device for Model Performance Prediction", and Chinese Patent Application No. 202411059831.6, filed on August 2, 2024, entitled "Method, Apparatus and Computing Device for Cluster Performance Prediction", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud computing, and more specifically, to a method, apparatus, and computing device for predicting cluster performance. Background Technology

[0003] With the continuous advancement and improvement of artificial intelligence technology, deep neural network models have been widely applied in fields such as artificial intelligence generated content (AIGC), natural language processing, computer vision, and speech recognition. Recent research and applications have shown that increasing the model size can further enhance its ability to solve complex tasks.

[0004] As model size increases, the computational and storage requirements for model training grow exponentially. Distributed and parallel training using large-scale hardware clusters has become an essential mode for training large models. Therefore, the industry has proposed building training performance prediction and analysis systems to predict and analyze the training performance of specific training tasks, distributed parallel strategies, and optimization strategies on specific training clusters at a cost far lower than that of actually running training tasks. This allows for the calculation of training task costs, assistance to users in adjusting training strategies, selecting training clusters, and developing new optimized parallel strategies.

[0005] One related technical solution involves predicting performance by collecting the runtime of each step through a profile during specific steps. However, this approach is costly. Another related technical solution uses an analytical model to predict training performance, but this requires building the analytical model separately, making its implementation complex.

[0006] Therefore, how to reduce the cost of prediction and improve the efficiency and accuracy of prediction in the process of predicting the performance of training clusters has become an urgent technical problem to be solved. Summary of the Invention

[0007] This application provides a method for predicting cluster performance, which can reduce the cost of prediction and improve the efficiency and accuracy of prediction in the process of predicting the performance of training clusters.

[0008] Firstly, a method for predicting cluster performance is provided. This method includes: acquiring user input information, including details of a large-scale cluster to be predicted and a large task to be executed on the large-scale cluster; constructing a small-scale cluster based on the details of the large-scale cluster, wherein the number of devices in the small-scale cluster is less than the number of devices in the large-scale cluster, and the interconnection method between devices in the small-scale cluster is the same as that between devices in the large-scale cluster; constructing a small task to run on the small-scale cluster based on the details of the large-scale cluster, the large task, and the constructed small-scale cluster, wherein the load on each device in the small-scale cluster when running the small task is the same as the load on each device in the large-scale cluster when running the large task; collecting performance data of the small-scale cluster during the execution of the small task; and using the performance data of the small-scale cluster as the performance data of the large-scale cluster when executing the large task.

[0009] In the above technical solution, since the load running on each device is largely similar, in a distributed scenario, it is not necessary to run the entire large-scale cluster. Only data from a representative subset of devices within the large-scale cluster needs to be collected. This is equivalent to obtaining the performance data of the entire large-scale cluster while executing a task, thereby reducing the cost of predicting the performance of a large-scale cluster and improving the accuracy of the prediction.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the large task to be executed on a large-scale cluster includes the following information: the structural information of the model to be executed on the large-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: the distributed strategy of the model, and optimization strategy information.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the small task executed on the small-scale cluster includes the following information: the structural information of the model executed on the small-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: distributed strategy information and optimization strategy information.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the distributed strategy includes synchronous parallelism, the large-scale cluster and the small-scale cluster include a first device, and the method further includes: determining the amount of data transmitted by the first device in the small-scale cluster based on the bandwidth of the first device in the large-scale cluster, the amount of data transmitted in the large-scale cluster, and the bandwidth of the small-scale cluster.

[0013] In the above technical solutions, if the distributed parallel strategy used by the small-scale cluster and the large-scale cluster is synchronous parallelism, in order to predict and obtain more accurate performance data of the large-scale cluster, the embodiments of this application can also modify the communication volume (i.e. the amount of data transmitted) of each device in the small-scale cluster during the communication process, so that the communication operation time when the small-scale cluster performs the training task is theoretically the same as the operation time when the large-scale cluster performs the training task, thereby further improving the accuracy of the performance prediction of the large-scale cluster.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the distributed strategy includes pipelined parallelism, and the method further includes: determining the number of additional computational processes in the small-scale cluster based on the difference between the number of equipment groups participating in the pipeline process in the large-scale cluster and the number of equipment groups participating in the pipeline process in the small-scale cluster.

[0015] In the above technical solutions, if the distributed parallel strategy used by the small-scale cluster and the large-scale cluster is synchronous parallelism, in order to predict and obtain more accurate performance data of the large-scale cluster, the embodiments of this application can also add a corresponding number of calculation processes in the small-scale cluster, so that the communication operation time when the small-scale cluster performs the training task is theoretically the same as the operation time when the large-scale cluster performs the training task, thereby further improving the accuracy of the performance prediction of the large-scale cluster.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the distributed strategy includes pipeline parallelism and synchronous parallelism, and the method further includes adding a communication process between groups of devices participating in the pipeline process in the small-scale cluster.

[0017] In the above technical solutions, if the distributed parallel strategy used by the small-scale cluster and the large-scale cluster is a mixture of pipeline parallelism and synchronous parallelism, the embodiments of this application can also add a communication process between the equipment groups participating in the pipeline process in the small-scale cluster, so that the communication operation time when the small-scale cluster performs the training task is theoretically the same as the operation time when the large-scale cluster performs the training task, thereby further improving the accuracy of the performance prediction of the large-scale cluster.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the detailed information of the large-scale cluster includes at least one of the following: the number of devices contained in the large-scale cluster, the specifications of the devices, and the interconnection method between the devices.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the user-input information is detailed information about a small-scale cluster, which includes at least one of the following: the number of devices in the small-scale cluster, the specifications of the devices, and the interconnection method between the devices.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the performance data of the small-scale cluster includes at least one of the following: device resource usage and execution time.

[0021] Secondly, a cluster performance prediction device is provided, comprising: an acquisition module, a construction module, a collection module, and a prediction module. The acquisition module acquires user input information, including details of the large-scale cluster to be predicted and the large task to be executed on the large-scale cluster. The construction module constructs a small-scale cluster based on the details of the large-scale cluster, wherein the number of devices in the small-scale cluster is less than the number of devices in the large-scale cluster, and the interconnection method between devices in the small-scale cluster is the same as that between devices in the large-scale cluster. The construction module also constructs a small task to run on the small-scale cluster based on the details of the large-scale cluster, the large task, and the constructed small-scale cluster, wherein the load on each device in the small-scale cluster when running the small task is the same as the load on each device in the large-scale cluster when running the large task. The collection module collects performance data of the small-scale cluster during the execution of the small task. The prediction module uses the performance data of the small-scale cluster as the performance data of the large-scale cluster when executing the large task.

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the large task to be executed on the large-scale cluster includes the following information: the structural information of the model to be executed on the large-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: the distributed strategy of the model, and optimization strategy information.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the small task executed on the small-scale cluster includes the following information: the structural information of the model executed on the small-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: distributed strategy information and optimization strategy information.

[0024] In conjunction with the second aspect, in some implementations of the second aspect, the distributed strategy includes synchronous parallelism, the large-scale cluster and the small-scale cluster include a first device, and the device further includes: a determining module, configured to determine the amount of data transmitted by the first device in the small-scale cluster based on the bandwidth of the first device in the large-scale cluster, the amount of data transmitted in the large-scale cluster, and the bandwidth of the small-scale cluster.

[0025] In conjunction with the second aspect, in some implementations of the second aspect, the distributed strategy includes pipelined parallelism, and a determining module is used to determine the number of additional computational processes in the small-scale cluster based on the difference between the number of equipment groups participating in the pipeline process in the large-scale cluster and the number of equipment groups participating in the pipeline process in the small-scale cluster.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, the distributed strategy includes pipeline parallelism and synchronous parallelism, and a module is used to increase communication between groups of devices participating in pipeline processes in the small-scale cluster.

[0027] In conjunction with the second aspect, in some implementations of the second aspect, the detailed information of the large-scale cluster includes at least one of the following: the number of devices contained in the large-scale cluster, the specifications of the devices, and the interconnection method between the devices.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, the user-input information is detailed information about a small-scale cluster, which includes at least one of the following: the number of devices in the small-scale cluster, the specifications of the devices, and the interconnection method between the devices.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, the performance data of the small-scale cluster includes at least one of the following: device resource usage and execution time.

[0030] It should be understood that the beneficial effects of the second aspect and its various implementations can be found in the first aspect and its various implementations, and will not be elaborated here.

[0031] Thirdly, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, causing the program to execute the method of the first aspect or any possible implementation thereof.

[0032] Optionally, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0033] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method of the first aspect or any possible implementation thereof.

[0034] Fifthly, a chip is provided that acquires and executes instructions to implement the methods described in the first aspect and any implementation thereof.

[0035] Optionally, as one implementation, the chip includes a processor and a data interface, through which the processor reads instructions stored in the memory and executes the methods in the first aspect and any implementation thereof.

[0036] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect and any implementation thereof.

[0037] In a sixth aspect, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the methods described in the first aspect and any implementation thereof.

[0038] In a seventh aspect, a computer program product containing instructions is provided, which, when run by a cluster of computing devices, cause the cluster of computing devices to perform the methods described in the first aspect and any implementation thereof.

[0039] Eighthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a computing device, perform the method as described in the first aspect and any implementation thereof.

[0040] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0041] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium.

[0042] A ninth aspect provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method as described in the first aspect and any implementation thereof.

[0043] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0044] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0045] Figure 1 is a schematic block diagram of a cloud scenario applicable to an embodiment of this application.

[0046] Figure 2 is a schematic flowchart of a cluster performance prediction method provided in an embodiment of this application.

[0047] Figure 3 is a schematic diagram of a large-scale cluster (training devices 1-4) performing model training tasks in a pipelined parallel manner according to an embodiment of this application.

[0048] Figure 4 is a schematic diagram of a small-scale cluster (training devices 3-4) performing model training tasks in a pipelined parallel manner according to an embodiment of this application.

[0049] Figure 5 is a schematic diagram of another large-scale cluster (training devices 1-4) performing model training tasks in a pipelined parallel manner according to an embodiment of this application.

[0050] Figure 6 is a schematic diagram of another small-scale cluster (training device 0, training device 2) performing model training tasks in a pipelined parallel manner according to an embodiment of this application.

[0051] Figure 7 is a schematic block diagram of a cluster performance prediction device 700 provided in an embodiment of this application.

[0052] Figure 8 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application.

[0053] Figure 9 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application.

[0054] Figure 10 is a schematic diagram of the connection between computing devices 1500A and 1500B via a network according to an embodiment of this application. Detailed Implementation

[0055] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0056] This application will present various aspects, embodiments, or features relating to systems comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0057] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0058] In the embodiments of this application, "corresponding" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, their intended meanings are consistent.

[0059] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0060] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0061] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0062] With the continuous advancement and improvement of artificial intelligence technology, deep neural network models have been widely applied in fields such as artificial intelligence generated content (AIGC), natural language processing, computer vision, and speech recognition. Recent research and applications have shown that increasing the model size can further enhance its ability to solve complex tasks.

[0063] However, as model size increases, the computation and storage required for model training grow exponentially. In the past two years, the computational cost of training common large models such as generative pre-trained transformers (GPT) and bidirectional encoder representations from Transformers (BERT) has increased several hundredfold. The growth rate of model size far exceeds the development speed of hardware specifications, and a single training device is completely unable to handle current model training tasks. Therefore, distributed, parallel training using large-scale hardware clusters has become an essential mode for training large models.

[0064] However, distributed parallel training of large models introduces additional overhead, such as the overhead of synchronization between training devices. This overhead increases exponentially with the size of the model and cluster, further increasing training time and cost, and even becoming a major bottleneck in distributed training. Therefore, improving distributed training performance (e.g., model training speed, resource utilization) and reducing the cost of distributed training have become key issues restricting the development of the field of artificial intelligence.

[0065] As an example, the performance of distributed training of deep learning models is generally affected by the following factors: training cluster, distributed parallel training strategy, and training optimization strategy.

[0066] 1. Training Cluster

[0067] A training cluster refers to a cluster of devices that connect training equipment (such as graphics processing units (GPUs) and neural processing units (NPUs)) and their host computers through interconnected devices, forming a cluster capable of executing distributed training tasks. The capacity / performance and communication bandwidth of the device cluster are fundamental to improving the performance of distributed training.

[0068] Typically, the training devices in a device cluster are homogeneous, meaning they all have the same hardware specifications. This simplifies cluster management, distributed parallelism, and the development and selection of optimization strategies. While some current work explores heterogeneous cluster training scenarios, their ultimate goal is also to build an adaptive task allocation module to make the workload on each device as similar as possible, achieving a similar homogeneous allocation.

[0069] 2. Training distributed parallel strategies

[0070] Distributed parallel training strategies generally refer to partitioning the memory-intensive components—such as the model, its inputs, and intermediate values ​​generated during training—and distributing them across multiple training devices. During training, each device performs computations independently and in parallel based on its local storage, while periodically communicating with other devices to synchronize its parameters and results. By configuring distributed parallel training strategies, the results of distributed computations can be identical to those of non-distributed execution, enabling the collaborative processing of a single task across multiple training devices. Due to the communication and synchronization involved, strategies typically aim to distribute the load across all training devices as evenly as possible; otherwise, mutual waiting between devices can occur, reducing system efficiency.

[0071] For example, distributed parallelism strategies can include, but are not limited to: data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), and sequence parallelism (SP). The following is an explanation of each distributed parallelism strategy.

[0072] A) Data parallelism (DP) is an important strategy in distributed model training, primarily used to process data in parallel across multiple training devices (such as GPUs) to accelerate the model training process. The core idea of ​​data parallelism is to replicate a complete model on each training device (such as a GPU), with each device processing a subset of the dataset (i.e., one or more batches of data) and independently calculating gradients. Then, the gradients from all training devices are aggregated (usually through some form of communication mechanism, such as a parameter server or AllReduce operation) to update the global model parameters. In this way, data parallelism can utilize the computational resources of multiple computing devices to process more data simultaneously, thereby accelerating model training.

[0073] B) Tensor Model Parallelism (TP) is another important model parallelism strategy in distributed training, particularly suitable for training large-scale deep learning models. TP distributes the tensors in the model across different training devices, allowing each device to compute only a portion of the model, thus reducing the computational burden and memory consumption of a single device. Specifically, it divides the model parameters into multiple tensors, each distributed to a different training device for computation. After computing its assigned tensor, each training device passes the results to other devices for further computation or aggregation.

[0074] C) Pipeline parallelism (PP) is another important parallel strategy in distributed model training, especially effective when dealing with large-scale deep learning models. Pipeline parallelism distributes different layers (or modules) of a large model across different training devices, forming a pipeline. Each training device is responsible for processing a portion of the model. During training, data flows sequentially through each training device in a pipelined manner. After each training device completes the computation of its assigned model layer, it passes the result to the next training device. In this way, pipeline parallelism can significantly improve the training efficiency of the model while reducing the memory and computational burden on individual training devices.

[0075] D) Sequence parallelism (SP) is a technique for training and inferring extremely long sequence models. It splits the input sequence dimension to extend the model's parallel processing capabilities. By splitting the input sequence dimension, sequence parallelism distributes the computational task across multiple training devices, thus effectively processing extremely long sequence data. This technique has become an important tool in current research and practice for handling large-scale model long-sequence problems.

[0076] 3. Training optimization strategies

[0077] Training optimization strategies generally refer to accelerating the efficiency of training devices when performing specific training workloads by optimizing computation, memory, and communication, such as achieving faster computation and consuming less memory for the same task. In distributed training, computation and memory optimization strategies typically optimize the load on a single training device. Furthermore, due to the homogeneity of the cluster, the training optimization strategies are generally the same on each training device.

[0078] Currently, numerous parallel and optimization techniques have emerged for distributed training of large models, such as recomputation, memory / computation offloading, and operator optimization. However, the introduction of these techniques further increases the complexity of the training system, and different techniques may be complementary or conflicting, making it difficult for users to evaluate which to choose. Furthermore, the hardware of training devices in the training cluster is constantly iterating; for example, new models of GPUs and NPUs are released almost every year, bringing significant performance improvements and architectural updates. Often, a high-performance parallel or optimization solution on one training cluster hardware cannot fully utilize the corresponding hardware capabilities in another cluster. At the same time, the enormous overhead of large model training tasks also drives up the cost of trial and error. If a training task is misconfigured, the cost, whether it's the extended training time or the expense of restarting the training task, is unacceptable to users.

[0079] Therefore, the industry has proposed building a training performance prediction and analysis system to predict and analyze the training performance of specific training tasks, distributed parallel strategies, and optimization strategies on specific training clusters at a cost far lower than that of actually running training tasks. This allows for the calculation of training task costs, assistance to users in adjusting training strategies, selecting training clusters, and developing new optimized parallel strategies.

[0080] In one related technical solution, during the execution of specific steps, the runtime of each step is collected through profiles based on the data acquisition capabilities of the underlying hardware driver, thereby obtaining performance predictions. On the one hand, its cost is relatively low due to the need for a complete training environment and the same startup overhead as real training. On the other hand, profile data collection incurs certain hardware overhead and cannot accurately reflect actual performance, which is more pronounced when hardware devices are heavily used. The results collected by the profile cannot be used as absolutely accurate predictions and can only be used for comparative analysis.

[0081] Another related technical solution uses analytical models to predict training performance. Analytical models can be completely independent of the actual training hardware and software, performing performance prediction offline. An analytical model is based on algorithms, software, hardware architecture, implementation principles, mathematical formulas, etc., implementing independent logic to simulate the specific behaviors of the algorithms, software, and hardware during the training task, thereby modeling the performance of the training task from the bottom up. However, the drawback of analytical models is that they are usually too complex. Furthermore, building an analytical model requires a deep theoretical background in the entire stack of algorithms, software, and hardware. If an error is made in a single detail of the modeling, it can have a ripple effect, and such complex analytical models are difficult to debug and troubleshoot, further increasing the difficulty of applying analytical models.

[0082] Another related technical solution uses a lightweight profile simulator to predict training performance. This simulator collects real hardware execution information, such as operator execution time, through profile analysis. It then analyzes the model to model different upper-layer software and algorithm optimization schemes, combining the accuracy, ease of use, and flexibility of the analysis model. However, the profile data used in the simulator solution often cannot cover all scenarios in actual training. With changes in model, batch size, and splitting strategy, the tensor shapes and sizes of the underlying computational communication operators vary significantly with each training task. Therefore, to achieve a good model, it is necessary to profile the computation results of a large number of operators with different tensor shapes and sizes, making the profile process complex and costly. Furthermore, the types of operators are constantly evolving, and pre-configured offline operator profile results may not cover the modeling needs of new models and optimization strategies. The training phase uses many complex operators and fusion operators that were not encountered during the profiling process. The modeling of each special operator is also different. When developing new technologies, modeling cannot be performed immediately. Instead, a complex profiling process needs to be run, which increases the complexity of the simulator solution.

[0083] In view of this, embodiments of this application provide a method for predicting cluster performance. Since the load running on each training device is largely similar, in a distributed training scenario, it is not necessary to run the entire training cluster; only data from a representative subset of training devices within the cluster needs to be collected. This is equivalent to obtaining the performance data of the entire training cluster when executing a task, thereby reducing the cost of predicting the performance of the training cluster and improving the accuracy of the prediction.

[0084] The following are some scenarios in which the cluster performance prediction method provided in the embodiments of this application is used.

[0085] Scenario 1: GPU-NPU migration performance evaluation

[0086] As an example, post-migration performance is a crucial factor influencing migration decisions when users migrate hardware devices. Typically, achieving the desired post-migration performance requires running actual training tasks, which necessitates building a pre-built NPU cluster, resulting in high costs. Using the solution of this invention, performance predictions based on real hardware and software can be obtained on a smaller NPU cluster, and the prediction logic is easy for users to understand.

[0087] Scenario 2: Training scheme optimization

[0088] As an example, numerous training optimization and distributed technologies are emerging, and these technologies may conflict with each other, potentially at the hardware level, making them difficult for users to perceive when making choices. This solution allows for performance predictions based on real hardware under specific combinations of training optimization techniques at a relatively low cost. After making predictions using different combinations, users can select the optimal training scheme and run it on large-scale clusters, reducing trial-and-error costs.

[0089] Scenario 3: Training Cost Prediction

[0090] As an example, after obtaining performance prediction results, users can directly assess the time required to run real tasks on a real cluster. For cloud scenarios, the time multiplied by the resource specifications equals the cost for the user to implement this strategy.

[0091] Scenario 4: Evaluation of the new optimization scheme

[0092] As an example, specific performance optimization schemes often exhibit different performance characteristics under different model sizes, which are related to factors such as the actual execution order of various operators, hardware scheduling strategies, and operator input shapes. Currently, during the development phase, due to cost considerations, optimization schemes generally only utilize smaller clusters to run smaller tasks. This results in many optimization schemes not being deeply analyzed for large-scale task scenarios, leading to performance issues in real-world large-scale task environments. However, using modeling raises concerns that the modeling may not accurately reflect the optimization techniques. Based on the performance prediction of this scheme, we can directly obtain the actual performance of the new performance optimization scheme under real-world large-scale task loads, providing strong analytical support for the new optimization scheme.

[0093] It should be noted that the above scenarios are merely illustrative examples and are not intended to limit the application scenarios of the embodiments of this application. The cluster performance prediction method provided in the embodiments of this application can be used in any scenario where cluster performance needs to be predicted.

[0094] In one possible implementation, the cluster performance prediction method provided in this application embodiment can be applied to a cloud service scenario, where the method is executed by a cloud management platform within the cloud service scenario. For ease of description, the cloud service scenario will be described in detail below with reference to Figure 1.

[0095] Figure 1 is a schematic block diagram of a cloud scenario applicable to an embodiment of this application. As shown in Figure 1, the cloud scenario may include: a cloud management platform 110, the Internet 120, and a client 130.

[0096] As shown in Figure 1, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each cloud data center includes multiple servers, and each server includes cloud service resources to provide corresponding cloud services to tenants.

[0097] The cloud management platform 110 can be located in a cloud data center and provides access interfaces (such as user interfaces or application program interfaces, APIs). Tenants can use client 130 to remotely access the cloud management platform 110, register a cloud account and password, and log in. After successful authentication of the cloud account and password, the tenant can further select and purchase virtual machines of specific specifications (processor, memory, disk) on the cloud management platform 110. After successful purchase, the cloud management platform 110 provides the remote login account and password for the purchased virtual machine, allowing client 130 to remotely log in and install and run the tenant's applications. Therefore, tenants can create, manage, log in to, and operate virtual machines in the cloud data center through the cloud management platform 110. Virtual machines can also be referred to as Elastic Compute Service (ECS) or Elastic Instances (different cloud service providers may use different names).

[0098] It should be understood that cloud service tenants can be individuals, businesses, schools, hospitals, government agencies, etc.

[0099] The cloud management platform 110 includes, but is not limited to, a user console, compute management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interaction with tenants. The compute management services manage servers running virtual machines and containers, as well as bare metal servers. The network management services manage network services (such as gateways and firewalls). The storage management services manage storage services (such as data bucket services). The authentication services manage tenant account passwords. The image management services manage virtual machine images. Tenants can log in to the cloud management platform 110 via client 130 and the internet 120 to manage their rented cloud services.

[0100] The following description, with reference to Figure 2, details a method for predicting cluster performance provided in this application. It should be understood that the examples in Figure 2 are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments to the specific values ​​or scenarios illustrated in Figure 2. Those skilled in the art can obviously make various equivalent modifications or variations based on the examples given below in Figure 2, and such modifications and variations also fall within the scope of the embodiments of this application.

[0101] Figure 2 is a schematic flowchart of a cluster performance prediction method provided in an embodiment of this application. As shown in Figure 2, the method includes steps 210-250, which will be described in detail below.

[0102] Step 210: Obtain the model training information input by the user.

[0103] In this embodiment of the application, user input information can be obtained, which may include, but is not limited to: detailed information of the large-scale cluster to be predicted, and the large task to be executed on the large-scale cluster.

[0104] The large tasks to be executed on the large-scale cluster mentioned above include the following information: the structural information of the model to be executed on the large-scale cluster, and the running parameters of the model to be executed on the large-scale cluster. The running parameters of the model to be executed on the large-scale cluster include at least one of the following: the distributed strategy and optimization strategy information of the model to be executed on the large-scale cluster.

[0105] It should be understood that the aforementioned major task is a task conceived by the user and to be executed.

[0106] As an example, the large-scale cluster to be predicted mentioned above refers to a large-scale cluster whose performance needs to be predicted.

[0107] It should be understood that the aforementioned large-scale cluster refers to a conceptualized cluster capable of performing the aforementioned large-scale tasks. That is, this large-scale cluster does not actually exist, nor does it actually run the aforementioned large-scale tasks.

[0108] As an example, the detailed information of the large-scale cluster to be predicted includes, but is not limited to: the number of training devices contained in the large-scale cluster to be predicted, the specifications of each training device, and the interconnection method between the training devices.

[0109] It should be understood that the aforementioned training equipment refers to the equipment that performs the training process of the model, which may include, but is not limited to, GPUs, NPUs, etc.

[0110] As an example, the aforementioned small-scale cluster to be built refers to constructing a small-scale cluster that meets the requirements based on the large-scale cluster to be predicted. The number of training devices in the constructed small-scale cluster is less than the number of training devices in the large-scale cluster to be predicted, and the interconnection method between devices in the small-scale cluster is the same as the interconnection method between devices in the large-scale cluster.

[0111] Optionally, users can also enter details about the small-scale cluster to be built.

[0112] As an example, the details of the small-scale cluster to be built include, but are not limited to: the number of training devices included in the small-scale cluster to be built, the specifications of each training device, and the interconnection method between each training device.

[0113] As an example, the definition information of the above model includes, but is not limited to, the model's weight information.

[0114] Step 220: Construct the small-scale cluster based on the details of the large-scale cluster.

[0115] In this embodiment of the application, after obtaining the information input by the user, a small-scale cluster that meets the requirements can be constructed based on the detailed information of the large-scale cluster to be predicted.

[0116] In one possible implementation, a small-scale cluster that meets the requirements can be constructed based on the number of training devices, the specifications of each training device, and the interconnection method between the training devices contained in the details of the large-scale cluster. This small-scale cluster contains fewer training devices than the large-scale cluster to be predicted, and the interconnection method between the devices in the small-scale cluster is the same as that between the devices in the large-scale cluster.

[0117] It should be noted that, in order to use the performance data of the small-scale cluster when performing tasks as the performance data of the large-scale cluster when performing tasks, and thus to predict the performance of the large-scale cluster, the small-scale cluster constructed in this embodiment is the same as the large-scale cluster in all aspects except that the number of training devices in the small-scale cluster is less than that in the large-scale cluster. For example, the specifications of each training device in the small-scale cluster and the interconnection method between each training device are the same as those in the large-scale cluster.

[0118] To further improve the accuracy of predicting the performance of large-scale clusters, the interconnection method between the training devices in the small-scale cluster constructed above needs to include the various communication interconnection methods in the large-scale cluster, so as to accurately predict or simulate the performance of the large-scale cluster.

[0119] It should be understood that, taking training devices including GPU cards as an example, the above interconnection methods include interconnection of GPU cards within a host and interconnection of GPU cards between different hosts.

[0120] Step 230: Based on the details of the large-scale cluster, the large task, and the small-scale cluster being built, construct the small task to run on the small-scale cluster.

[0121] In this embodiment of the application, a small task can also be constructed to run on a small-scale cluster based on the detailed information of the large-scale cluster, the large task to be executed on the large-scale cluster, and the small-scale cluster constructed.

[0122] The small tasks executed on the aforementioned small-scale clusters include the following information: the structural information of the model executed on the small-scale cluster, and the running parameters of the model executed on the small-scale cluster. The running parameters of the model executed on the small-scale cluster include at least one of the following: distributed strategy information and optimization strategy information.

[0123] In this embodiment of the application, by constructing small tasks that run on a small-scale cluster, the load of each device in the small-scale cluster when running small tasks is the same as the load of each device in the large-scale cluster when running large tasks. Thus, the performance data of the large-scale cluster can be accurately predicted based on the performance data of the small-scale cluster.

[0124] It should be noted that the small tasks constructed above and running on small-scale clusters have no specific training significance and generally do not conform to the logic of deep learning training. They are designed specifically to simulate the performance of large tasks.

[0125] Step 240: Collect performance data of the small cluster while executing small tasks in the small cluster.

[0126] In this embodiment of the application, after constructing the aforementioned small-scale cluster and the small task running on the small-scale cluster, the small task can be executed on the small-scale cluster.

[0127] This application embodiment can also collect performance data of a small-scale cluster during the execution of the small task in a small-scale cluster.

[0128] One possible implementation is to leverage the data acquisition (profile) capabilities of the basic training framework to collect real performance data of a small-scale cluster during the execution of the aforementioned small tasks.

[0129] The performance data mentioned above can also be called profile data. Profile data typically includes a series of detailed information related to model performance, resource usage, and execution status. This data is crucial for analyzing model behavior, identifying performance bottlenecks, and optimizing model performance.

[0130] For example, the profile data of the small-scale cluster collected above may include, but is not limited to: resource usage information of the training device when the small-scale cluster performs the above-mentioned small task, execution time required by the small-scale cluster to perform the above-mentioned small task, etc.

[0131] For example, the resource usage information of the training device when the above-mentioned large-scale cluster performs the small task includes, but is not limited to, the resource utilization rate and memory usage of the training device. The execution time required by the above-mentioned small-scale cluster to perform the small task includes, but is not limited to, the execution time of a single operation step (including the forward propagation process, the back propagation process, and the gradient update process) during the execution of the small training task by the small-scale cluster.

[0132] Step 250: Predict the performance data of the large-scale cluster based on the performance data of the small-scale cluster collected.

[0133] In this embodiment of the application, after collecting the performance data of a small-scale cluster, the performance data of a large-scale cluster can be predicted based on the performance data of the small-scale cluster, thereby predicting the performance of the large-scale cluster.

[0134] For example, performance data from a small-scale cluster can be used to predict the performance data of a large-scale cluster. For instance, the resource usage information of the training equipment when a small-scale cluster performs the aforementioned small task can be used as the resource usage information of the training equipment when a large-scale cluster performs the same large task. Similarly, the execution time required by a small-scale cluster to perform the aforementioned small task can be used as the execution time required by a large-scale cluster to perform the same large task.

[0135] It should be understood that when a large-scale cluster performs the aforementioned large tasks, the resource usage information of the training devices (e.g., GPU memory usage) determines whether the training task can be started in the large-scale cluster under the current task configuration. If the GPU memory usage has exceeded the upper limit of the training devices in the large-scale cluster, normal training cannot be completed, and users need to consider using some optimization techniques, such as GPU memory optimization techniques.

[0136] It should also be understood that the training task consists of multiple repeated steps, each involving a series of sequential computational operations, each of which takes a certain amount of time. Users can choose whether to enable a series of computational optimization capabilities, such as acceleration and load balancing operations, to shorten the execution time based on the execution time required for each step when performing large tasks on a large-scale cluster.

[0137] The performance data collected above represents the actual performance profile for small-scale clusters, while for large-scale clusters it represents a predicted performance profile.

[0138] In the above technical solution, in scenarios involving distributed training of models, based on the isomorphic resources and average partitioning of various training devices, the computational, communication, and memory loads on each training device when executing large tasks on a small-scale cluster are the same as those when executing small tasks on a small-scale cluster. Therefore, without changing the actual training code, training optimization implementation, training framework implementation, or underlying hardware implementation, a small-scale cluster can be built, and small tasks can be actually executed on this small-scale cluster. This allows for the acquisition of near-real-world large-scale cluster performance prediction and analysis data at a lower cost, thereby predicting the performance of large-scale clusters when executing large tasks. On one hand, running actual workloads on real devices improves the accuracy of large-scale cluster performance prediction; on the other hand, it eliminates the need to design new modeling schemes, thus increasing the credibility of large-scale cluster performance prediction results for users and reducing implementation complexity.

[0139] For example, if the distributed parallel strategy used in a small task executed in a small cluster is synchronous parallelism, this synchronous parallelism includes any of the following: data parallelism (DP), tensor model parallelism (TP), and sequence parallelism (SP). To predict and obtain more accurate performance data for large-scale clusters, embodiments of this application also need to modify the communication volume (i.e., the amount of data transmitted) of each training device in the small cluster during the communication process, so that the communication operation time when the small cluster executes a training task is theoretically the same as the operation time when executing a training task in a large-scale cluster.

[0140] For example, the communication volume (i.e., the amount of data transmitted) of each training device in a small cluster can be determined using the following formula.

[0141] For ease of description, the following explanation will take the determination of the communication volume of the first training device during the communication process as an example. The determination of the communication volume of other training devices during the communication process is the same as that of the first training device, and will not be repeated here.

[0142] For example, the communication operation time of the first training device when performing a large task in a large-scale cluster is shown in formula (1).

[0143] In the above formula (1), T a This represents the communication operation time required for the first training device to perform large tasks in a large-scale cluster. N represents the amount of data transmitted by the first training device when performing large tasks in a large-scale cluster; a BW represents the number of training devices in a large-scale cluster. a This represents the bandwidth of the first training device when performing large tasks in a large-scale cluster. This represents the amount of communication when the first training device performs a large task in a large-scale cluster.

[0144] For example, the communication operation time of the first training device when performing a small task in a small cluster is shown in Equation (1).

[0145] In the above formula (2), T e This represents the communication operation time required for the first training device to perform small tasks in a small cluster; The amount of data transmitted by the first training device when performing small tasks in a small cluster; N e The number of training devices in a small-scale cluster; BW e The bandwidth of the first training device when performing small tasks in a small cluster. This represents the communication volume of the first training device when performing small tasks in a small cluster.

[0146] In one possible implementation, taking the RingAllReduce scenario as an example, the amount of data transmitted by the first training device when executing a small task in a small-scale cluster can be determined by the following formula (3).

[0147] In another possible implementation, taking a fullmesh scenario as an example, the amount of data transmitted by the first training device when performing a small task in a small-scale cluster can be determined by the following formula (4).

[0148] Another example is a small-scale cluster using a distributed parallel strategy of pipelined parallelism (PP). Since pipelined parallelism is a special type of parallel scenario where multiple training devices execute training tasks asynchronously, each device has a different computational load. Therefore, if pipelined parallelism is used, the performance data for executing small tasks on a small-scale cluster will differ from the performance data for executing large tasks on a large-scale cluster. Thus, to predict more accurate performance data for large-scale clusters, this embodiment of the application needs to add communication or computation processes to the small-scale cluster, so that the execution time required for executing small tasks on a small-scale cluster is theoretically the same as the execution time required for executing large tasks on a large-scale cluster.

[0149] In one possible implementation, as illustrated in Figures 3 and 4, taking the addition of computational processes as an example, the number of computational processes added to the small-scale cluster is determined based on the difference between the number of equipment groups participating in the pipeline process in the large-scale cluster and the number of equipment groups participating in the pipeline process in the small-scale cluster. This ensures that the execution time required for the small-scale cluster to execute small tasks is theoretically the same as the execution time required for the large-scale cluster to execute large tasks.

[0150] The aforementioned equipment group may include one device or multiple devices, and this application embodiment does not specifically limit this.

[0151] It should be understood that the examples in Figures 3 and 4 are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of the application to the specific values ​​or specific scenarios illustrated in Figures 3 and 4. Those skilled in the art can obviously make various equivalent modifications or variations based on the examples given in Figures 3 and 4 below, and such modifications and variations also fall within the scope of the embodiments of this application.

[0152] For example, Figure 3 shows an example of a model being executed in a pipelined parallel manner by four training devices (training devices 1-4) in a large-scale cluster, with eight steps (including eight forward computation steps and eight backward computation steps).

[0153] As shown in Figure 3, the first forward computation is performed by training devices 1-4 in a pipeline manner, with a pipeline of 4 steps. For example, training device 1 performs its first forward computation and sends the result to training device 2 after completing its first forward computation. Then, training device 1 immediately performs its second forward computation, and so on. After receiving the result of its first forward computation from training device 1, training device 2 performs its first forward computation and sends the result to training device 3 after completing its first forward computation. Then, training device 2 immediately performs its second forward computation, and so on. After receiving the result of its first forward computation from training device 2, training device 3 performs its first forward computation and sends the result to training device 4 after completing its first forward computation. At this point, training device 3 immediately performs its second forward computation, and so on. After receiving the result of its first forward computation from training device 3, training device 4 performs its first forward computation, and after completing its first forward computation, it begins its first backward computation, and after completing its first backward computation, it sends the result of its first backward computation to training device 3. Then, training device 4 immediately performs its second forward computation, and so on. After receiving the result of its first backward computation from training device 4, training device 3 performs its first backward computation, and after completing its first backward computation, it sends the result of its first backward computation to training device 2. After receiving the result of its first backward computation from training device 3, training device 2 performs its first backward computation, and after completing its first backward computation, it sends the result of its first backward computation to training device 1. The subsequent processes follow the same pattern, and will not be elaborated further here.

[0154] As shown in Figure 3, in large-scale computing, the number of pipeline steps in each process (including forward computing and backward computing) is 4. That is, in large-scale computing, the number of devices participating in the pipeline steps is 4, namely training devices 1-4.

[0155] For example, assume that training devices 3 and 4 constitute a small-scale cluster. Figure 4 illustrates this by showing two training devices (training device 3 and training device 4) in a small-scale cluster executing eight steps of the model (including eight forward computation steps and eight backward computation steps) in a pipelined parallel manner. As can be seen from the process shown in Figure 4, each step (including forward and backward computation steps) has two pipelined steps. That is, in large-scale computation, the number of devices participating in the pipelined steps is two, namely training devices 3 and 4.

[0156] Therefore, based on the difference between the number of devices participating in the pipeline process in a large-scale cluster and the number of devices participating in the pipeline process in a small-scale cluster, two more steps of computation (including two forward computation steps and two backward computation steps) can be executed in the small-scale cluster shown in Figure 4. This makes the execution time required for the training task in the small-scale cluster theoretically the same as the execution time required for the training task in the large-scale cluster. For example, in Figure 4, "A" represents the newly added step, and "9" represents another newly added step.

[0157] In another possible implementation, as illustrated in Figures 5 and 6, we will take the example of increasing the communication process so that the execution time required for a small cluster to execute a small task is theoretically the same as the execution time required for a large cluster to execute a large task.

[0158] It should be understood that the examples in Figures 5 and 6 are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of the application to the specific values ​​or specific scenarios illustrated in Figures 5 and 6. Those skilled in the art can obviously make various equivalent modifications or variations based on the examples given in Figures 5 and 6 below, and such modifications and variations also fall within the scope of the embodiments of this application.

[0159] For example, Figure 5 illustrates another process where four training devices (training devices 0-3) in a large-scale cluster perform model training in a hybrid manner, employing both pipelined parallelism and synchronous parallelism (including DP, TP, SP, etc.). As shown in Figure 5, a pipelined parallelism strategy is executed between training devices 0 and 2, and between training devices 1 and 3. For instance, training devices 0 and 2 compute B3 in a pipelined parallel manner, while training devices 1 and 3 compute B3 separately in a pipelined manner. A synchronous parallelism strategy is executed between training devices 0 and 1, and between training devices 2 and 3. For instance, training devices 0 and 1 exchange their computed results during communication, and training devices 2 and 3 exchange their computed results during communication.

[0160] For example, suppose training device 0 and training device 2 constitute a small-scale cluster. As shown in Figure 6, the two training devices (training device 0 and training device 2) in the small-scale cluster execute only in a pipelined parallel manner. For example, training device 0 and training device 2 each compute B3 in a pipelined parallel manner. There is no interaction between training device 0 and training device 2 during communication. Therefore, in order to make the execution time required for the small-scale cluster to perform the training task theoretically the same as the execution time required for the training task to perform the training task in a large-scale cluster, a synchronous parallel strategy can be implemented between training device 0 and training device 2 in the small-scale cluster. For example, an additional communication process can be added between training device 0 and training device 2, during which training device 0 and training device 2 interact with each other's computed B3.

[0161] In other words, in the large-scale cluster shown in Figure 5, training device 0 and training device 1 need to communicate. However, in the small-scale cluster shown in Figure 6, since training device 1 is not included, training device 0 in the small-scale cluster actually has one less communication process, and its overall execution time is less than that of the large-scale cluster. Therefore, since the interconnection method between training device 0 and training device 1 is the same as that between training device 0 and training device 2, and the bandwidth between training device 0 and training device 1 is the same as that between training device 0 and training device 2, in the small-scale cluster shown in Figure 6, a communication can be added between training device 0 and training device 2, which would not normally need to communicate, to compensate for the time difference, so that the overall execution time of the small-scale cluster is theoretically the same as that of the large-scale cluster.

[0162] This application's embodiments can run actual workloads on real devices without modeling the complexities of hardware, drivers, scheduling, etc., thus improving the accuracy and reliability of performance prediction. If new optimizations, parallel strategies, or new hardware implementations are available, no new modeling schemes need to be designed; performance results can be obtained directly. Throughout the overall performance prediction process, no modifications to the underlying training framework or logic are required, optimizing the flexibility of performance prediction.

[0163] It should be noted that the above description is based on the model training process. The embodiments of this application can also be applied to the model inference process. Please refer to the description above for details, which will not be repeated here.

[0164] The methods provided by the embodiments of this application have been described in detail above with reference to Figures 1 to 6. The embodiments of the apparatus of this application will be described in detail below with reference to Figures 7 to 10. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the preceding method embodiments.

[0165] Figure 7 is a schematic block diagram of a cluster performance prediction device 700 provided in an embodiment of this application. The device 700 can be implemented by software, hardware, or a combination of both. The device 700 provided in this embodiment can implement the method flow shown in Figure 2 of this embodiment. The device 700 includes: an acquisition module 710, a construction module 720, a collection module 730, and a prediction module 740. The acquisition module 710 is used to acquire user input information, including details of the large-scale cluster to be predicted and the large task to be executed on the large-scale cluster. The construction module 720 is used to construct a small-scale cluster based on the details of the large-scale cluster, wherein the number of devices in the small-scale cluster is less than the number of devices in the large-scale cluster, and the interconnection method between devices in the small-scale cluster is the same as that between devices in the large-scale cluster. The construction module 720 is also used to construct a small task to run on the small-scale cluster based on the details of the large-scale cluster, the large task, and the constructed small-scale cluster, wherein the load of each device in the small-scale cluster when running the small task is the same as the load of each device in the large-scale cluster when running the large task. The acquisition module 730 is used to acquire performance data of the small-scale cluster during the execution of the small task. The prediction module 740 is used to use the performance data of the small-scale cluster as the performance data of the large-scale cluster when executing the large task.

[0166] Optionally, the user input may also include model information, which includes at least one of the following: the model's definition information and operating parameters, wherein the operating parameters include the model's distributed strategy.

[0167] Optionally, the distributed strategy includes synchronous parallelism, and the large-scale cluster and the small-scale cluster include a first device. The device 700 further includes: a determining module, used to determine the amount of data transmitted by the first device in the small-scale cluster based on the bandwidth of the first device in the large-scale cluster, the amount of data transmitted in the large-scale cluster, and the bandwidth of the small-scale cluster.

[0168] Optionally, the distributed strategy includes pipelined parallelism, and the determining module is used to determine the number of additional computational processes in the small-scale cluster based on the difference between the number of equipment groups participating in the pipeline process in the large-scale cluster and the number of equipment groups participating in the pipeline process in the small-scale cluster.

[0169] Optionally, the distributed strategy includes pipeline parallelism and synchronous parallelism, and the determination module is used to increase the communication process between equipment groups participating in the pipeline process in this small-scale cluster.

[0170] Optionally, the detailed information of the large-scale cluster includes at least one of the following: the number of devices in the large-scale cluster, the specifications of the devices, and the interconnection method between the devices.

[0171] Optionally, the user input may also include details about the small cluster.

[0172] Optionally, the details of the small-scale cluster may include at least one of the following: the number of devices in the small-scale cluster, the specifications of the devices, and the interconnection method between the devices.

[0173] Optionally, the performance data for this small cluster may include at least one of the following: device resource usage and execution time.

[0174] The device 700 here can be embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0175] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. For instance, the implementation of module 710 will be described below using module 710 as an example. Similarly, the implementation of other modules, such as module 720, acquisition module 730, prediction module 740, and determination module, can refer to the implementation of module 710.

[0176] As an example of a software functional unit, the acquisition module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0177] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0178] As an example of a hardware functional unit, the acquisition module 710 may include at least one computing device, such as a server. Alternatively, the acquisition module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0179] The multiple computing devices included in the acquisition module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0180] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] It should be noted that the above embodiments of the device, when executing the above methods, are only illustrative examples of the division of functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 710 can be used to execute any step in the above methods, the construction module 720 can be used to execute any step in the above methods, the acquisition module 730 can be used to execute any step in the above methods, the prediction module 740 can be used to execute any step in the above methods, and the determination module can be used to execute any step in the above methods. The steps implemented by the acquisition module 710, construction module 720, acquisition module 730, prediction module 740, and determination module can be specified as needed. By implementing different steps in the above methods through the acquisition module 710, construction module 720, acquisition module 730, prediction module 740, and determination module, all the functions of the above device can be realized.

[0182] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0183] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0184] The following describes in detail, with reference to Figure 8, a computing device provided in an embodiment of this application.

[0185] Figure 8 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application. The computing device 1500 can be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in Figure 8 includes at least one processor 1510 and a memory 1520.

[0186] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0187] The processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the method provided in this application. Alternatively, the processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0188] Optionally, the computing device 1500 also includes a communication interface 1530. The communication interface 1530 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 1500 and other devices or communication networks.

[0189] Optionally, the computing device 1500 also includes a system bus 1540, wherein the processor 1510, memory 1520, and communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540; for example, the processor 1510 can perform data read / write or code execution in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 8, but this does not mean that there is only one bus or one type of bus.

[0190] In one possible implementation, the processor 1510 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in memory 1520 or cache 1516.

[0191] Optionally, processor 1510 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1510 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Among these, a general-purpose processor is a microprocessor, etc. For example, processor 1510 may be a central processing unit (CPU).

[0192] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514.

[0193] Optionally, the processing unit 1512, also known as the core, is the most important component of the processor. The processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, utilizing parallel computing capabilities to accelerate program execution. Various processing units have fixed logical structures; for example, a processing unit includes logical units such as a Level 1 cache, a Level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0194] In one implementation example, the memory control unit 1514 controls the data interaction between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0195] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbitrator (not shown in Figure 8) is configured on the system bus to handle and coordinate contention for access by the multiple processing units 1512.

[0196] In one implementation example, the processing unit 1512 and the memory control unit 1514 are connected via internal chip connection lines, such as address lines, thereby enabling communication between the processing unit 1512 and the memory control unit 1514.

[0197] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When the processing unit 1512 needs to read data, it first looks for the required data in the cache. If the data is found, it is executed directly; otherwise, it looks for the data in memory. Since the cache operates much faster than memory, its purpose is to help the processing unit 1512 run faster.

[0198] The memory 1520 provides runtime space for processes in the computing device 1500. For example, the memory 1520 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1520. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1520 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0199] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 1510, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1510 will load the data that needs to be processed into RAM for processing, and after the processing is completed, the processing unit 1512 will send the result out.

[0200] By way of example and not limitation, memory 1520 is volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0201] The structure of the computing device 1500 listed above is merely illustrative and is not limited thereto. The computing device 1500 in this application includes various hardware components in existing computer systems. For example, the computing device 1500 also includes other memories besides memory 1520, such as disk storage. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing device 1500 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in FIG8.

[0202] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0203] As shown in Figure 9, the computing device cluster includes at least one computing device 1500. The memory 1520 of one or more computing devices 1500 in the computing device cluster may store the same instructions for performing the above-described methods.

[0204] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also each store a portion of the instructions for executing the above-described methods. In other words, a combination of one or more computing devices 1500 can jointly execute the instructions of the above-described methods.

[0205] It should be noted that the memory 1520 in different computing devices 1500 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned device. That is, the instructions stored in the memory 1520 of different computing devices 1500 can implement the functions of one or more modules within the aforementioned device.

[0206] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices, 1500A and 1500B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.

[0207] It should be understood that the functions of computing device 1500A shown in Figure 10 can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0208] In this embodiment, a computer program product containing instructions is also provided. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When run on a computing device, it causes the computing device to perform the methods provided above, or causes the computing device to perform the functions of the apparatus provided above.

[0209] In this embodiment, a computer-readable storage medium is also provided. This computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method described above.

[0210] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0211] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0212] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0216] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting cluster performance, characterized in that, The method includes: Obtain information input by the user, including detailed information about the large-scale cluster to be predicted and the large task to be executed on the large-scale cluster; A small-scale cluster is constructed based on the detailed information of the large-scale cluster, wherein the number of devices in the small-scale cluster is less than the number of devices in the large-scale cluster, and the interconnection method between the devices in the small-scale cluster is the same as the interconnection method between the devices in the large-scale cluster. Based on the details of the large-scale cluster, the large task, and the constructed small-scale cluster, a small task is constructed to run on the small-scale cluster, wherein the load of each device in the small-scale cluster is the same as the load of each device in the large-scale cluster when the large task is run in the large-scale cluster. During the execution of the small task by the small cluster, performance data of the small cluster is collected; The performance data of the small-scale cluster is used as the performance data of the large-scale cluster when executing the large task.

2. The method according to claim 1, characterized in that, The large task to be executed on the large-scale cluster includes the following information: the structure information of the model to be executed on the large-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: distributed strategy information and optimization strategy information.

3. The method according to claim 1 or 2, characterized in that, The small tasks executed on the small-scale cluster include the following information: the structural information of the model executed on the small-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: distributed strategy information and optimization strategy information.

4. The method according to claim 3, characterized in that, The distributed strategy includes synchronous parallelism, and the small-scale cluster includes a first device. The method further includes: The amount of data transmitted by the first device in the small-scale cluster is determined based on the bandwidth of the first device in the large-scale cluster, the amount of data transmitted in the large-scale cluster, and the bandwidth of the first device in the small-scale cluster.

5. The method according to claim 3, characterized in that, The distributed strategy includes pipelined parallelism, and the method further includes: The number of computational processes added in the small-scale cluster is determined based on the difference between the number of equipment groups participating in the pipeline process in the large-scale cluster and the number of equipment groups participating in the pipeline process in the small-scale cluster.

6. The method according to claim 3, characterized in that, The distributed strategy includes pipelined parallelism and synchronous parallelism, and the method further includes: A communication process is added between equipment groups participating in the assembly line process within the small-scale cluster.

7. The method according to any one of claims 1 to 6, characterized in that, The detailed information of the large-scale cluster includes at least one of the following: the number of devices in the large-scale cluster, the specifications of the devices, and the interconnection method between the devices.

8. The method according to any one of claims 1 or 7, characterized in that, The information input by the user also includes detailed information about the small-scale cluster, which includes at least one of the following: the number of devices in the small-scale cluster, the specifications of the devices, and the interconnection method between the devices.

9. The method according to any one of claims 1 to 8, characterized in that, The performance data of the small-scale cluster includes at least one of the following: resource usage and execution time of each device in the small-scale cluster.

10. A device for predicting cluster performance, characterized in that, The device includes: The acquisition module is used to acquire information input by the user, including detailed information about the large-scale cluster to be predicted and the large task to be executed on the large-scale cluster. The building module is used to build a small-scale cluster based on the details of the large-scale cluster, wherein the number of devices in the small-scale cluster is less than the number of devices in the large-scale cluster, and the interconnection method between the devices in the small-scale cluster is the same as the interconnection method between the devices in the large-scale cluster. The construction module is further configured to construct a small task running on the small-scale cluster based on the detailed information of the large-scale cluster, the large task, and the constructed small-scale cluster, wherein the load of each device in the small-scale cluster is the same as the load of each device in the large-scale cluster when the small-scale cluster runs the small task. The data acquisition module is used to collect performance data of the small-scale cluster during the distributed execution of the small task in the small-scale cluster. The prediction module is used to use the performance data of the small-scale cluster as the performance data of the large-scale cluster when executing the large task.

11. The apparatus according to claim 10, characterized in that, The large task to be executed on the large-scale cluster includes the following information: the structure information of the model to be executed on the large-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: distributed strategy information and optimization strategy information.

12. The apparatus according to claim 10 or 11, characterized in that, The small tasks executed on the small-scale cluster include the following information: the structural information of the model executed on the small-scale cluster, and the running parameters of the model, wherein the running parameters of the model include at least one of the following: distributed strategy information and optimization strategy information.

13. The apparatus according to claim 12, characterized in that, The distributed strategy includes synchronous parallelism, and the small-scale cluster includes a first device. The device further includes: The determining module is configured to determine the amount of data transmitted by the first device in the small-scale cluster based on the bandwidth of the first device in the large-scale cluster, the amount of data transmitted in the large-scale cluster, and the bandwidth of the first device in the small-scale cluster.

14. The apparatus according to claim 12, characterized in that, The distributed strategy includes pipelined parallelism. The determination module is used to determine the number of computational processes added in the small-scale cluster based on the difference between the number of equipment groups participating in the pipeline process in the large-scale cluster and the number of equipment groups participating in the pipeline process in the small-scale cluster.

15. The apparatus according to claim 12, characterized in that, The distributed strategy includes pipelined parallelism and synchronous parallelism. A determination module is used to add communication processes between equipment groups participating in the assembly line process in the small-scale cluster.

16. The apparatus according to any one of claims 10 to 15, characterized in that, The detailed information of the large-scale cluster includes at least one of the following: the number of devices in the large-scale cluster, the specifications of the devices, and the interconnection method between the devices.

17. The apparatus according to any one of claims 10 or 16, characterized in that, The information input by the user also includes detailed information about the small-scale cluster, which includes at least one of the following: the number of devices in the small-scale cluster, the specifications of the devices, and the interconnection method between the devices.

18. The apparatus according to any one of claims 10 to 17, characterized in that, The performance data of the small-scale cluster includes at least one of the following: resource usage and execution time of each device in the small-scale cluster.

19. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

20. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 8.

21. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 8.

Images