Configuration determination method and apparatus

Abstract

The application provides a configuration determination method and device, which can improve the accuracy of determining a first target configuration from multiple configurations. The method comprises: obtaining a first probability distribution under each configuration in the multiple configurations, the first probability distribution being used to indicate a probability distribution of a first time length; the first time length comprising a forward propagation time length of a first model under the corresponding configuration; and determining a first target configuration of the first model based on the first probability distribution under each configuration in the multiple configurations, the first target configuration being one of the multiple configurations.

Description

[0001] This application is a divisional application. The original application has the application number 202411649059.3 and the original application date is November 18, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to, but is not limited to, the field of artificial intelligence, and in particular to a configuration determination method and apparatus. Background Technology

[0003] As model sizes continue to grow, a single computing node can no longer meet the training needs of large models. Therefore, distributed training techniques for large models have emerged, which distribute the model's computational tasks to multiple computing nodes through parallel strategies. For any given model, training can be performed under various parallel configurations, but the computation time varies under different configurations. It is necessary to determine an optimal parallel configuration from among these options. Training the model under the optimal parallel configuration can reduce the computation time and improve training efficiency.

[0004] Currently, when searching for the optimal parallel configuration from various parallel configurations, it is necessary to first determine the end-to-end computation time of the model under different parallel configurations, and then determine the optimal parallel configuration based on the end-to-end computation time. However, since the computation time of the model will fluctuate under any identical parallel configuration, while the end-to-end computation time determined by related technologies under any configuration is a fixed value, it will affect the accuracy of determining the optimal parallel configuration. Summary of the Invention

[0005] This application provides a configuration determination method and apparatus, which reflects the volatility of end-to-end computation time through a first probability distribution, and determines a first target configuration from multiple configurations of a first model based on the probability distribution of the first time, thereby improving the accuracy of the first target configuration.

[0006] Firstly, a configuration determination method is provided, the method comprising: Obtain the first probability distribution for each of the multiple configurations. The first probability distribution indicates the probability distribution for a first duration. The first duration is the end-to-end computation time, which includes the forward propagation time during the first model's runtime. Based on the first probability distribution for each of the multiple configurations, determine the first target configuration for the first model. The first target configuration is one of the multiple configurations. The first target configuration is the configuration that satisfies the target conditions. For example, the target conditions could be the shortest first duration, the longest first duration, or the minimum fluctuation of the first duration, etc.

[0007] Based on the above technical solution, since the first probability distribution can reflect the various possibilities of the first duration under the corresponding configuration, determining the first target configuration from multiple configurations based on the first probability distribution can take into account the various probabilities of the first duration under each configuration, reflecting the actual fluctuation of the first duration. Thus, determining the first target configuration by using the first duration and its probability can improve the accuracy of the first target configuration compared to related technologies.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the first model includes multiple operators, each of the multiple operators corresponding to a second probability distribution under each configuration; the second probability distribution is used to indicate the probability distribution of a second duration, the second duration being the computation duration of an operator; the first probability distribution under each of the multiple configurations is determined based on the second probability distribution of the multiple operators of the first model under the corresponding configuration.

[0009] Based on the above technical solution, since the second probability distribution of each operator is different under each configuration, the second probability distribution of each operator under each configuration is synthesized to obtain the first probability distribution, which can fully consider the influence of the corresponding configuration on the first duration. The first probability distribution under the corresponding configuration is obtained by synthesizing the second probability distribution of each operator under each configuration, thereby improving the accuracy of the first probability distribution.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, each of the plurality of operators corresponds to an error probability distribution under each configuration, the error probability distribution being used to indicate the probability distribution of the error between the predicted value and the true value of the computation time of an operator; the first probability distribution corresponding to each of the plurality of operators under each configuration is determined based on the corresponding error probability distribution and the predicted value of the corresponding second duration.

[0011] Based on the above technical solution, since the fluctuation of the operator's error is less than the fluctuation of the operator's computation time, the accuracy of the error probability distribution is higher than that of the probability distribution for the first time duration. Correcting the predicted value for the second time duration based on the error probability distribution to obtain the second probability distribution can improve the accuracy of the second probability distribution.

[0012] Optionally, the second probability distribution corresponding to each of the multiple operators is obtained through reasoning using a probability distribution model. The probability distribution model can reason about each operator to obtain the corresponding second probability distribution, thus increasing the probability of obtaining the second probability distribution.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the error probability distribution corresponding to each of the multiple operators under each configuration is obtained through inference from an error probability distribution model.

[0014] Based on the above technical solution, the probability distribution model can infer the corresponding error probability distribution for each operator, thereby improving the efficiency of obtaining the error probability distribution.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the error probability distribution model is obtained by training based on training data; the training data includes the input size of the operator and the error between the operator's predicted value and the true value.

[0016] Based on the above technical solution, the error probability distribution model obtained by training the input size and error of the operator can infer the corresponding error probability distribution for the input size of all operators corresponding to the first model. Since the input size of the operator will be different under different configurations, and the input size of different operators may be the same under different configurations, the error probability distribution model trained in this way can reduce the amount of computation during the inference of the error probability distribution model while realizing the inference of the error probability distribution distribution of each operator under each configuration.

[0017] Optionally, the error probability distribution corresponding to each of the multiple operators in any configuration is determined based on the errors corresponding to the corresponding operators in various models under various configurations.

[0018] The various models can include any one of the multiple operators, and the various configurations include the parallel configurations used by each model during runtime. The error of any operator under various configurations is determined based on the error between the actual value and the predicted value of the operator at the second duration under the corresponding configuration. Since any operator may appear in various models, and its operation under different configurations of various models will lead to different actual values ​​of the operator's second duration, directly constructing an error probability distribution based on this can improve the accuracy of the error probability distribution.

[0019] Optionally, there are M configurations, where M is a positive integer greater than 1. Based on the first probability distribution under each of the multiple configurations, the first target configuration of the first model is determined, including: determining the first target configuration of the first model based on N×M values ​​of the first duration. The larger the value of N, the better the N values ​​of the first duration under each configuration fit the probability distribution of that configuration. Determining the first target configuration of the first model based on N×M values ​​of the first duration can consider the possibilities of the first duration values ​​under all configurations, thus improving the accuracy of the first target configuration.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, there are M configurations, where M is a positive integer greater than 1; based on the first probability distribution under each of the multiple configurations, the first target configuration of the first model is determined, including: based on N×M values ​​of the first duration, N first configurations are determined; the first probability distribution of each configuration corresponds to N values ​​of the first duration, the first configuration is one of the multiple configurations, and N is a positive integer greater than 1; based on the N first configurations, the first target configuration of the first model is determined.

[0021] Based on the above technical solution, N values ​​of the first duration are obtained from the first probability distribution of each configuration. For M configurations, there are N×M values ​​of the first duration. By determining N first configurations based on the N×M values ​​of the first duration, the configuration range can be narrowed and the efficiency of determining the first target configuration can be improved.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the first configuration is the configuration corresponding to the minimum value among the M values ​​of the first duration, and the M values ​​correspond one-to-one with the M configurations.

[0023] Based on the above technical solution, N first configurations can reflect the degree of fit between the first probability distribution of all configurations and the actual first probability distribution. When the degree of fit between the first probability distribution of each configuration and the corresponding actual first probability distribution in the N first configurations meets the fitting condition, the first target configuration can be determined from them, which can improve the accuracy of the first target configuration.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the first target configuration is the configuration with the largest proportion among N first configurations.

[0025] Based on the above technical solution, the configuration with the largest proportion is the one with the highest probability of the shortest first duration among the N first configurations. In this way, a first target configuration with a relatively short first duration can be quickly determined.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: obtaining the actual duration data of the first model under the first target configuration, updating the first probability distribution under each of the multiple configurations based on the actual duration data; determining N second configurations from the multiple configurations based on the updated first probability distribution, determining the second target configuration of the first model based on the N second configurations; and determining the third target configuration of the first model based on the first target configuration and the second target configuration.

[0027] Based on the above technical solution, since the real duration data of the first model under the first target configuration is used to update the second probability distribution of each operator under each configuration, the second probability distribution of each operator under each configuration after the update has a higher degree of fit with the real second probability distribution. Based on this, the first probability distribution under each configuration after the update also has a higher degree of fit with the real first probability distribution. The accuracy of the second target configuration determined from the first probability distribution under each configuration after the update can be higher than the accuracy of the first target configuration.

[0028] Optionally, the real duration data includes the true value of the second duration of each operator; the configuration determining device can determine the relative error of the corresponding operator based on the true value of the second duration of each operator and the predicted value of the second duration of the corresponding operator, add the relative error of each operator to the training data, retrain to obtain a new error probability distribution model, and use the new error probability distribution model to determine the updated first probability distribution, which can improve the fitting degree between the first probability distribution and the actual first probability distribution.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: a first target configuration is determined based on N1 first configurations, wherein the N1 first configurations are determined based on N1 values ​​of a second probability distribution and a second duration before the update of multiple configurations; a second target configuration is determined based on N2 second configurations, wherein the N2 second configurations are determined based on N2 values ​​of a second probability distribution and a second duration after the update of multiple configurations; wherein the first configuration and the second configuration are both one of multiple configurations; and N1 and N2 are both positive integers greater than 1.

[0030] Based on the above technical solution, the value of N can be set as needed in each round of search, which can improve the flexibility of the number of values. Thus, during the update process of the first probability distribution, as the fit between the first probability distribution and the actual first probability distribution becomes higher and higher, the value of N can be gradually reduced, thereby reducing the amount of computation in the search process.

[0031] In conjunction with the first aspect, in some implementations of the first aspect, updating the first probability distribution under each of the multiple configurations based on real duration data is performed when N first configurations do not meet the stopping search condition. The stopping search condition includes at least one of the following: the proportion of the maximum proportion configuration among the multiple first configurations is greater than a first proportion threshold, and the maximum proportion configuration is the first configuration with the largest proportion among the multiple first configurations; the multiple first configurations are determined based on N values ​​of the first duration corresponding to the first probability distribution under each configuration; and the total number of all configuration types among the multiple first configurations is less than the number of types threshold.

[0032] Based on the above technical solution, the first probability distribution can be automatically updated by judging the stopping search conditions, so that the first probability distribution fits the true first probability distribution better and better in the process of multiple rounds of updates.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the first duration of the first model under the second target configuration is less than the first duration of the first model under the first target configuration.

[0034] Based on the above technical solution, the target configuration obtained by the configuration determination device in each round of search is shorter than the first duration of the target configuration in the previous round. In this way, in the process of multiple rounds of search, the speed of finding the configuration corresponding to the shortest first duration can be accelerated, thereby improving search efficiency.

[0035] Optionally, based on N first configurations, determining a first target configuration for the first model includes: randomly selecting one of the N first configurations as a recommended configuration; if the recommended configuration satisfies the performance improvement conditions for the first target configuration, then using the recommended configuration as the second target configuration; the performance improvement conditions for the first target configuration include: if there is a communication bottleneck in the first target configuration, the communication duration of the recommended configuration is less than the communication duration of the first target configuration; or, if there is a computational bottleneck in the first target configuration, the computational duration of the recommended configuration is less than the computational duration of the first target configuration.

[0036] Based on the above technical solution, randomly selecting the recommended configuration maximizes the probability of selecting the first configuration with the highest probability among N first configurations. Simultaneously, since other first configurations with varying probabilities may also be selected, this reduces the probability that the second target configuration is only a local optimum, increasing the probability of finding the global optimum and thus improving the accuracy of the global optimum. Furthermore, because the second target configuration can alleviate the communication or computational bottlenecks of the first target configuration, there is a high probability that the first duration of the second target configuration is shorter than that of the first target configuration. By selecting the second target configuration with a shorter first duration based on the performance improvement conditions of the first target configuration, profiling of the second target configuration can be reduced, thereby decreasing the computational load during the search process.

[0037] In conjunction with the first aspect, in some implementations of the first aspect, the first configurations that are the same as the recommended configuration are removed from multiple first configurations, and the remaining first configurations form a set of recommended configurations; if the recommended configuration does not meet the performance improvement conditions for the first target configuration, a new recommended configuration is randomly selected from the set of recommended configurations.

[0038] Based on the above technical solution, the randomly selected recommended configuration may not perform as well as the first target configuration, requiring a reselection. Avoiding the already selected first configuration during the reselection increases the probability of selecting a first configuration with superior performance, and also increases the probability of selecting the second target configuration.

[0039] Optionally, the method further includes: stopping the search if the remaining recommendable proportion is less than the recommendable threshold; the remaining recommendable proportion is the ratio of the number of the first configuration in the recommendable configuration set to N.

[0040] Based on the above technical solution, the fewer the number of first configurations in the recommended configuration set, the less likely it is to find other configurations with better performance than the first target configuration. By setting a stop search condition based on the remaining recommended proportion, the method of determining the stop search condition can be simplified and the search efficiency can be improved, based on the randomly selected recommended configuration scheme.

[0041] In a second aspect, a configuration determination apparatus is provided. The apparatus includes: an acquisition module, configured to acquire a first probability distribution under each of a plurality of configurations, the first probability distribution indicating a probability distribution for a first duration; the first duration including the forward propagation duration of a first model under the corresponding configuration; and a determination module, configured to determine a first target configuration of the first model based on the first probability distribution under each of the plurality of configurations, the first target configuration being one of the plurality of configurations.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the first model includes multiple operators, each of the multiple operators corresponding to a second probability distribution under each configuration; the second probability distribution is used to indicate the probability distribution of a second duration, the second duration being the computation duration of an operator; the first probability distribution under each of the multiple configurations is determined based on the second probability distribution of the multiple operators of the first model under the corresponding configuration.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, each of the plurality of operators corresponds to an error probability distribution under each configuration, the error probability distribution being used to indicate the probability distribution of the error between the predicted value and the true value of the computation time of an operator; the second probability distribution corresponding to each of the plurality of operators under each configuration is determined based on the corresponding error probability distribution and the predicted value of the corresponding second duration.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the error probability distribution corresponding to each of the plurality of operators under each configuration is obtained through inference from a probability distribution model.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the probability distribution model is obtained by training based on training data; the training data includes the input size of the operator and the error between the operator's predicted value and the true value.

[0046] In conjunction with the second aspect, in some implementations of the second aspect, the multiple configurations are M configurations, where M is a positive integer greater than 1; the determining module 1020 is further used to determine based on N M values ​​of the first duration are used to determine N first configurations; the first probability distribution of each configuration corresponds to N values ​​of the first duration, the first configuration is one of the multiple configurations, and N is a positive integer greater than 1; based on the N first configurations, the first target configuration of the first model is determined.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, the first configuration is the configuration corresponding to the minimum value among the M values ​​of the first duration, and the M values ​​correspond one-to-one with the M configurations.

[0048] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition module is further configured to acquire the actual duration data of the first model under the first target configuration; the determination module is further configured to update the first probability distribution under each of the multiple configurations based on the actual duration data; determine the second target configuration of the first model from the multiple configurations based on the updated first probability distribution; and determine the third target configuration of the first model based on the first target configuration and the second target configuration.

[0049] In conjunction with the second aspect, in some implementations of the second aspect, the first target configuration is determined based on N1 first configurations, wherein the N1 first configurations are determined based on the first probability distribution of the multiple configurations before the update and N1 values ​​of the first duration; the second target configuration is determined based on N2 second configurations, wherein the N2 second configurations are determined based on the first probability distribution of the multiple configurations after the update and N2 values ​​of the first duration; wherein the first configuration and the second configuration are both one of the multiple configurations; and N1 and N2 are both positive integers greater than 1.

[0050] In conjunction with the second aspect, in some implementations of the second aspect, updating the first probability distribution under each of the multiple configurations based on the true value of the first duration is performed when the N first configurations do not meet the stopping search condition. The stopping search condition includes at least one of the following: the proportion of the configuration with the largest proportion among the N first configurations is greater than a first proportion threshold, wherein the configuration with the largest proportion among the N first configurations is the first configuration with the largest proportion. The N first configurations are N times the first duration corresponding to the first probability distribution under the M configurations. The total number of all configuration types in the N first configurations is less than the threshold number of types, determined by M values.

[0051] In conjunction with the second aspect, in some implementations of the second aspect, the first duration of the first model under the second target configuration is less than the first duration of the first model under the first target configuration.

[0052] Thirdly, a computing device is provided. The computing device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, cause the electronic device to perform instructions of the method in the first aspect or any possible implementation thereof.

[0053] Fourthly, embodiments of this application provide a computer-readable medium for storing a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation of the first aspect.

[0054] Fifthly, embodiments of this application provide a computer program including instructions for performing the method in the first aspect or any possible implementation thereof. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a configuration determination method provided in an embodiment of this application.

[0056] Figure 2 This is a flowchart illustrating a training method for an error probability distribution model provided in an embodiment of this application.

[0057] Figure 3a This is a schematic diagram illustrating an application scenario of a configuration determination device provided in an embodiment of this application.

[0058] Figure 3b This is a schematic diagram illustrating another application scenario of the configuration determination device provided in the embodiments of this application.

[0059] Figure 4aThis is a schematic diagram of the first probability distribution under various configurations provided in the embodiments of this application.

[0060] Figure 4b This is a schematic diagram of a sampling method provided in an embodiment of this application.

[0061] Figure 5 This is a schematic flowchart of a method for updating a second probability distribution provided in an embodiment of this application.

[0062] Figure 6 This is a flowchart illustrating another configuration determination method provided in an embodiment of this application.

[0063] Figure 7 This is a schematic flowchart of a method for determining the target configuration in the kth round, provided in an embodiment of this application.

[0064] Figure 8 This is a schematic diagram illustrating the computation and communication times for whole-network training provided in an embodiment of this application.

[0065] Figure 9 This is a flowchart illustrating another configuration determination method provided in an embodiment of this application.

[0066] Figure 10 This is a schematic diagram illustrating the determination of N first configurations provided in an embodiment of this application.

[0067] Figure 11 This is a schematic diagram illustrating an application scenario of another configuration determination device provided in an embodiment of this application.

[0068] Figure 12 This is a schematic block diagram of a configuration determination device provided in an embodiment of this application.

[0069] Figure 13 This is a schematic block diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] The terms "first," "second," and similar terms used in this article do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, "one" or similar terms do not indicate a quantity limitation, but rather indicate the existence of at least one.

[0072] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0073] Parallelism strategies for running large models across multiple computing nodes can include: Data Parallelism (DP), Pipeline Parallelism (PP), Tensor Parallelism (TP), Sequence Parallelism (SP), and Expert Parallelism (EP). DP places all layers of the large model on each computing node, configuring different sample sizes for training on different nodes; PP splits the model by layers, assigning different layers to different computing nodes; TP allocates different operators for each layer to different chips; SP segments the input sequence to reduce the cache requirements of individual computing nodes; EP decomposes the large model into multiple smaller expert models, each responsible for processing a specific task or a portion of the data. Hybrid parallelism refers to employing multiple parallel strategies simultaneously for model training.

[0074] For example, the model consists of 4 layers and can be trained on 2 computing nodes. Vertically splitting the model, with 2 layers processed on one computing node and the other 2 layers on the other, is called PP (Programming Partition). Dividing each layer in half, with one half processed on one computing node and the other half on the other, is called TP (Programming Partition). Alternatively, placing 2 layers on separate computing nodes, while processing the other 2 layers in half on one node and the other half on the other, is a hybrid parallel processing approach combining PP and TP.

[0075] For example, the model includes four layers, namely layers 1-4; and four computing nodes, namely computing nodes 1-4. Layer 1 is processed on computing node 1, and layer 2 is processed on computing node 2. That is, layers 1-2 are processed using the PP (Programmable Component) method on nodes 1-2. Layers 3 and 4 are deployed on computing nodes 3 and 4 respectively, with different sample sizes configured for processing on computing nodes 3 and 4. That is, layers 3-4 are processed using the DP (Programmable Component) method on computing nodes 3-4. Thus, the parallel configuration of the model is a hybrid parallelism of DP and PP.

[0076] As the cluster size, model size, and parallelism dimension of computing nodes increase, the search space for possible parallel configurations of the model, given a set of computing nodes, continuously expands. Electronic devices need to find the best-performing parallel configuration with the shortest runtime among these increasingly numerous configurations. In one related technique, the electronic device performs performance profiling on a single layer within any parallel configuration in the search space, obtaining the computation time of a single layer for each configuration. Thus, the electronic device can determine the end-to-end computation time for each parallel configuration based on the single-layer computation time and the number of layers in the model, and then determine the optimal parallel configuration from the search space based on the end-to-end computation time. Here, the single-layer computation time is the time required for all operators in a single layer of the model to complete the computation, and the end-to-end computation time is the time required for the model to perform one end-to-end computation process. The end-to-end computation time during application includes the forward propagation time, and the end-to-end computation time during training includes both forward and back propagation times. In another related technique, the electronic device performs theoretical calculations based on the input size of the operator for any parallel configuration in the search space to obtain the computation time and communication time of a single layer. These two times are combined to obtain the single-layer computation duration, which is then used to model the end-to-end computation duration. However, since the training time for each model may vary under any parallel configuration, and related techniques search for the optimal parallel configuration for a fixed end-to-end computation duration under each configuration without considering the volatility of the end-to-end computation duration, the accuracy of the optimal parallel configuration determined based on the end-to-end computation duration may be insufficient.

[0077] To address the aforementioned issues, this application provides a configuration determination method and apparatus. The configuration determination apparatus can determine a first target configuration from multiple configurations of a first model based on the probability distribution of end-to-end computation time, thereby improving the accuracy of the determined first target configuration. Here, the configuration determination apparatus can be applied to any computing device with computing capabilities. The computing device can be a computer, laptop, or other user terminal, or it can be a server, a cluster, or at least one computer within a cluster.

[0078] In this embodiment, for a given first model, the configuration determining device can exhaustively obtain all parallel configurations for computation on a given computing node, using the set of all parallel configurations as the search space, and then search for the first target configuration within the search space. The configuration determining device can obtain all possible parallel configurations through permutations and combinations, then perform memory modeling on each parallel configuration, and based on the memory modeling results, remove parallel configurations that would cause Out of Memory (OOM) errors from all possible parallel configurations, resulting in multiple configurations in the search space. This reduces the number of parallel configurations in the search space and improves search efficiency.

[0079] The search space includes multiple configurations. The configuration determining device needs to determine a first probability distribution corresponding to each configuration. The first probability distribution is the probability distribution of the first model for a first duration under a corresponding configuration. The first duration is the end-to-end computation duration, which includes the forward propagation duration of the first model. Optionally, the end-to-end computation duration may include the forward propagation duration and the back propagation duration of the first model.

[0080] In one possible implementation, the configuration determining device can first obtain the predicted value of the first duration corresponding to each configuration, thereby constructing a probability distribution of the first duration under each configuration, and obtaining a first probability distribution corresponding to each configuration, wherein the first probability distribution conforms to a Gaussian distribution. Specifically, the configuration determining device can first obtain the predicted values ​​of the second duration of each operator of the first model under multiple configurations for each configuration, and for each configuration, synthesize the predicted values ​​of the second duration of each operator to obtain the predicted value of the first duration under that configuration; the second duration is the computation duration of a single operator.

[0081] In one possible implementation, the configuration determination device can obtain the predicted value of the second duration of each operator under each configuration by calling a single operator.

[0082] In one possible implementation, the configuration determining device can use the input scale of each operator of the first model under each configuration and the computing power of the computing nodes of the first model under the corresponding configuration to determine the second duration of each operator of the first model under each configuration, thereby obtaining the predicted value of the second duration of each operator under each configuration. Here, the input scale of the operator can characterize the shape of the data input to the operator, that is, the size of the space occupied by the data in memory. A single operator has a corresponding input scale under different parallel configurations, and the input scale of a single operator may be different under different parallel configurations; different input scales of operators correspond to different computational loads.

[0083] For example, Table 1 shows the input size and corresponding computational cost of four operators. Taking the operator MatMul as an example, the input size is... Configuration 1 corresponds to , and They are 2, 3, and 4 respectively, with configuration 2 corresponding to... , and The input sizes are 3, 4, and 1 respectively. Under configuration 1, the input size is... The input size is configured as follows: .

[0084] Table 1

[0085] In one possible implementation, the configuration determining device can also obtain the true values ​​of the second duration of each operator of various models under historical parallel configurations. The various models may include a first model or may not include a first model. The historical parallel configuration is the parallel configuration adopted by various models during historical operation. Based on the input scale of each operator of various models under historical parallel configurations and the corresponding true values ​​of the second duration, a duration prediction model is constructed. The duration prediction model is used to infer all the corresponding input scales of the first model under each configuration to obtain the predicted value of the second duration of the first model under each of the multiple configurations.

[0086] Figure 1 A flowchart illustrating a configuration determination method is shown, such as... Figure 1 As shown, the method includes, but is not limited to: S101-S103.

[0087] S101. Obtain the second probability distribution corresponding to each operator under each of the multiple configurations.

[0088] Here, the configuration determining device can obtain a second probability distribution for each operator under each configuration. The second probability distribution is the probability distribution of the second duration. The second probability distribution represents the probability of different values ​​of the second duration of the corresponding operator. The second probability distribution can reflect the fluctuation of the computation duration of a single operator.

[0089] In this embodiment, the configuration determining device can obtain the true value of the second duration of each operator under each parallel configuration, and construct the second probability distribution of each operator under each parallel configuration based on this. The second probability distribution conforms to a Gaussian distribution. Here, the configuration determining device can obtain the true value of the second duration of each operator under historical parallel configurations for various models, and use this to construct training data, train a Gaussian distribution model, and obtain a probability distribution model; the second probability distribution of each operator under each configuration of the first model is inferred through the probability distribution model; thus, the efficiency of obtaining the second probability distribution can be improved.

[0090] In one possible implementation, the configuration determining device can acquire the tensor shapes corresponding to the input scales of each operator under historical parallel configurations of various models, i.e., historical tensor shapes; establish a correspondence between the historical tensor shapes and the true values ​​of the second duration; and train a Gaussian distribution model based on this correspondence to obtain a probability distribution model. The input tensor shapes are then inferred using the probability distribution model to obtain the corresponding second probability distribution; that is, by inputting the tensor shapes corresponding to the operator scales of each operator under each configuration of the first model into the probability distribution model, the configuration determining device can obtain the second probability distribution of each operator under each configuration of the first model. Since different operators may have the same input scale under different configurations, inference is performed on the tensor shapes instead of inferring for each operator individually, reducing the computational load of the probability distribution model.

[0091] Optionally, the configuration determining device can also establish a correspondence between the predicted values ​​of the historical tensor shape and the predicted values ​​of the second duration for each operator under various models in historical parallel configurations. Then, combining this correspondence with the actual values ​​of the historical tensor shape and the second duration, the error between the predicted value and the actual value of the second duration is determined. Here, the error can be the difference between the predicted value and the actual value of the second duration, or it can be the relative error between the two. After determining the error, the configuration determining device can establish a correspondence between the historical tensor shape and the error, thereby establishing training data, training a Gaussian distribution model, and obtaining an error probability distribution model. Thus, through the error probability distribution model, inference can be performed on any historical tensor shape of the first model to obtain the error probability distribution corresponding to that historical tensor shape. Then, the error probability distribution corresponding to each operator of the first model under each configuration is determined. Based on the error probability distribution of the operator under each configuration, the predicted value of the second duration of that operator under the corresponding configuration is corrected to obtain the second probability distribution of that operator under each configuration. Since the volatility of the error is less than that of the true value, determining the corresponding second probability distribution based on the error probability distribution can improve the accuracy of the second probability distribution.

[0092] For example, the relative error of the second duration It can be expressed by formula (1).

[0093] Formula (1).

[0094] in, The actual value for the second duration. This is the predicted value for the second duration.

[0095] In any configuration, the operator The probability distribution of the relative error is expressed as: The configuration determination device can determine the probability distribution of the second duration of operator i under the configuration based on formula (2). .

[0096] Formula (2).

[0097] It should be noted that the module used to train the error probability distribution model is the training module. The training module can be located inside or outside the configuration determining device. When the training module is outside the configuration determining device, it can communicate with the configuration determining module.

[0098] Figure 2 A flowchart illustrating the training method for the error probability distribution model is shown, as follows: Figure 2 As shown, the method includes, but is not limited to: S201-S204.

[0099] S201. Traverse the real performance data of various models during historical runtime.

[0100] In this context, after any model in the various models runs under its corresponding parallel configuration, the training module can obtain the model's actual performance data; the parallel configurations in which the model has run are called historical parallel configurations. Actual performance data can include: actual duration data and actual time data. Actual duration data includes the actual values ​​of the second duration of each operator for each model under historical parallel configurations and the actual values ​​of the first duration for each model under various configurations. Actual time data includes the computation time and communication time of each model within the first duration under historical parallel configurations.

[0101] It should be noted that the configuration determination device is applied to a computing device. The computing device used by various models during historical runtime may be the same computing device as the computing device where the configuration determination device is located, or it may be a different computing device.

[0102] Figure 3a An application scenario for a configuration determination device is shown, such as... Figure 3a As shown, the configuration determination device 1000 is applied on computing device 1, and computing device 1 and computing device 2 are connected. Various models are run in historical mode on computing device 2, which stores the real performance data of various models during historical mode. The configuration determination device 1000 can obtain the real performance data of various models during historical mode from computing device 2 through the connection.

[0103] It should be noted that computing device 1 and computing device 2 can be servers, clusters, computers, laptops, etc. Optionally, computing device 1 and computing device 2 can be different computers in the same cluster.

[0104] In this context, computing device 1 and computing device 2 can be computing devices of the same type, for example, both being servers. Alternatively, computing device 1 and computing device 2 can be computing devices of different types; for example, computing device 1 can be a server, and computing device 2 can be a cluster.

[0105] Figure 3b An application scenario for a configuration determination device is shown, such as... Figure 3b As shown, various models undergo historical execution on cluster 11. Computers 2-3 in cluster 11 store real performance data from the historical execution of various models. The configuration determination device 1000 is applied to computer 1 in the cluster. In one possible implementation, the computer in cluster 11 may include at least one card; when there are multiple cards, the cards can communicate with each other. The card can be a graphics processing unit (GPU), a central processing unit (CPU), or a natural processing unit (NPU), etc. For example, cluster 11 includes a CPU and multiple GPUs / NPUs. The configuration determination device is applied to the CPU. Through the configuration determination device of this embodiment, the deployment method of the first model on multiple GPUs / NPUs can be determined, that is, the parallel configuration method of the first model on multiple GPUs / NPUs.

[0106] S202. Based on real performance data, establish the correspondence between the historical tensor shape and the true value of the second duration.

[0107] The training module can obtain the true values ​​of the second duration of each operator under historical parallel configurations from the real performance data of various models during historical runtime. It iterates through the input scale of each operator for each model under the corresponding configuration, using it as the historical input scale, and establishes a first correspondence between the historical input scale and the true values ​​of the second duration. The tensor shape corresponding to the historical input scale is the historical tensor shape. The correspondence between the input scale and the historical tensor shape is the second correspondence; based on the second and first correspondences, the correspondence between the historical tensor shape and the true values ​​of the second duration is determined.

[0108] S203. Based on the relative errors of each operator under historical parallel configuration, and the correspondence between the true values ​​of historical tensor shape and second duration, construct training data.

[0109] The training module first obtains the predicted values ​​of each operator for the second duration under various historical parallel configurations. Then, combining this with the second correspondence, it obtains the correspondence between the historical tensor shape and the predicted values ​​for the second duration. Next, combining this correspondence with the actual values ​​of the historical tensor shape and the second duration, it determines the relative error between the predicted value and the actual value for the second duration corresponding to each historical tensor shape, thus obtaining the third correspondence between the historical tensor shape and the relative error. Training data is constructed based on this third correspondence, with the data format being (tensor shape, relative error). Since the historical tensor shape contains information about the computational cost of the operator under various historical parallel configurations, and different operators may have the same historical tensor shape under different historical parallel configurations, constructing training data through the third correspondence can reduce the computational cost of training the error probability distribution model and improve training efficiency.

[0110] In one possible implementation, the training module can determine the computational cost based on the input scale of each operator of various models under historical parallel configurations. Here, the training module can use the computational cost corresponding to the input scale of each operator of various models under historical parallel configurations and the computing power of the computing nodes used under historical parallel configurations to determine the second duration corresponding to the input scale of each operator of various models under historical parallel configurations; then, combined with the input scale of each operator of various models under historical parallel configurations, it can determine the predicted value of the second duration corresponding to each operator of various models under historical parallel configurations.

[0111] In one possible implementation, the training module can use the second duration of each operator under the historical parallel configuration, obtained through single operator calls, as the predicted value of the second duration of each operator under the historical parallel configuration for each model.

[0112] S204. Train a Gaussian distribution model based on the training data to obtain an error probability distribution model.

[0113] The error probability distribution model can infer a corresponding second probability distribution based on any tensor shape corresponding to the first model. This second probability distribution follows a Gaussian distribution. The tensor shape corresponding to the first model includes the tensor shapes corresponding to the input sizes of each operator in each configuration of the first model.

[0114] In this embodiment, the error probability distribution model is configured in the configuration determination module. The configuration determination module can input the first model into the error probability distribution model for each corresponding tensor shape to obtain the second probability distribution corresponding to each tensor shape. Then, based on the tensor shape, it can obtain the second probability distribution of each operator under various configurations. Here, one tensor shape can correspond to at least one operator under at least one configuration. By reasoning with the tensor shapes, the second probability distribution of each operator under all configurations can be obtained. Compared to reasoning with all operators under all configurations one by one, this reduces the computational load of reasoning with the error probability distribution model.

[0115] S102. Based on the second probability distribution corresponding to each operator under a single configuration, synthesize the first probability distribution.

[0116] The configuration determining device can synthesize the second probability distributions of each operator under each configuration into a first probability distribution under that configuration, and determine a first target configuration from multiple configurations based on the first probability distributions under multiple configurations.

[0117] For any given configuration, the second probability distributions of each operator are combined to form the first probability distribution under that parallel configuration. The method can be implemented by referring to formula (3).

[0118] Formula (3).

[0119] Different parallel configurations correspond to different G, where G represents the synthesis rule under the corresponding parallel configuration. Indicates the first operator among all operators The second probability distribution of each operator The probability distribution of the first duration is obtained by synthesizing the distributions of the second duration of all operators in each layer of the model. This refers to the duration of the communication.

[0120] S103. Based on the first probability distribution under each configuration, determine the first target configuration from multiple configurations.

[0121] Here, the first target configuration is the configuration that meets the target conditions among multiple configurations. For example, the target condition could be the shortest first duration, the smallest proportion of communication time in the first duration, the smallest proportion of computation time in the first duration, the longest first duration, or the least fluctuation in the first duration, etc.; these can be set according to actual needs, and this application embodiment does not impose any restrictions.

[0122] In one possible implementation, the configuration determining device can select the first probability duration with the highest probability under each configuration as the first duration under that configuration based on a first probability distribution under each configuration, thereby obtaining the first durations for multiple configurations. The configuration determining device can determine a first target configuration from a set of multiple configurations based on the first durations under multiple configurations.

[0123] In one possible implementation, the first target configuration can be the one with the shortest first duration among multiple configurations, and the configuration determining device can directly determine the first target configuration as the optimal parallel configuration. In this way, the configuration with the shortest first duration and the highest probability can be quickly determined, improving the efficiency of determining the optimal parallel configuration.

[0124] In one possible implementation, the configuration determining device can arrange the first durations under multiple configurations from smallest to largest, select the first number of configurations with the highest probability value for the first duration from the first number of configurations, and then select the configuration with the highest probability value for the first duration from the first number of configurations as the first target configuration. The configuration determining device can determine one configuration as the first target configuration from the first number of configurations. In one possible implementation, the configuration determining device can select the configuration with the shortest communication time from these configurations as the first target configuration. In another possible implementation, the configuration determining device can select the configuration with the shortest calculation time from these configurations as the first target configuration. Thus, by determining the first target configuration from configurations with a probability of having a shorter first duration, the configuration determining device can improve the accuracy of the first target configuration.

[0125] Figure 4a The diagram illustrates the curves of the first probability distribution under various configurations, including configurations 1 to 4. The curves of the first probability distribution differ under different configurations; configuration 2 exhibits the most dispersed probability distribution, while configuration 4 shows the steepest. It can be seen that configuration 4 has the highest probability of obtaining the maximum value for the first duration, but also the longest duration; while configuration 1 has the lowest probability of obtaining the maximum value for the first duration, but also the shortest duration.

[0126] Optionally, the configuration determining device can perform N sampling operations based on the first probability distribution under multiple configurations to obtain N sampled values ​​for each configuration, with each sampled value representing a first duration. N is a positive integer, and the larger the value of N, the better the fit between the N sampled values ​​and the first probability distribution. It should be noted that the sampling method for the first probability distribution under multiple configurations can be acceptance / rejection sampling, importance sampling, or MCMC (Markov Chain Monte Carlo) sampling, etc., which can be set according to actual needs, and this application embodiment does not impose any limitations.

[0127] The following example, using the method of receiving rejection sampling, illustrates how to obtain N sampled values ​​of the first probability distribution under various configurations. Figure 4b An illustrative example shows a curve representing the first probability distribution under one configuration, with the horizontal axis representing the first duration and the vertical axis representing the probability. For example... Figure 4b As shown, random sampling is performed within a rectangle enclosed by the horizontal and vertical coordinate distribution ranges of the first probability distribution curve. Circles represent the first sampling points below the first probability distribution curve, and crosses represent the second sampling points above the curve. The second sampling points conform to the first probability distribution. It can be seen that probability value 1 is less than probability value 2, and the number of circles near the dashed line corresponding to probability value 1 is significantly less than the number of circles near the dashed line corresponding to probability value 2. The configuration determining device can use the horizontal coordinate corresponding to the second sampling point as the Nth sampling value.

[0128] Using the above sampling method, the higher the probability value of the first duration, the more values ​​are sampled, and the higher its proportion appears in the N sampled values. The configuration determining device can determine the target configuration from multiple configurations based on the N sampled values ​​under each configuration. In one possible implementation, the configuration determining device can select at least one configuration whose occurrence frequency is greater than a first frequency threshold from the N sampled values ​​under each configuration, and determine the first target configuration from these at least one configuration. A higher occurrence frequency indicates a steeper first probability distribution under the corresponding configuration, and a higher degree of fit between the volatility of the first duration value and the actual volatility.

[0129] In one possible implementation, there are M possible configurations, where M is a positive integer greater than 1. The configuration determining device M samples the first probability distribution for each configuration N times. Each configuration corresponds to N values ​​for a first duration, i.e., N sampled values. Therefore, the M configurations can yield N... There are M values ​​for the first duration. The configuration determining device can select a first configuration corresponding to a value that satisfies the consistency condition from the M values ​​for the first duration sampled in a single sampling. N samplings correspond to N first configurations, and the first configuration is one of several configurations. The consistency condition includes any of the following: maximum value, minimum value, median value, etc. Thus, any configuration among the multiple configurations will appear in the N first configurations only if the probability of satisfying the consistency condition is greater than 0, and the higher the probability of satisfying the consistency condition, the higher the frequency of its appearance in the N first configurations.

[0130] Optionally, the configuration determining device can select the configuration corresponding to the minimum value from M values ​​of the first duration in a single sampling as the first configuration corresponding to the single sampling. N samplings correspond to N first configurations, and the first configuration is one of multiple configurations. Based on the N first configurations, the first target configuration with the shortest first duration can be determined.

[0131] Optionally, the configuration determining device can select the configuration corresponding to the maximum value from the M first duration values ​​of a single sample as the first configuration corresponding to that single sample. N samples correspond to N first configurations, and the first configuration is one of multiple configurations. Based on the N first configurations, the first target configuration with the longest first duration can be determined.

[0132] In one possible implementation, the configuration determining device can select the configuration with the shortest first duration from N first configurations as the first target configuration. In this way, the first target configuration with the shortest first duration can be selected from all possible first durations.

[0133] For example, the N first configurations include four: configuration 1, configuration 2, and configuration 3. The end-to-end computation time for configuration 1 is 3.5 μs, the end-to-end computation time for configuration 2 is 3.1 μs, and the end-to-end computation time for configuration 3 is 4.2 μs. The configuration determining device can use configuration 2 as the first target configuration.

[0134] In one possible implementation, the configuration determining device can select the most frequently occurring configuration from N first configurations as the first target configuration. In this way, the first target configuration with the highest probability of becoming the one with the shortest first duration can be selected.

[0135] For example, the N first configurations include four: configuration 1, configuration 2, and configuration 3. The configuration determining device can use configuration 1 as the first target configuration.

[0136] In one possible implementation, after the configuration determining device completes the first round of search, it can continue with a second round of search; after the second round of search, it can continue with a third round of search, until the number of search rounds reaches a threshold, at which point the search stops. From the target configurations obtained in each round of search, one is determined as the optimal parallel configuration.

[0137] For example, the first round of search yields the first round target configuration, the second round of search yields the second round target configuration, and so on, until the fifth round of search stops. The configuration determining device can select the one with the shortest first duration from the first round target configuration, the second round target configuration, ... and the fifth round target configuration as the optimal parallel configuration.

[0138] In one possible implementation, after the configuration determining device completes the first round of search, it can obtain the first round of search results, which include N1 first configurations and a first target configuration. Then, the configuration determining device can determine whether the N1 first configurations meet the stop search condition. If so, the search stops; otherwise, it can update the first probability distribution of the first model under each configuration based on the actual performance data of the first round of search results, and perform a second round of search based on the new first probability distribution under each configuration. From the multiple new first probability distributions, it determines N2 new first configurations, obtaining N2 second configurations; from the N2 second configurations, it determines the second target configuration of the first model; and from the first target configuration and the second target configuration, it determines the third target configuration of the first model. The stop search condition can include at least one of the following: the proportion of the maximum percentage configuration among the N1 first configurations is greater than a first proportion threshold; the maximum percentage configuration is one of multiple configurations among the N1 first configurations; and the total number of types of all configurations among the N1 first configurations is less than a type number threshold. The first proportion threshold and the type number threshold can be set as needed, and this embodiment does not impose any restrictions. N1 and N2 can be the same or different. Thus, the value of N can be flexibly set in each round of search. In one possible implementation, N2 can be less than N1. In this way, when the fit between multiple first probability distributions and the corresponding actual first probability distributions becomes better, the value of N can be reduced, thereby reducing the amount of computation during the update process.

[0139] It should be noted that if the first probability distributions under the M configurations all fit the corresponding actual first probability distributions sufficiently well, then all M first probability distributions will be sufficiently steep. Thus, the configuration with the shortest second duration among the M configurations obtained in each of the N samplings can be stably represented by one of the configurations, and this configuration will appear with a sufficiently high frequency among the N first configurations. In extreme cases, all N first configurations will be this single configuration. The flatter the first probability distribution of any configuration, the more evenly the probabilities of its different second duration values, and the greater the number of configuration types appearing among the N first configurations.

[0140] For example, the first proportion threshold is 70%, the number of categories threshold is 3, and N is 5. The N first configurations are configuration 1, configuration 1, configuration 1, configuration 1, and configuration 2. The number of configuration categories in the five first configurations is 2, which is less than the category threshold; configuration 1 accounts for 80% of the five first configurations, which is greater than the first proportion threshold; and configuration 2 accounts for 20% of the five first configurations. Therefore, the configuration determining device can determine that the N first configurations satisfy the stopping search condition. The first target configuration is the optimal parallel configuration.

[0141] In one possible implementation, the configuration determining device can profiling N first configurations to obtain real performance data for the N first configurations, and update the first probability distribution under each configuration based on the real performance data of the N first configurations. In this way, profiling needs to be performed on each of the N first configurations, and a large amount of real data is obtained, which helps to improve the fit between the updated first probability distribution and the actual probability distribution, but the computation is large and the computation time is long.

[0142] Optionally, the configuration determining device can perform profiling on the first target configuration to obtain the actual performance data of the first target configuration, and update the first probability distribution under each configuration based on the actual performance data of the first target configuration; in this way, only one first target configuration needs to be profiled, the amount of computation is small, and the computation time is short, which can improve the update efficiency of the first probability distribution.

[0143] In this embodiment, the real performance data may include the true value of the first duration of the first model under each configuration. The configuration determining device may reconstruct the corresponding first probability distribution based on the true value of the first duration in the first round of search results. The real performance data may include the true value of the second duration of each operator of the first model under each configuration. The configuration determining device may reconstruct the second probability distribution corresponding to each operator under multiple configurations based on the true value of the second duration in the first round of search results. Based on the new second probability distribution of each operator under each configuration, a new first probability distribution under the corresponding configuration is synthesized.

[0144] In one possible implementation, the second probability distribution is obtained by inferring the input scale of the operators through a probability distribution model. The configuration determining device can determine the correspondence between the input scale and the true value of the second duration corresponding to each operator of the first model under the first target configuration based on the true value of the second duration. This correspondence is then added to the training data, and the Gaussian distribution model is retrained using the new training data to obtain a new probability distribution model. The input scale of each operator under various configurations of the first model is then re-inferred using the new probability distribution model to obtain a new second probability distribution for each operator of the first model under various configurations.

[0145] based on Figure 2 , Figure 5 A flowchart illustrating a method for updating the second probability distribution is shown, as follows: Figure 5 As shown, the method includes, but is not limited to: S301-S305.

[0146] S301. Obtain the true value of the second duration corresponding to each operator of the first model under the first target configuration.

[0147] It should be noted that the configuration determining device can send the first target configuration to the performance analysis module, which then performs profiling on the first target configuration to obtain the actual performance data of the first model under the first target configuration. The configuration determining device can then obtain the second duration of each operator of the first model under the first target configuration from the actual performance data, thus obtaining the actual value of the second duration corresponding to each operator under the first target configuration. Here, the performance analysis module can be located inside or outside the configuration determining device. If the performance analysis module is located outside the configuration determining device, a communication connection is established between the performance analysis module and the configuration determining device for data transmission.

[0148] S302. Determine the relative error corresponding to each operator under the first target configuration based on the true value of the second duration corresponding to each operator of the first model under the first target configuration.

[0149] The first target configuration is one of several configurations. The configuration determination device can directly obtain the predicted value of the second duration corresponding to each operator under the first target configuration obtained by the training module in S203; and determine the relative error corresponding to each operator under the first target configuration based on the true value and predicted value of the second duration of each operator under the first target configuration.

[0150] S303. Based on the relative errors of each operator under the first target configuration, and the correspondence between the tensor shape and the true value of the second duration under the first target configuration, construct supplementary training data.

[0151] The configuration determination device can determine the tensor shape corresponding to each operator under the first target configuration, and combine it with the relative error corresponding to each operator under the first target configuration to determine the correspondence between the tensor shape and the relative error under the first target configuration, and use these correspondences as supplementary training data.

[0152] S304. Combine the supplementary training data with the training data as updated training data to train the Gaussian distribution model and obtain a new error probability distribution model.

[0153] The configuration determination device adds supplementary training data to the training data, resulting in updated training data. A Gaussian distribution model is then trained based on the updated training data to obtain a new error probability distribution model.

[0154] S305. By reasoning about all tensor shapes corresponding to the first model through the new error probability distribution model, a new second probability distribution corresponding to each operator of the first model under each configuration is obtained.

[0155] In this embodiment, the configuration determination module can input all tensor shapes corresponding to the first model into a new error probability distribution model to obtain new error probability distributions corresponding to these tensor shapes, and then obtain new error probability distributions for each operator under multiple configurations. Based on the new error probability distribution of a single operator under each configuration, the predicted value of the second duration of the operator under the corresponding configuration is revised to obtain a new second probability distribution for each operator under each configuration. The synthesis of the new second probability distributions of each operator under each configuration can obtain a new first probability distribution for the corresponding configuration. The configuration determination device can determine a second target configuration from multiple configurations based on the new first probability distribution under each configuration.

[0156] The method by which the configuration determining device determines the first target configuration based on the first probability distribution under each configuration and the method by which it determines the second target configuration based on the new second probability distribution under each configuration can be the same, and will not be described in detail here.

[0157] It should be noted that the higher the consistency between the performance of the computing nodes corresponding to the historical parallel configuration and the performance of the computing nodes running the first model, the better the fit between the error probability distribution inferred by the error probability distribution model and the actual error probability distribution. Furthermore, the better the fit between the second probability distribution corresponding to each operator under each configuration of the first model determined based on the error probability distribution and the actual second probability distribution, the fewer times the error probability distribution model needs to be updated. Optionally, the computing nodes corresponding to the historical parallel configuration and the computing nodes running the first model are the same type of computing chip.

[0158] In this embodiment, the configuration determining device can determine N second configurations based on a new first probability distribution for each configuration, and then determine a second target configuration from the N second configurations. If the N second configurations satisfy the stopping search condition, the configuration with the shorter first duration is selected as the third target configuration from the first and second target configurations; this third target configuration is the optimal parallel configuration found. If the N second configurations do not satisfy the stopping search condition, the configuration determining device can continue to update the first probability distribution for each configuration based on the second target configuration, perform a third round of search, and so on, until the N first configurations determined in the Xth round satisfy the stopping search condition, at which point the search stops, where X is a positive integer greater than or equal to 3.

[0159] Figure 6 A flowchart illustrating a configuration determination method is shown, such as... Figure 6 As shown, the search results obtained in the j-th round of search include: Nj first configurations and the j-th round target configuration, where j is a positive integer greater than 2. The method includes, but is not limited to: S401-S407.

[0160] S401. Determine whether the Nj first configurations meet the search stopping condition. If yes, execute S407; otherwise, execute S402.

[0161] In this process, the first round of search yields N1 first configurations, the second round yields N2 second configurations, and the third round and any subsequent j-th round yields Nj j-th configurations. The values ​​of N1, N2, and Nj can be the same or different; this can be set according to actual needs, and this embodiment does not impose any restrictions. In one possible implementation, the configuration determining device can decrease the value of N every preset number of rounds, for example, decreasing the value of N by 10% every round. Both the preset number of rounds and the amount of decrease in the value of N can be set as needed, and this embodiment does not impose any restrictions.

[0162] S402. Obtain the true values ​​of the second duration of each operator of the first model under the target configuration of the j-th round.

[0163] In this process, the first target configuration obtained in the first round of search is used as the first-round target configuration, the second target configuration obtained in the second round of search is used as the second-round target configuration, and the target configuration for the j-th round is obtained in the third round and any subsequent round. The method for obtaining the true values ​​of the second duration of each operator of the first model under the j-th-round target configuration is the same as the method for obtaining the true values ​​of the second duration of each operator of the first model under the first target configuration, and will not be repeated here.

[0164] S403. Based on the true values ​​of the second duration of each operator of the first model under the target configuration of the j-th round, determine the relative error of each operator of the first model under the target configuration of the j-th round.

[0165] S404. Based on the relative errors of each operator under the target configuration of the j-th round, and the correspondence between the true values ​​of tensor shape and second duration under the target configuration of the j-th round, construct the supplementary training data for the j-th round.

[0166] For explanations of S403-S404, please refer to the descriptions in S302-S303; they will not be repeated here.

[0167] S405. Add the supplementary training data from round j to the training data from round j to obtain the training data from round j+1.

[0168] The configuration determination device can fill in the correspondence between the true values ​​of the tensor shape and the second duration under the j-th target configuration into the j-th round of training data to obtain the (j+1)-th round of training data.

[0169] In S203, the new training data constructed for the first time is the first round of supplementary training data. The first round of supplementary training data is added to the first round of training data to obtain the second round of training data. The updated training data obtained in S304 is the second round of training data.

[0170] S406. Train a Gaussian distribution model based on the training data of the (j+1)th round to obtain the error probability distribution model of the (j+1)th round.

[0171] The larger the value of j+1, the more relative errors are obtained for the true values ​​of the second duration under the parallel configuration based on the first model in the training data of the (j+1)th round. The degree of fit between the error probability distribution obtained by the error probability distribution model in the (j+1)th round of training and the actual error probability distribution of each operator under the first model will be higher and higher. The degree of fit between the first probability distribution obtained in the (j+1)th round and the actual first probability distribution under each configuration will be higher and higher.

[0172] S407. Stop searching.

[0173] After the search stops in the j-th round, the configuration determination device can determine the (j+1)-th target configuration from the target configurations in the j-th rounds. The (j+1)-th target configuration can be the one with the shortest true value of the first duration among the target configurations in the j-th rounds. The target configurations in the j-th rounds include: the target configuration in the 1st round, the target configuration in the 2nd round, ..., the target configuration in the (j-1)-th round, and the target configuration in the j-th round.

[0174] In one possible implementation, the target configuration for round j is different from the target configurations for rounds j-1 before round j. Thus, the target configuration obtained in any round of search is a new configuration compared to all rounds prior to that round. The training data from the previous round is updated based on the relative errors corresponding to each operator under this configuration. For example, new data from the first model can be added to the training data from the previous round, allowing the error probability distribution model in the next round to more accurately infer the second probability distribution of each operator under each configuration, thereby improving the fit between the first probability distribution under each configuration and the actual first probability distribution of the first model under each configuration.

[0175] For example, after performing the second round of search, the target configuration obtained in the second round is different from the target configuration obtained in the first round. The target configuration obtained in the third round of search is different from the target configuration in the first round, and also different from the target configuration in the second round, and so on.

[0176] In one possible implementation, the first duration of the first model under the target configuration in round k is shorter than the first duration of the first model under the target configuration in round (k-1), where k is a positive integer greater than 1. Thus, during multiple rounds of search, as k increases, configurations with increasingly shorter first durations can be found, improving the efficiency of searching for the optimal parallel configuration. Here, the configuration determining device can determine Mk k-th configurations from Nk k-th configurations, where the first duration of each of the Mk k-th configurations is less than the first duration of the target configuration in round (k-1), and Mk is a positive integer less than or equal to Nk; then, one of the Mk k-th configurations is selected as the target configuration for round k.

[0177] In one possible implementation, after obtaining Nk k-th configurations, the configuration determining device can perform profiling on each of the Nk k-th configurations to obtain the true value of the first duration under each of the Nk k-th configurations; and perform profiling on the (k-1)-th round target configuration to obtain the true value of the first duration under the (k-1)-th round target configuration. The true value of the first duration under each of the Nk k-th configurations is compared with the true value of the first duration under the (k-1)-th round target configuration to determine the Mk k-th configurations with smaller true values ​​of the first duration under the (k-1)-th round target configuration, and select one of the Mk k-th configurations as the k-th round target configuration.

[0178] In one possible implementation, after obtaining Nk first configurations, the configuration determining device can, based on the first probability distribution of each configuration obtained in the kth round, obtain the value of the first duration with the maximum probability corresponding to the Nk kth configurations, and the value of the first duration with the maximum probability corresponding to the configuration with the same target configuration in the (k-1)th round. Based on the comparison between these two values, Mk kth configurations are determined. This eliminates the need for profiling, improving the efficiency of determining the target configuration for the kth round. The target configuration for the kth round is the one with the smallest true value of the first duration among the target configurations from the k rounds. After stopping the search in the kth round, the configuration determining device can directly use the target configuration for the kth round as the optimal parallel configuration.

[0179] Figure 7 A flowchart illustrating a method for determining the target configuration in round k is shown, as follows: Figure 7 As shown, the method may include: S501-S505.

[0180] S501. Randomly select one of the Nk k-th configurations as the recommended configuration.

[0181] S502. Determine whether the recommended configuration meets the time bottleneck improvement conditions for the historical target configuration in round k; if yes, execute S505; otherwise, execute S503.

[0182] It should be noted that the configuration determination device can compare the performance of the recommended configuration and the historical target configuration of the k-th round. If the recommended configuration outperforms the historical target configuration of the k-th round, it indicates that the recommended configuration meets the performance improvement conditions for the historical target configuration of the k-th round. The historical target configuration of the k-th round is the target configuration with the shortest first duration among the target configurations of the k-1 rounds prior to the k-th round; for example, the historical target configuration of the second target configuration is the first target configuration. Superior performance can be characterized by at least one of the following methods: reduction in the computation time of model operation, reduction in the communication time of model operation, and reduction in the first duration of the recommended configuration. Specifically, the configuration determination device can compare the configuration methods of the historical target configuration of the k-th round and the recommended configuration to obtain a configuration comparison result. If the configuration comparison result is the first comparison result, the configuration determination device can determine the first performance change result corresponding to the first comparison result from the correspondence between the configuration comparison result and the performance change result, and determine whether the performance of the first model under the recommended configuration is superior to that under the historical target configuration of the k-th round based on the first performance change result. The configuration comparison result represents the configuration change of the recommended configuration relative to the historical target configuration in round k. This comparison result can be characterized by changes in the proportion of various parallel strategies in the hybrid parallel configuration. For example, Table 2 shows a correspondence between the configuration comparison result and the performance change result. As shown in Table 2, different configuration comparison results correspond to different performance change results, including changes in computation time and communication time. Changes in computation time include increases, decreases, or no change in computation time, while changes in communication time include increases, decreases, or no change in communication time. If superior performance is characterized by a decrease in computation time, then a configuration comparison result of 1, 3, or 4 satisfies the performance improvement condition for the historical target configuration in round k. If superior performance is characterized by a decrease in communication time, then a configuration comparison result of 2 satisfies the performance improvement condition for the historical target configuration in round k.

[0183] Table 2

[0184] Here, mbs (micro batch size) is the micro batch size, which is obtained by splitting the global batch size of the pipeline parallel PP.

[0185] Optionally, the configuration determining device can profiling a given large model under the historical target configuration of the k-th round to obtain the real performance data of the first model under the historical target configuration of the k-th round. It then obtains the communication time and computation time from the real time data in the real performance data, and determines the overlapping communication time based on the communication time and computation time. The overlapping communication time is the time during which computation and communication occur simultaneously. The duration of the individual communication time is obtained by subtracting the duration of the overlapping communication time from the duration of the communication time. The proportion of the individual communication time in the total communication time is compared with the value of the individual communication threshold. If the proportion is greater than or equal to the individual communication threshold, it indicates that there is a communication bottleneck in the historical target configuration of the k-th round; otherwise, it indicates that there is a computation bottleneck in the historical target configuration of the k-th round. The individual communication threshold can be set according to actual needs, and this embodiment does not impose any restrictions.

[0186] Since the first duration is likely to decrease for any configuration with a computational or communication bottleneck, the configuration that can improve the computational or communication bottleneck will be less likely to be used. However, if the target configuration for the kth round is selected based on the performance improvement conditions, it is only necessary to profil the historical target configuration for the kth round, but not the target configuration for the kth round. This can greatly reduce the computational load of the configuration determination device when the target configuration for the kth round is found.

[0187] For example, Figure 8 A schematic diagram illustrating the computation and communication times for whole-network training is shown. Figure 8 As shown, the start time is the beginning time of the entire network training. Communication time is The calculation time is and The overlap communication time is and The duration of the communication is The duration of the overlapping communication time is Therefore, the duration of a single communication is .

[0188] If a communication bottleneck exists in the historical target configuration of round k, and the performance change result corresponding to the configuration comparison result indicates a reduction in communication time, then the configuration determining device can determine that the recommended configuration meets the performance improvement conditions for the historical target configuration of round k. If a computational bottleneck exists in the historical target configuration of round k, and the performance change result corresponding to the configuration comparison result indicates a reduction in computation time, then the configuration determining device can determine that the recommended configuration meets the performance improvement conditions for the historical target configuration of round k.

[0189] S503. Remove the first configuration that is the same as the recommended configuration from the Nk k-th configurations to obtain the set of recommended configurations.

[0190] S504. Randomly select one from the set of recommended configurations as the new recommended configuration, and return to S502.

[0191] If the recommended configuration does not meet the performance improvement conditions for the historical target configuration in the kth round, the configuration determination device needs to remove all kth configurations that are the same as the recommended configuration from the Nk kth configurations, and reselect one from the set of recommended configurations composed of the remaining kth configurations. In this way, the new recommended configuration is different from the previous recommended configuration, reducing the probability that the reselected recommended configuration does not meet the performance improvement conditions for the historical target configuration, and improving the efficiency of selecting the recommended configuration.

[0192] In this embodiment, the configuration determining device needs to select a recommended configuration from the set of recommended configurations. When selecting a recommended configuration for the first time, the Nk k-th configurations are directly combined as recommended configurations. Each time a recommended configuration is selected, the k-th configuration that is identical to it is removed from the set of recommended configurations, and the set of recommended configurations is updated. If the recommended configuration does not meet the performance improvement conditions for the historical target configuration in the k-th round, a new recommended configuration is selected from the set of recommended configurations until a configuration that meets the performance improvement conditions for the historical target configuration in the k-th round is selected.

[0193] S505. Use the recommended configuration as the target configuration for round k.

[0194] If the recommended configuration meets the performance improvement conditions for the historical target configuration in round k, the configuration determination device can determine the recommended configuration as the target configuration in round k.

[0195] It should be noted that the historical target configuration for round k+1 is the shorter of the historical target configuration for round k and the target configuration for round k.

[0196] Optionally, after determining the target configuration for the kth round, the configuration determining device can determine whether the remaining recommendable proportion is less than the recommendable threshold. If so, it can determine that Nk k-th configurations satisfy the stopping search condition; otherwise, it can determine that Nk k-th configurations do not satisfy the stopping search condition. The remaining recommendable proportion is the ratio of the number of k-th configurations in the recommendable configuration set corresponding to the target configuration for the kth round to Nk. The recommendable threshold can be set according to actual needs, and this embodiment does not impose any restrictions.

[0197] For example, Nk is 6, and the Nk k-th configurations are configuration 1, configuration 1, configuration 2, configuration 3, configuration 3, and configuration 4. The configuration determining device randomly selects configuration 3 from the 6 first configurations. If configuration 3 cannot improve the communication bottleneck compared to the historical target configuration in the k-th round, the configuration determining device needs to randomly select a new configuration from the recommended configuration set. The recommended configuration set includes configuration 1, configuration 1, configuration 2, and configuration 4. If the reselected recommended configuration is configuration 1, and configuration 1 can improve the communication bottleneck compared to the historical target configuration in the k-th round, then the configuration determining device can use configuration 4 as the target configuration for the k-th round. At this time, the first configurations in the recommended configuration set are configuration 2 and configuration 4, and the remaining recommended ratio is 1 / 3. If the recommended threshold is 0.2, the configuration determining device needs to update the first probability distribution under each configuration using configuration 3 and perform the (k+1)-th round of search. If the recommended threshold is 0.4, the configuration determination device can stop searching, select the one with the shortest true value of the first duration from the historical target configuration of the kth round and the target configuration of the kth round as the kth target configuration, and determine the kth target configuration as the optimal parallel configuration of the first model.

[0198] Based on the above embodiments, Figure 9 A flowchart illustrating a configuration determination method is shown, such as... Figure 9 As shown, the method includes, but is not limited to: S11-S32.

[0199] S11. Iterate through the input scales of each operator in the first model under M configurations to obtain multiple input scales.

[0200] Among them, multiple input scales are all the input scales corresponding to each operator under the M configurations of the first model.

[0201] S12. By reasoning about multiple input scales through the error probability distribution model, the error probability distribution corresponding to each input scale is obtained.

[0202] S13. Based on the input size of each operator under M configurations and the error probability distribution corresponding to each input size, determine the error probability distribution corresponding to each operator under each configuration.

[0203] S14. Based on the error probability distribution of a single operator under each configuration, revise the predicted value of the second duration of the operator under the corresponding configuration to obtain the second probability distribution of the single operator under each configuration.

[0204] S15. Combine the second probability distributions of each operator under a single configuration to obtain the first probability distribution of that configuration.

[0205] S16. Sample the first probability distribution for each configuration N times to obtain N There are M possible values ​​for the first duration, each value representing a first duration.

[0206] S17. From the M values ​​of the first duration corresponding to a single sampling, select the first configuration corresponding to the minimum value as a first configuration corresponding to the single sampling; N samplings correspond to N first configurations.

[0207] For example, Figure 10 A schematic diagram illustrating the selection of N first configurations is shown, such as... Figure 10 As shown, N samples yield N There are M possible values ​​for the first duration. The minimum value among the M values ​​obtained in the first sampling is the first duration corresponding to configuration 2, and configuration 2 is the first configuration corresponding to the first sampling. The minimum value among the M values ​​obtained in the second sampling is the first duration corresponding to configuration 3, and configuration 3 is the first configuration corresponding to the second sampling. The minimum value among the M values ​​obtained in the third sampling is the first duration corresponding to configuration 1, and configuration 1 is the first configuration corresponding to the third sampling. And so on, N first configurations can be obtained.

[0208] It should be noted that for any configuration corresponding to N sampled values, it is sufficient to have one sampled value for each sample. The specific daily sampled value corresponding to each sample can be randomly assigned, and this application does not impose any restrictions on this.

[0209] S18. Randomly select one from N first configurations as a candidate target configuration.

[0210] Among them, the candidate target configuration is the target configuration of the first round.

[0211] S19. Obtain the true value of the second duration of each operator under the candidate target configuration.

[0212] S20. Based on the true value of the second duration of each operator under the candidate target configuration and the predicted value of the second duration of each operator under the candidate target configuration, determine the correspondence between the input size and the relative error under the first round target configuration.

[0213] S21. The correspondence between the input size and relative error under the first round of target configuration is added to the training data, and the Gaussian distribution model is retrained to obtain a new error probability distribution model.

[0214] S22. By inferring from multiple input scales using a new error probability distribution model, a new error probability distribution corresponding to each input scale is obtained.

[0215] S23. Based on the new error probability distribution corresponding to each input size, determine the new first probability distribution corresponding to each of the multiple configurations.

[0216] The implementation method of S23 is described in S13-S15, and will not be repeated here.

[0217] S24. Sample the new first probability distribution corresponding to each configuration N times to obtain the new N. There are M possible values, each of which represents a first duration.

[0218] S25. Based on the new N M values ​​determine N new first configurations, and the N new first configurations form a set of recommended configurations.

[0219] S26. Randomly select one from the set of recommended configurations as the recommended configuration.

[0220] S27. Remove the first configuration that is the same as the recommended configuration from the set of recommended configurations to obtain a new set of recommended configurations.

[0221] S28. Determine whether the recommended configuration meets the stop search condition for the historical target configuration. If yes, execute S29; otherwise, execute S26.

[0222] S29. The new recommended configuration is used as a new candidate target configuration.

[0223] The new candidate target configuration obtained in the k-th round of search is the target configuration in the k-th round. K rounds of search can yield k candidate target configurations.

[0224] S30. Determine whether the remaining recommendable proportion is less than the recommendable threshold. If so, execute S31; otherwise, continue executing S19 based on the new candidate target configuration.

[0225] S31. Stop searching.

[0226] In this embodiment of the application, the configuration determining device can select the one with the shortest actual value of the first duration from the k candidate target configurations that correspond one-to-one with the k rounds as the optimal parallel configuration.

[0227] In this embodiment of the application, after determining the optimal parallel configuration, the configuration determining device can send notification information to the computing nodes running the first model; in this way, each computing node can run according to the configuration indicated by the notification information.

[0228] For example, such as Figure 11As shown, the configuration determining device 1000 is applied to computer 1 in cluster 10. Computers 2 and 3 are computing nodes for the first model, and each of computers 2 and 3 deploys the first model; the first model includes two layers. When determining the optimal parallel configuration as running the first layer of the first model on computer 2 and the second layer of the first model on computer 3, the configuration determining device 1000 can broadcast notification information to computers 2 and 3, instructing computer 2 to run the first layer of the first model and computer 3 to run the second layer of the first model. Cluster 10 can be a GPU cluster, where each computer in the GPU cluster is configured with a GPU.

[0229] Optionally, the configuration determination device can run on a single card, while the first model can be deployed on multiple other cards, with each card serving as a computing node for the first model, and all cards interconnected. After determining the optimal parallel configuration, the configuration determination device can send notification information to the other cards, instructing them on the configuration of the first model on each of the other cards, thus enabling the other cards to achieve the optimal parallel configuration. In this way, the other cards can run the first model with the optimal parallel configuration.

[0230] Figure 12 A schematic block diagram of a configuration determination device 1000 according to an embodiment of this application is shown. The device 1000 may include: The acquisition module 1010 is used to acquire a first probability distribution under each of multiple configurations, the first probability distribution being used to indicate the probability distribution of a first duration; the first duration includes the forward propagation duration of the first model under the corresponding configuration. The determining module 1020 is used to determine a first target configuration of the first model based on a first probability distribution under each of the multiple configurations, wherein the first target configuration is one of the multiple configurations.

[0231] In one possible implementation, the first model includes a plurality of operators, each of which corresponds to a second probability distribution in each configuration; the second probability distribution is used to indicate the probability distribution of a second duration, which is the computation duration of an operator; the first probability distribution in each of the plurality of configurations is determined based on the second probability distribution of the plurality of operators of the first model in the corresponding configuration.

[0232] In one possible implementation, each of the plurality of operators corresponds to an error probability distribution for each configuration, the error probability distribution indicating the probability distribution of the error between the predicted value and the true value of the computation time of an operator; the second probability distribution corresponding to each of the plurality of operators for each configuration is determined based on the corresponding error probability distribution and the predicted value of the corresponding second time.

[0233] In one possible implementation, the error probability distribution corresponding to each of the plurality of operators in each configuration is obtained through probability distribution model inference.

[0234] In one possible implementation, the probability distribution model is trained based on training data, which includes the input size of the operator and the error between the operator's predicted value and the true value.

[0235] In one possible implementation, the multiple configurations are M configurations, where M is a positive integer greater than 1; the determining module 1020 is further configured to determine based on N M values ​​of the first duration are used to determine N first configurations; the first probability distribution of each configuration corresponds to N values ​​of the first duration, the first configuration is one of the multiple configurations, and N is a positive integer greater than 1; based on the N first configurations, the first target configuration of the first model is determined.

[0236] In one possible implementation, the first configuration is the configuration corresponding to the minimum value among the M values ​​of the first duration, and the M values ​​correspond one-to-one with the M configurations.

[0237] In one possible implementation, the acquisition module 1010 is further configured to acquire the actual duration data of the first model under the first target configuration; the determination module 1020 is further configured to update the first probability distribution under each of the multiple configurations based on the actual duration data; determine the second target configuration of the first model from the multiple configurations based on the updated first probability distribution; and determine the third target configuration of the first model based on the first target configuration and the second target configuration.

[0238] In one possible implementation, the first target configuration is determined based on N1 first configurations, wherein the N1 first configurations are determined based on a first probability distribution of the plurality of configurations before the update and N1 values ​​of the first duration; the second target configuration is determined based on N2 second configurations, wherein the N2 second configurations are determined based on a first probability distribution of the plurality of configurations after the update and N2 values ​​of the first duration; wherein the first configuration and the second configuration are both one of the plurality of configurations; and N1 and N2 are both positive integers greater than 1.

[0239] In one possible implementation, updating the first probability distribution for each of the multiple configurations based on the true value of the first duration is performed when the N first configurations do not meet the stopping search condition. The stopping search condition includes at least one of the following: the proportion of the configuration with the largest proportion among the N first configurations is greater than a first proportion threshold, where the configuration with the largest proportion among the N first configurations is the first configuration with the largest proportion. The N first configurations are N times the first duration corresponding to the first probability distribution of the M configurations. The total number of all configuration types in the N first configurations is less than the threshold number of types, determined by M values.

[0240] In one possible implementation, the first duration of the first model under the second target configuration is less than the first duration of the first model under the first target configuration.

[0241] Figure 13 A schematic block diagram of another computing device 1100 according to an embodiment of this application is shown. The computing device 1100 may include: The processor 1110 implements the communication method provided in the embodiments of this application through logic circuits or by executing code instructions.

[0242] Optionally, the computing device 1100 may also include interface circuitry 1120. Processor 1110 and interface circuitry 1120 are coupled to each other. It is understood that interface circuitry 1120 may be a transceiver or an input / output interface.

[0243] Optionally, the computing device 1100 may also include a memory 1130 for storing instructions executed by the processor 1010, or storing input data required by the processor 1110 to execute instructions, or storing data generated after the processor 1110 executes instructions.

[0244] The aforementioned processor 1110 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by software instructions. The aforementioned processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0245] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0246] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments.

[0247] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the methods described in the above embodiments.

[0248] In addition, the device 1000 in the above method embodiment can be applied in the computing device 1100. The device 1000 can specifically be a chip, component or module.

[0249] In this embodiment, the computing device, computer storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0250] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0251] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0252] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0253] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0254] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0255] It should be understood that in the embodiments of this application, the designations "first", "second", etc. are only for distinguishing different objects, such as different terminal devices or different network devices, and do not constitute a limitation on the scope of the embodiments of this application. The embodiments of this application are not limited thereto.

[0256] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. The term "at least one" in this application can represent "one" and "two or more." For example, A, B, and C can represent: A existing alone, B existing alone, C existing alone, A and B existing simultaneously, A and C existing simultaneously, C and B existing simultaneously, and A, B, and C existing simultaneously.

[0257] In this embodiment of the application, expressions such as "A includes B" are used to indicate that A may or may not include other items besides B. When other items are not included, it can be understood as "A is B", in which case "A" can be replaced with "B".

[0258] In this application, unless otherwise specified, the same or similar parts between the various embodiments can be referred to each other. In the various embodiments of this application, and in the various implementation methods / methods / implementations within each embodiment, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments and between the various implementation methods / methods / implementations within each embodiment are consistent and can be mutually referenced. The technical features in different embodiments and the various implementation methods / methods / implementations within each embodiment can be combined according to their inherent logical relationships to form new embodiments, implementation methods, methods, or implementation approaches. The embodiments described below do not constitute a limitation on the scope of protection of this application.

[0259] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0260] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0262] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0263] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0264] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0265] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A configuration determination method, characterized in that, include: Obtain the first probability distribution under each of the multiple configurations of the first model, and use the first probability distribution under the corresponding configuration to indicate the probability distribution of the first duration; The first duration includes the forward propagation duration of the first model under the corresponding configuration, and the configuration includes the configuration of the first model performing computation on the computing node; Based on the first probability distribution under each of the multiple configurations, a first target configuration of the first model is determined, wherein the first target configuration is one of the multiple configurations.

2. The method according to claim 1, characterized in that, The first model includes multiple operators, each of which corresponds to a second probability distribution under each configuration; the second probability distribution is used to indicate the probability distribution of a second duration, which is the computation time of an operator; The first probability distribution under each of the multiple configurations is determined based on the second probability distribution of multiple operators of the first model under the corresponding configuration.

3. The method according to claim 2, characterized in that, Each of the plurality of operators corresponds to an error probability distribution under each configuration. The error probability distribution is used to indicate the probability distribution of the error between the predicted value and the true value of the computation time of an operator. The second probability distribution for each of the plurality of operators in each configuration is determined based on the corresponding error probability distribution and the predicted value of the corresponding second duration.

4. The method according to claim 3, characterized in that, The error probability distribution for each of the plurality of operators in each configuration is obtained through inference from an error probability distribution model.

5. The method according to claim 4, characterized in that, The error probability distribution model is obtained by training based on training data; the training data includes the input size of the operator and the error between the operator's predicted value and the true value.

6. The method according to any one of claims 1 to 5, characterized in that, The multiple configurations are M types, where M is a positive integer greater than 1; determining the first target configuration of the first model based on the first probability distribution under each of the multiple configurations includes: Based on N M values ​​of the first duration are used to determine N first configurations; the first probability distribution of each configuration corresponds to N values ​​of the first duration, the first configuration is one of the multiple configurations, and N is a positive integer greater than 1; Based on the N first configurations, the first target configuration of the first model is determined.

7. The method according to claim 6, characterized in that, The first configuration is the configuration corresponding to the minimum value among the M values ​​of the first duration, and the M values ​​correspond one-to-one with the M configurations.

8. The method according to claim 6 or 7, characterized in that, The first target configuration is the configuration with the largest proportion among the N first configurations.

9. The method according to claim 1, characterized in that, Also includes: Obtain the actual duration data of the first model under the first target configuration; The first probability distribution under each of the multiple configurations is updated based on the actual duration data. Based on the updated first probability distribution, a second target configuration for the first model is determined from the multiple configurations; A third target configuration for the first model is determined based on the first target configuration and the second target configuration.

10. The method according to claim 9, characterized in that, The first target configuration is determined based on N1 first configurations, and the N1 first configurations are determined based on the first probability distribution of the multiple configurations before the update and N1 values ​​of the first duration; The second target configuration is determined based on N2 second configurations, which are determined based on the updated first probability distribution of the multiple configurations and N2 values ​​of the first duration. Wherein, the first configuration and the second configuration are both one of the multiple configurations; N1 and N2 are both positive integers greater than 1.

11. The method according to claim 9, characterized in that, Also includes: The step of updating the first probability distribution for each of the multiple configurations based on the real duration data is performed when the N first configurations do not meet the stopping search condition, which includes at least one of the following: The maximum proportion configuration has a proportion greater than a first proportion threshold among the N first configurations, and the maximum proportion configuration is the first configuration with the largest proportion among the N first configurations; the N first configurations are N times the first duration corresponding to the first probability distribution under the M configurations. Determined by M possible values; The total number of types of all configurations in the N first configurations is less than the threshold number of types.

12. The method according to claim 9, characterized in that, Also includes: The first duration of the first model under the second target configuration is less than the first duration of the first model under the first target configuration.

13. A computing device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, Includes a computer program that, when the computer program is run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 12.

15. A computer program product, characterized in that, The method includes a computer program that, when executed by an electronic device, causes the electronic device to perform the method according to any one of claims 1 to 12.

Images