Model training strategy determination method and apparatus, electronic device, medium, and product

Abstract

Embodiments of the present disclosure disclose a model training strategy determination method and device, electronic equipment, medium and product. The method comprises: obtaining training configuration information corresponding to a to-be-trained model; obtaining a plurality of available parallel strategies and parallel optimization criteria corresponding to the to-be-trained model based on the training configuration information; determining at least one selectable parallel strategy from the plurality of available parallel strategies based on the training configuration information and the parallel optimization criteria; determining a strategy evaluation index of the at least one selectable parallel strategy, and determining a target parallel strategy from the at least one selectable parallel strategy based on the at least one strategy evaluation index, so as to train the to-be-trained model based on the target parallel strategy and a target cluster. The technical solution can select an optimal target parallel strategy that is adapted to the cluster and the model by relying on complete training configuration to obtain parallel strategies and performing layer-by-layer screening and evaluation, thereby effectively improving the rationality, resource utilization rate and overall training efficiency of distributed training.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, medium, and product for determining a model training strategy. Background Technology

[0002] With the rapid development of large-parameter artificial intelligence models, distributed heterogeneous clusters have become the mainstream hardware platform for model training. Clusters often utilize a mix of computing power chips, such as Graphics Processing Units (GPUs) and Neural Processing Units (NPUs), and employ various parallel strategies, including data, tensor, pipeline, and expert parallelism, to split the model and data, thereby distributing computing power and memory pressure and ensuring normal model training. Currently, the industry generally adopts combined parallel methods to improve training efficiency. Therefore, how to match reasonable parallel strategies based on the characteristics of heterogeneous hardware has become a core research direction in the field of distributed model training. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, medium, and product for determining a model training strategy, in order to solve the problem that the determined model training strategy is difficult to adapt to heterogeneous clusters or complex model structures, resulting in poor strategy adaptability, unbalanced resource load, and poor model training effect.

[0004] According to one aspect of this disclosure, a method for determining a model training strategy is provided, the method comprising: Obtain training configuration information corresponding to the model to be trained, wherein the training configuration information is used to describe the model training configuration corresponding to the model to be trained; the training configuration information includes cluster configuration information of the target cluster; Based on the training configuration information, a variety of available parallel strategies and parallel optimization criteria corresponding to the model to be trained are obtained. The available parallel strategies are used to indicate the parallel training methods that can be used to train the model to be trained based on the target cluster and their related adjustable parameters. Based on the training configuration information and the parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies; Determine a policy evaluation metric for at least one of the optional parallel strategies, and based on at least one of the policy evaluation metric, determine a target parallel strategy from at least one of the optional parallel strategies, so as to train the model to be trained based on the target parallel strategy.

[0005] According to another aspect of this disclosure, a model training strategy determination apparatus is provided, the apparatus comprising: The first module is used to obtain training configuration information corresponding to the model to be trained. The training configuration information describes the model training configuration corresponding to the model to be trained. The training configuration information includes the cluster configuration information of the target cluster. The second module is used to obtain, based on the training configuration information, multiple available parallel strategies and parallel optimization criteria corresponding to the model to be trained. The available parallel strategies are used to indicate the parallel training methods and their related adjustable parameters that can be used to train the model to be trained based on the target cluster. The third module is used to determine at least one optional parallel strategy from a variety of available parallel strategies based on the training configuration information and the parallel optimization criteria. The fourth module is used to determine a policy evaluation index for at least one of the optional parallel strategies, and based on the at least one policy evaluation index, to determine a target parallel strategy from the at least one of the optional parallel strategies, so as to train the model to be trained based on the target parallel strategy.

[0006] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: One or more processors; Storage device for storing one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement a model training strategy determination method as described in any of the embodiments of this disclosure.

[0007] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement any of the model training strategy determination methods in the embodiments of this disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, which, when executed by a processor, implements a model training strategy determination method as described in any of the embodiments of the present disclosure.

[0009] The technical solution of this disclosure collects complete training configuration information, enabling a comprehensive understanding of the basic conditions of the model and cluster. This provides an accurate basis for subsequent parallel strategy generation and selection rules, avoiding mismatches between strategies and the actual environment due to missing information. Furthermore, obtaining a parallel training method suitable for the cluster based on the training configuration information ensures that all available parallel strategies have the foundation for implementation. Moreover, clearly defining parallel optimization criteria provides a unified standard for subsequent selection work, ensuring consistency in the selection direction. Furthermore, combining training configuration information and parallel optimization criteria for preliminary selection can quickly eliminate obviously unsuitable parallel strategies, narrowing the scope of subsequent evaluation, reducing overall processing workload, and improving process efficiency. Furthermore, conducting in-depth analysis of available strategies through dedicated evaluation indicators allows for the selection of the strategy with the best overall performance from the alternatives, making the model training process more aligned with preset requirements and ensuring stable and efficient training. The technical solution provided in this disclosure addresses the limitations of existing model training strategies, which are difficult to adapt to heterogeneous clusters or complex model structures, resulting in poor strategy adaptability, unbalanced resource load, and poor model training performance. It effectively adapts to heterogeneous clusters and complex model structures, improves the overall adaptability of parallel training strategies, balances cluster resource load, and thus improves model training quality. Furthermore, by using hierarchical screening and indicator evaluation to select the best strategy at each level, it simplifies the strategy selection process and selects the target parallel strategy that fits the optimization objective, thereby improving the rationality and operational efficiency of parallel model training.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a method for determining a model training strategy provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating another method for determining a model training strategy provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating another method for determining a model training strategy provided in this embodiment of the disclosure; Figure 4This is a schematic diagram of the structure of a model training strategy determination device provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0016] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0017] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0018] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0019] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0020] Before introducing this technical solution, we can first illustrate the application scenarios. This technical solution can be applied to the intelligent adaptation of parallel strategies for distributed training of machine learning models in heterogeneous clusters. For example, it can be widely used in the computing power acceleration stages of pre-training, fine-tuning and incremental training of machine learning models, multimodal models and hybrid expert models, and can be adapted to the large-scale training and intelligent deployment scenarios of heterogeneous hybrid computing power clusters of GPU and NPU.

[0021] Currently, the configuration of parallel training strategies in heterogeneous cluster scenarios typically relies on manual selection and parameter tuning based on human experience. This is an experience-driven, passive configuration mode that lacks standardized and automated intelligent optimization mechanisms. Furthermore, most existing parallel training strategy optimization schemes are designed for homogeneous clusters with similar equipment, resulting in limited adaptability and difficulty in compatibility with heterogeneous training environments featuring mixed GPU and NPU networks. The aforementioned parallel training strategy configuration methods have significant drawbacks: manual strategy selection is highly subjective and random, failing to simultaneously consider multi-dimensional training optimization objectives such as memory usage, communication overhead, and computing power utilization; it cannot quantify the differences in training complexity caused by model structure, parallel dimensions, and heterogeneous cluster topology, easily leading to problems such as poor parallel strategy adaptability, cluster load balancing, resource waste, training stuttering, and even memory overflow; moreover, the determination of parallel training strategies has certain limitations, making it difficult to dynamically optimize based on real-world training scenarios, ultimately resulting in low overall training efficiency and low resource utilization in heterogeneous clusters.

[0022] Based on this, this embodiment proposes a method for determining model training strategies. It constructs complete training configuration information by automatically detecting cluster nodes and network features, adaptively generates available parallel strategies and parallel optimization criteria suitable for the current model and cluster, and combines the quantified training complexity to achieve hierarchical screening, multi-dimensional evaluation, and intelligent selection, ultimately determining the optimal target parallel strategy. This solution can adapt to various heterogeneous computing power clusters, eliminating reliance on manual parameter tuning experience, and taking into account multi-dimensional optimization objectives such as training speed, resource consumption, load balancing, and communication efficiency. It effectively solves the technical problems of poor adaptability and low resource utilization of traditional parallel training strategies, achieving automated adaptation, accurate matching, and globally optimal configuration of distributed parallel training strategies for models. This provides reliable technical support for efficient, stable, and low-cost intelligent training of models under heterogeneous clusters.

[0023] Figure 1 This is a flowchart illustrating a model training strategy determination method provided in an embodiment of this disclosure. This embodiment is applicable to distributed parallel training scenarios for machine learning models, especially to scenarios involving large-scale model training in heterogeneous clusters with varying hardware specifications and complex network environments. The method can be executed by a model training strategy determination device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers or servers. Figure 1 As shown, the method in this embodiment includes: S110. Obtain the training configuration information corresponding to the model to be trained. The training configuration information is used to describe the model training configuration corresponding to the model to be trained. The training configuration information includes at least the cluster configuration information of the target cluster.

[0024] In one scenario, the model to be trained can be understood as the subject to be trained. The model to be trained can be a machine learning model undergoing iterative parameter training. It can also be an artificial intelligence model that has not yet completed training and requires parameter learning and iterative optimization using a computing cluster. The model to be trained can be a machine learning model with a large number of parameters, such as a large language model, a multimodal model, or a hybrid expert model; it can also be a relatively simple, lightweight business model that only requires fine-tuning of local parameters, such as an image recognition model, a text classification model, or a time series prediction model.

[0025] In one scenario, training configuration information can be understood as a set of information constraining the entire model training process. Training configuration information can integrate various parameters and information related to the model's own attributes, training constraints, and the operating hardware and network environment. It serves as the foundation for conducting distributed model training, defining the overall operating conditions for model training. Furthermore, it can describe the model training configuration corresponding to the model to be trained. In other words, the model training configuration can be the rules, parameters, and environmental requirements to be followed when training the model, and the training configuration information can be the information carrier containing these elements, presenting the entire training configuration intuitively and completely by integrating various types of data such as model, cluster, and network data. Optionally, training configuration information may include at least one of the following: model structure information, training constraints, target cluster configuration information, and network communication parameters.

[0026] In one scenario, a target cluster can be understood as a distributed computing power cluster formed by a network of multiple computing nodes, specifically designed to carry out model training tasks. The target cluster can be a dedicated hardware environment selected based on training requirements. It can consist of multiple interconnected independent target nodes, containing different types of target nodes, providing distributed operation, computation, and storage support for the model to be trained. Simply put, a target cluster can be used to train a model; it is a distributed cluster used for training a model. A target cluster can include multiple different target nodes used for training the model; alternatively, it can also include multiple identical target nodes used for training the model. The node types of target nodes include at least one of the following: hardware devices, graphics processing unit (GPU) nodes, neural network processor (NN) nodes, and general-purpose computing nodes. For example, a target cluster can be a server group used for distributed training of large language models and multimodal models, formed by a hybrid network of GPU and NN nodes.

[0027] In one scenario, cluster configuration information can be understood as a set of parameters used to record the status of each computing node and the overall state of the target cluster. Cluster configuration information can comprehensively describe the node type, number, topology, and hardware performance attributes within the target cluster. It fully reflects the cluster's hardware structure and network configuration. Cluster configuration information can be a crucial basis for judging the cluster's computing power and conducting parallel training. Optionally, cluster configuration information may include at least one of the following: node device type, number of node devices, node distribution topology, node computing power parameters, node memory capacity, and node load threshold.

[0028] In one scenario, the provided solution may include obtaining training configuration information corresponding to the model to be trained, which may include at least one of the following: receiving a model training task for the model to be trained issued by a user, and obtaining training configuration information corresponding to the model to be trained based on the model training task; reading a locally stored configuration file, and obtaining model configuration information corresponding to the model to be trained based on the configuration file; calling a preset model training template, and performing parameter adaptation between the model to be trained and the target cluster based on the model training template to obtain training configuration information.

[0029] In one scenario, the system receives a user's configuration operation for the training task of the model to be trained, and generates a model training task corresponding to the model to be trained based on the configuration operation. Further, in response to receiving the model training task corresponding to the model to be trained, the system parses the model training task to obtain basic configuration information. Further still, it traverses the target nodes within the target cluster based on the basic configuration information to obtain the cluster configuration information corresponding to the target cluster. Further still, the system can obtain the training configuration information corresponding to the model to be trained based on the basic configuration information and the cluster configuration information.

[0030] S120. Based on the training configuration information, obtain a variety of available parallel strategies and parallel optimization criteria corresponding to the model to be trained.

[0031] In one context, available parallel strategies can be understood as parallel execution schemes that can be adopted for distributed model training. Available parallel strategies can be specific implementation methods for parallel model training based on a target cluster. Available parallel strategies can be used to indicate parallel training methods that can train the model to be trained based on the target cluster and their related adjustable parameters. That is, available parallel strategies can include parallel training methods that can perform parallel model training on the target cluster, and the adjustable parameters corresponding to these parallel training methods. Parallel training methods can be understood as execution modes where multiple nodes collaboratively complete training. Parallel training methods can be specific operational forms that split the training task and distribute it to different nodes within the target cluster for collaborative computation. Parallel training methods can be splitting and distributing training data to various nodes for computation, or deploying the model to be trained hierarchically across different nodes for computation. Optionally, available parallel methods can include at least one of the following: data parallelism, tensor parallelism, pipelined parallelism, expert parallelism, and context parallelism. Among these, data parallelism (DP) can involve dividing training data into different nodes, with each node executing the same model copy, independently calculating gradients, and then synchronizing gradients through aggregate communication. Tensor parallelism (TP) involves partitioning the linear layer weight matrix of a model into blocks, allowing multiple devices to execute corresponding sub-operations in parallel, reducing the memory pressure on a single device and making it suitable for training models with extremely large numbers of parameters. Pipeline parallelism (PP) divides the model into multiple stages by layer or sub-module, mapping them to different devices and using a pipelined sequential scheduling to process small batches of samples, achieving cross-node pipeline acceleration. Expert parallelism (EP) can be applied to sparse gating models, where each node is responsible for a set of expert subnetworks, dynamically selecting a subset of experts to participate in computation, achieving sparse activation and balanced computational load. Context parallelism (CP) can be used for multi-input context or sample parallel scenarios, performing forward / backward propagation of each context in parallel on independent nodes to improve the efficiency of long sequences or multi-sample training. Adjustable parameters can be understood as configuration items that support manual or automatic adjustment. Adjustable parameters are parameters that are compatible with parallel training methods and can be modified to optimize training results. Optionally, adjustable parameters may include at least one of the following: number of node groups, batch size, number of pipeline segments, gradient update frequency, communication interval, iteration step size, and learning rate.

[0032] In one context, parallel optimization criteria can be understood as the criteria and optimization objectives for screening and evaluating available parallel strategies. Parallel optimization criteria can be evaluation indicators and constraint rules formulated for available parallel strategies. They can be used to compare the performance of various available parallel strategies, providing a reference for parameter tuning and strategy selection, thus balancing training strategy and operational stability.

[0033] In one scenario, the provided solution obtains multiple available parallel strategies corresponding to the model to be trained based on training configuration information, which may include at least one of the following: determining multiple parallel training methods based on training configuration information, matching corresponding adjustable parameters for each parallel training method to obtain multiple available parallel strategies; acquiring historical training data, and obtaining multiple existing available parallel strategies from the historical training data based on training configuration information; generating multiple available parallel strategies corresponding to the model to be trained based on a preset parallel strategy template and training configuration information; receiving multiple parallel training methods selected by the user, and configuring corresponding adjustable parameters for each parallel training method based on training configuration information to obtain multiple available parallel strategies. One of these methods is described in detail below.

[0034] Optionally, based on the training configuration information, multiple available parallel strategies corresponding to the model to be trained are obtained, including: based on the training configuration information, determining multiple parallel training methods corresponding to the model to be trained; for each parallel training method, determining adjustable parameters related to the parallel training method, and obtaining available parallel strategies based on the parallel training method and the adjustable parameters.

[0035] In one scenario, given the training configuration information corresponding to the model to be trained, this information is parsed to obtain the model structure information, training data size, training constraints, and target cluster configuration information. Further, based on the model structure information, data size, and cluster configuration information, multiple parallel training methods suitable for the model to be trained can be selected from existing parallel training methods. Further, for each parallel training method, the cluster configuration information is first parsed to clarify the node type, computing power, storage, and network communication capabilities. Then, combined with the training constraints, resource limits, time requirements, synchronization rules, and other restrictions are defined. Based on this, and considering the operational characteristics of this parallel training method, adjustable parameters such as batch size, number of segments, and synchronization frequency are calculated and set one by one to ensure that the parameter values ​​conform to the node capabilities and training constraints, thus obtaining adjustable parameters corresponding to this parallel training method. Finally, the parallel training methods and their corresponding adjustable parameters are combined to obtain usable parallel strategies.

[0036] S130. Based on training configuration information and parallel optimization criteria, determine at least one optional parallel strategy from a variety of available parallel strategies.

[0037] In one scenario, optional parallel strategies can be understood as candidate parallel strategies initially selected from multiple available parallel strategies based on training configuration information and parallel optimization criteria. In other words, the determined optional parallel strategies can be available parallel strategies that meet the basic operating conditions of the model to be trained and align with the overall optimization direction, and are used in subsequent refined evaluation stages. For example, based on training configuration information and parallel optimization criteria, data parallelism and pipelined parallelism are initially selected from eight available parallel strategies.

[0038] In one scenario, the provided scheme determines at least one optional parallel strategy from multiple available parallel strategies based on training configuration information and parallel optimization criteria. This includes at least one of the following: screening multiple available parallel strategies based on parallel optimization criteria to determine optional parallel strategies that meet the criteria; determining the resource consumption of each available parallel strategy based on training configuration information, and using parallel optimization criteria to screen optional parallel strategies that meet the requirements based on resource consumption; determining the training complexity of the model to be trained based on training configuration information, and determining optional parallel strategies from multiple available parallel strategies based on training complexity and parallel optimization criteria. One such determination method is described in detail below.

[0039] Optionally, based on training configuration information and parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies, including: determining the training complexity corresponding to the model to be trained based on training configuration information and a variety of available parallel strategies; and determining at least one optional parallel strategy from a variety of available parallel strategies based on training complexity and parallel optimization criteria.

[0040] In one scenario, the model structural complexity can be determined based on training configuration information, and the parallel strategy complexity can be determined based on multiple available parallel strategies. Furthermore, the training complexity corresponding to the model to be trained can be determined based on the model structural complexity and the parallel strategy complexity. Further, the strategy selection method corresponding to this training complexity can be determined from multiple optional strategy selection methods. Further still, the determined strategy selection method can be used to select multiple available parallel strategies according to parallel optimization criteria to obtain at least one available parallel strategy that satisfies the parallel optimization criteria.

[0041] S140. Determine a policy evaluation index for at least one optional parallel strategy, and based on the at least one policy evaluation index, determine a target parallel strategy from at least one optional parallel strategy, so as to train the model to be trained based on the target parallel strategy and the target cluster.

[0042] In one context, strategy evaluation metrics can be understood as criteria used to quantitatively compare candidate parallel strategies. Strategy evaluation metrics can be used to intuitively characterize the performance, resource consumption, and other aspects of selectable parallel strategies. Strategy evaluation metrics can include single-dimensional or multi-dimensional integrated metrics. Optionally, strategy evaluation metrics may include at least one of the following: computing power utilization, communication efficiency, load balancing, resource utilization, and training convergence speed.

[0043] In one context, a target parallel strategy can be understood as the final selected and implemented parallel training strategy. The target parallel strategy can be the final solution determined through comparison and selection using strategy evaluation metrics, serving as the formal execution basis for training the model to be trained. The target parallel strategy can be used for distributed training of the model to be trained.

[0044] In one scenario, the provided scheme may include single-dimensional indicators and multi-dimensional integrated indicators for strategy evaluation metrics. Furthermore, determining the strategy evaluation metrics corresponding to the optional parallel strategies includes at least one of the following: determining the evaluation metric for the optional parallel strategy on a single evaluation dimension, and using the determined evaluation metric as the strategy evaluation metric for the optional parallel strategy; or determining the evaluation metric for the optional parallel strategy on multiple evaluation dimensions, and obtaining the strategy evaluation metric for the optional parallel strategy based on the multiple evaluation metrics. One such determination method is described in detail below.

[0045] Optionally, determining at least one optional parallel strategy evaluation metric includes: for at least one optional parallel strategy, determining the evaluation metrics of the optional parallel strategy on multiple evaluation dimensions; and obtaining the strategy evaluation metric corresponding to the optional parallel strategy based on the multiple evaluation metrics.

[0046] In one scenario, an evaluation dimension can be understood as an independent evaluation direction for assessing the performance of a parallel strategy. Evaluation dimensions can be independent evaluation directions defined according to different characteristics of training and execution, used to comprehensively break down evaluation criteria. Specific quantitative indicators can be derived from each evaluation dimension. Optionally, evaluation dimensions may include at least one of the following: a first evaluation dimension, a second evaluation dimension, a third evaluation dimension, a fourth evaluation dimension, and a fifth evaluation dimension. It should be noted that evaluation dimensions can be dynamically added or deleted, and multiple evaluation dimensions can be flexibly combined according to business needs.

[0047] The first evaluation dimension can be understood as a measurement direction for the cluster's computing power utilization. The first evaluation dimension can be used to indicate the degree of utilization of the cluster node's computing power during the execution of optional parallel strategies. For example, the first evaluation dimension can be a computing power utilization score. The method for determining the computing power utilization score can include: dividing the floating-point operation volume of each operator through a statistical strategy and comparing it with the peak computing power library of the cluster nodes to obtain the computing power utilization score.

[0048] The second evaluation dimension can be understood as a measurement direction for evaluating the communication efficiency between cluster nodes. The second evaluation dimension can be used to indicate the data transmission efficiency between cluster nodes during the execution of optional parallel strategies. For example, the second evaluation dimension can be a communication efficiency score. The determination of the communication efficiency score can include: calculating the total data transmission volume between cluster nodes according to the task partitioning method of the parallel strategy, and then combining this with the bandwidth and latency parameters of the node links to finally calculate the communication efficiency score.

[0049] The third evaluation dimension can be understood as a measure of the balance in training task allocation. This third evaluation dimension can indicate the degree of balance in task allocation among cluster nodes under an optional parallel strategy. For example, the third evaluation dimension could be a load balancing score. The load balancing score can be determined by: calculating the total time taken by each cluster node in the target cluster to complete the training task, and then calculating the standard deviation of the total time taken by each cluster node. The smaller the standard deviation, the more even the task load among the cluster nodes, and the higher the load balancing score.

[0050] The fourth evaluation dimension can be understood as a measure of the resource consumption of optional running strategies. This fourth evaluation dimension can indicate the resource consumption of optional parallel strategies. For example, it can be a resource-saving score, which can be determined by: based on the optional parallel strategy meeting predetermined training requirements, determining the actual resource consumption ratio of nodes; the smaller the actual consumption ratio, the higher the corresponding resource-saving score.

[0051] The fifth evaluation dimension can be understood as an evaluation direction used to measure the model training status corresponding to the optional parallel strategy. The fifth evaluation dimension can be used to predict the convergence speed and operational stability of the model training corresponding to the optional parallel strategy. For example, the fifth evaluation dimension can be a convergence prediction score. The determination of the convergence prediction score can include: retrieving the running data of similar training tasks from the strategy memory, and using a lightweight prediction model to calculate the predicted scores corresponding to the convergence speed and operational stability of the current optional parallel strategy in model training.

[0052] In one scenario, the provided scheme, in which the evaluation metrics of the optional parallel strategy are determined across multiple evaluation dimensions, may include at least one of the following: providing optional parallel strategies to multiple first machine learning models to obtain multiple evaluation metrics, wherein the multiple first machine learning models correspond to different evaluation dimensions; and, for the multiple evaluation dimensions, using an algorithm corresponding to the evaluation dimension to process the optional parallel strategy to obtain the evaluation metrics of the optional parallel strategy across the evaluation dimensions.

[0053] In one scenario, the provided scheme obtains a policy evaluation index corresponding to the optional parallel policy based on multiple evaluation metrics, which may include at least one of the following: processing multiple evaluation metrics through a weighted summation operation to obtain a policy evaluation index; or providing multiple evaluation metrics to a second machine learning model to obtain the output policy evaluation index.

[0054] In one scenario, for multiple evaluation dimensions, optional parallel strategies are provided to the first machine learning model corresponding to each evaluation dimension to obtain the evaluation metrics of the optional parallel strategies on the evaluation dimensions. Furthermore, multiple evaluation metrics are processed through a weighted summation operation to obtain the strategy evaluation metric corresponding to the optional parallel strategies.

[0055] In one scenario, the provided solution, based on at least one of the policy evaluation metrics, determines the target parallel policy from at least one of the optional parallel policies, which may include at least one of the following: determining the maximum value among at least one of the policy evaluation metrics, and determining the optional parallel policy corresponding to the maximum value as the target parallel policy; for at least one policy evaluation metric, determining the optional parallel policy corresponding to the policy evaluation metric that is greater than the evaluation threshold as the target parallel policy.

[0056] In one scenario, the maximum value among at least one policy evaluation metric is determined, and the optional parallel policy corresponding to the maximum value is identified as the target parallel policy. Further, the task splitting, data distribution, and communication configuration of the target parallel policy are loaded, and the target cluster environment is initialized. The model to be trained and the training data are split and distributed to each node of the target cluster according to the target parallel policy. Each node sequentially performs forward computation, loss calculation, and backward gradient update, while simultaneously following communication rules to complete cross-node data interaction. The training process is iterated repeatedly, and the running status is monitored in real time until the model reaches the preset termination condition. Finally, the parameters are integrated to obtain the trained model.

[0057] The technical solution of this disclosure collects complete training configuration information, enabling a comprehensive understanding of the basic conditions of the model and cluster. This provides an accurate basis for subsequent parallel strategy generation and selection rules, avoiding mismatches between strategies and the actual environment due to missing information. Furthermore, obtaining a parallel training method suitable for the cluster based on the training configuration information ensures that all available parallel strategies have the foundation for implementation. Moreover, clearly defining parallel optimization criteria provides a unified standard for subsequent selection work, ensuring consistency in the selection direction. Furthermore, combining training configuration information and parallel optimization criteria for preliminary selection can quickly eliminate obviously unsuitable parallel strategies, narrowing the scope of subsequent evaluation, reducing overall processing workload, and improving process efficiency. Furthermore, conducting in-depth analysis of available strategies through dedicated evaluation indicators allows for the selection of the strategy with the best overall performance from the alternatives, making the model training process more aligned with preset requirements and ensuring stable and efficient training. The technical solution provided in this disclosure addresses the limitations of existing model training strategies, which are difficult to adapt to heterogeneous clusters or complex model structures, resulting in poor strategy adaptability, unbalanced resource load, and poor model training performance. It effectively adapts to heterogeneous clusters and complex model structures, improves the overall adaptability of parallel training strategies, balances cluster resource load, and thus improves model training quality. Furthermore, by using hierarchical screening and indicator evaluation to select the best strategy at each level, it simplifies the strategy selection process and selects the target parallel strategy that fits the optimization objective, thereby improving the rationality and operational efficiency of parallel model training.

[0058] Figure 2 This is a flowchart illustrating another method for determining a model training strategy provided in this embodiment. The technical solution of this embodiment can be combined with other embodiments; for the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 2 As shown, the method in this embodiment may specifically include: S210. Receive the model training task corresponding to the model to be trained, and obtain the basic configuration information corresponding to the model to be trained based on the model training task. The basic configuration information includes the cluster layout information of the target cluster, the model structure information of the model to be trained, and the training constraints.

[0059] In one scenario, a model training task can be understood as a complete business instruction and task carrier for initiating model training. The model training task can be the initial trigger source of the entire model training process, used to clarify the training object, training objective, execution scope, and delivery requirements. The training object can be the specific model to be trained, i.e., the model to be trained. The training objective can be used to define the desired training effect and acceptance criteria. The execution scope can limit the hardware, data, and environment used for training. Delivery requirements can be used to specify the output format and delivery rules after the task is completed. In one scenario, the provided solution may include basic configuration information corresponding to the model to be trained within the model training task.

[0060] In one scenario, basic configuration information can be understood as the set of core raw parameters extracted from the model training task, serving as the underlying basis for setting up the training environment and defining operating rules. Optionally, basic configuration information may include the cluster layout information of the target cluster, the model structure information of the model to be trained, and training constraints.

[0061] Cluster layout information can be understood as fundamental data describing the overall architecture, node distribution, and organizational form of the target cluster. Cluster layout information can be used to record the number of nodes, their physical locations, and the network architecture, serving as a prerequisite for traversing nodes, planning communication, and allocating tasks. Optionally, cluster layout information may include at least one of the following: node quantity information, physical deployment location information, node grouping information, hierarchical architecture information, and region division information. Node quantity information can be used to record the total number of cluster nodes participating in training within the target cluster. Physical deployment location information can be used to indicate the physical location of each node in the target cluster. Node grouping information can be used to group nodes according to rules such as function, location, and load. Hierarchical architecture information can be used to describe the overall organizational form of the target cluster, such as a master-slave architecture or a hierarchical architecture, for example, a two-level distributed architecture using master nodes and worker nodes. Region division information can be used for cross-data center or cross-region clusters, recording the region to which the nodes belong.

[0062] The model structure information can be understood as the architecture and parameter information of the model to be trained. Model structure information may include inherent model attributes such as network layers, operator types, parameter sizes, and input / output specifications. Training constraints can be understood as the restrictive rules that must be followed during the training process. Training constraints can be hard requirements to ensure stable and compliant training operation; if the training constraints are not met, the model training task cannot be executed normally. Optionally, training constraints may include at least one of the following: single-node memory limit, maximum number of iterations, training duration limit, and accuracy threshold.

[0063] In one scenario, the system receives a user's configuration operation for the training task of the model to be trained, and generates a model training task corresponding to the model to be trained based on the configuration operation. Further, in response to receiving the model training task corresponding to the model to be trained, the system parses the model training task to obtain basic configuration information.

[0064] S220. Based on the cluster layout information, traverse the target nodes within the target cluster to obtain the node configuration parameters of the target nodes, and obtain the cluster communication parameters corresponding to the target cluster.

[0065] In one scenario, a target node can be understood as a single computing node within the target cluster that independently undertakes the training task. A target node can be a basic unit within the target cluster possessing computational, storage, and network communication capabilities, and can be used to execute local model training tasks. Optionally, the node type of the target node includes at least one of the following: hardware device, graphics processing unit (GPU) node, neural network processor node, or general-purpose computing node.

[0066] In one scenario, node configuration parameters can be understood as the hardware and operational parameters of a single target node. These parameters can be data obtained after traversing the nodes, such as node computing power, GPU memory, RAM, operating environment, and available resources. Node configuration parameters serve as the core basis for determining the load capacity of a target node and allocating computing tasks, and are specific to a single target node. Optionally, node configuration parameters may include at least one of the following: hardware resource parameters, such as node computing power, GPU memory, RAM, and storage hardware specifications; operating environment parameters, such as system version, drivers, frameworks, and dependent libraries software environment information; resource quota parameters, such as the maximum amount of resources a node can occupy and constraints on running permissions; and device identification parameters, such as node number, hardware identifier, and device type.

[0067] In one scenario, cluster communication parameters can be understood as a set of parameters describing the data transmission rules, network architecture, and transmission performance between cluster nodes, ensuring normal data interaction between multiple nodes. Cluster communication parameters can be derived from cluster layout information and are used to define node network relationships, transmission capabilities, and communication rules. They are crucial for achieving parameter, gradient, and intermediate data synchronization in distributed training. Cluster communication parameters can include node network topology and communication parameters. Node network topology can be understood as the network connection architecture of each target node within the target cluster. Node network topology can describe the topology of physical connections, networking methods, and node affiliation between nodes. Optionally, node network topology can include at least one of star topology, ring topology, multi-rack hierarchical networking, etc. Communication parameters can be understood as specific performance parameters related to network transmission. Communication parameters can include link bandwidth, transmission latency, communication protocols, data fragmentation rules, etc.

[0068] In one scenario, based on the acquired cluster layout information, the distribution of all target nodes within the target cluster is determined. Each target node is then sequentially traversed and detected, and its hardware resource parameters, operating environment parameters, resource limit parameters, and device identification parameters are read and summarized to obtain the node configuration parameters corresponding to each target node. Furthermore, based on the cluster layout information, relevant data such as inter-node connection relationships, network bandwidth, transmission latency, and communication protocols are collected to obtain the cluster communication parameters corresponding to the target cluster.

[0069] S230. Obtain cluster configuration information based on node configuration parameters and cluster communication parameters, and obtain training configuration information corresponding to the model to be trained based on model structure information, training constraints, and cluster configuration information.

[0070] In one scenario, the node configuration parameters and cluster communication parameters corresponding to each target node are integrated, and the resulting information is determined as the cluster configuration information. Furthermore, model structure information, training constraints, and cluster configuration information can be integrated to obtain training configuration information corresponding to the model to be trained.

[0071] S240. Based on the training configuration information, obtain a variety of available parallel strategies and parallel optimization criteria corresponding to the model to be trained.

[0072] S250. Based on training configuration information and parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies.

[0073] S260. Determine at least one policy evaluation metric for optional parallel strategies, and based on at least one policy evaluation metric, determine a target parallel strategy from at least one optional parallel strategy, so as to train the model to be trained based on the target parallel strategy and the target cluster.

[0074] The technical solution of this disclosure constructs a complete training configuration by comprehensively collecting multi-dimensional information such as node configuration, network communication, model structure and training constraints, providing accurate and comprehensive data support for subsequent parallel strategy formulation, and ensuring that the training scheme is highly matched with the cluster and model.

[0075] Figure 2 This is a flowchart illustrating another method for determining a model training strategy provided in this embodiment. The technical solution of this embodiment can be combined with other embodiments; for the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 2 As shown, the method in this embodiment may specifically include: S310. Obtain the training configuration information corresponding to the model to be trained. The training configuration information is used to describe the model training configuration corresponding to the model to be trained. The training configuration information includes the cluster configuration information of the target cluster.

[0076] S320. Based on the training configuration information, obtain a variety of available parallel strategies and parallel optimization criteria corresponding to the model to be trained.

[0077] S330. Based on training configuration information and various available parallel strategies, determine multiple complexity vectors corresponding to the model to be trained.

[0078] In one scenario, a complexity vector can be understood as a feature vector composed of at least one complexity metric, used to fully characterize the overall training difficulty of a single available parallel strategy. The complexity vector can consist of three dimensions: parallel strategy complexity, model structure complexity, and heterogeneous topological complexity; one available parallel strategy can correspond to one complexity vector.

[0079] Optionally, the training configuration information also includes model structure information; based on the training configuration information and multiple available parallel strategies, determining multiple training complexities corresponding to the model to be trained includes: based on the training configuration information and multiple available parallel strategies, determining multiple complexity vectors corresponding to the model to be trained includes: for multiple available parallel strategies, obtaining model structure complexity based on the model structure information, obtaining parallel strategy complexity based on the available parallel strategies, and obtaining heterogeneous topology complexity based on the cluster configuration information; obtaining the complexity vector corresponding to the model to be trained based on the model structure complexity, the parallel strategy complexity, and the heterogeneous topology complexity.

[0080] In one context, model structural complexity can be understood as a quantified metric based on model structural information, used to characterize the computational, memory, and communication pressure imposed by the model itself. Model structural complexity can be calculated based on model structural information. Within the same model being processed and the same cluster scenario, model structural complexity is a fixed value. For example, hybrid expert models have more cross-node data interactions and higher type weights, resulting in greater model structural complexity; shallow image classification models have fewer layers and smaller tensor data volumes, leading to lower model structural complexity.

[0081] In one context, the complexity of a parallel strategy can be understood as a metric quantified from the parameter characteristics of the optional parallel strategies, representing the tuning scale and configuration difficulty of the optional parallel strategies. The core of parallel strategy complexity can be considered as the dimension of parallel variables, i.e., the total number of adjustable parameters in a set of optional parallel strategies. Different optional parallel strategies correspond to different levels of parallel strategy complexity. Generally, the fewer adjustable parameters in an optional parallel strategy, the lower its parallel strategy complexity; conversely, the more adjustable parameters in an optional parallel strategy, the higher its parallel strategy complexity.

[0082] In one context, heterogeneous topology complexity can be understood as a comprehensive indicator calculated based on cluster configuration information, representing the degree of imbalance in cluster hardware, computing power, and network. Heterogeneous topology complexity can be calculated by weighting the number of node types, the variance of computing power distribution, and the variance of link bandwidth distribution. For example, a standard cluster with all target nodes being the same type of GPU and having consistent bandwidth has a lower heterogeneous topology complexity; a cluster using multiple types of target nodes and exhibiting significant differences in bandwidth across racks has a higher heterogeneous topology complexity.

[0083] In one scenario, the model structure information of the model to be trained is read, and the total number of layers, model type, and the specifications of activation tensors and gradient tensors for each layer are extracted. Combined with the weights corresponding to the model type, a fixed model structure complexity is calculated. Further, for multiple available parallel strategies, the number of all adjustable parameters within each available parallel strategy is counted, obtaining a parallel strategy complexity corresponding to each available parallel strategy. Additionally, cluster configuration information is read, and the number of device types, computing power distribution, and link bandwidth data are counted. A fixed heterogeneous topology complexity is obtained through normalized variance weighted calculation. Further, for multiple available parallel strategies, the parallel strategy complexity of each available parallel strategy, the globally unified model structure complexity, and the heterogeneous topology complexity are determined as the complexity vector corresponding to that available parallel strategy. Thus, multiple complexity vectors corresponding to the model to be trained can be obtained, and these complexity vectors can be composed of parallel strategy complexity, model structure complexity, and heterogeneous topology complexity.

[0084] S340. Based on multiple complexity vectors and parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies.

[0085] In one scenario, the provided solution determines at least one optional parallel strategy from multiple available parallel strategies based on multiple complexity vectors and parallel optimization criteria. This may include at least one of the following: globally ranking the multiple available parallel strategies based on the multiple complexity vectors and the parallel optimization criteria; selecting parallel strategies within a preset ranking range from the ranking results as at least one optional parallel strategy; setting a preset complexity threshold and eliminating available parallel strategies whose complexity vectors exceed the threshold; for the remaining parallel strategies, using a unified screening method and filtering according to the parallel optimization criteria to obtain at least one optional parallel strategy; dividing all available parallel strategies into two groups according to their complexity vectors, with each group using a different screening method; combining the parallel optimization criteria to complete the screening of each group, and merging the results of the two groups to obtain the final optional parallel strategy. One of the determination methods will be specifically described below.

[0086] Optionally, based on multiple complexity vectors and parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies, including: dividing the various available parallel strategies into a first strategy group and a second strategy group based on multiple complexity vectors, wherein the training complexity of the available parallel strategies included in the first strategy group is higher than that of the available parallel strategies included in the second strategy group; using a first strategy selection method corresponding to the first strategy group, and selecting the first strategy group according to the parallel optimization criteria to obtain a first selection result; and using a second strategy selection method corresponding to the second strategy group, and selecting the second strategy group according to the parallel optimization criteria to obtain a second selection result; and obtaining at least one optional parallel strategy based on the first selection result and the second selection result.

[0087] In one scenario, the first strategy group can be understood as a group of available parallel strategies with an overall high complexity vector. The available parallel strategies within the first strategy group have higher training difficulty, resource overhead, and search optimization difficulty, requiring adaptation to high-load processing logic. The second strategy group can be understood as a group of available parallel strategies with an overall low complexity vector. The available parallel strategies within the second strategy group have lower training pressure, simpler parameters, and lower optimization difficulty, focusing on efficient and fast processing. For example, the complexity vectors corresponding to the four available parallel strategies are (2,50,22), (4,50,22), (6,50,22), and (8,50,22), respectively. The dimensions of each element in these complexity vectors are, in order: parallel strategy complexity, model structure complexity, and heterogeneous topology complexity. The available parallel strategies with complexity vectors (6,50,22) and (8,50,22) are divided and assigned to the first strategy group; the available parallel strategies with complexity vectors (2,50,22) and (4,50,22) are divided and assigned to the second strategy group.

[0088] In one scenario, the first strategy selection method can be understood as the strategy selection and optimization logic adapted to high-complexity strategy groups. The first strategy selection method is suitable for scenarios with high training load, large parameter space, heterogeneous clusters, and high communication pressure. For example, the first strategy selection method could involve first constructing a node topology graph and using a minimum cut algorithm to eliminate available parallel strategies corresponding to high-latency links, then initializing a differential evolution population for optimization. If the iteration stalls, the population size is increased or a random restart is initiated. This entire process constitutes the first strategy selection method.

[0089] In one scenario, the second strategy selection method can be understood as a lightweight and fast selection scheme designed for low-complexity parallel strategies. This method is suitable for scenarios with small parameter spaces and low training loads. It relies on historical sampling data to rank strategies, performing heuristic mutation fine-tuning only on the top-ranked available parallel strategies to reduce redundant computation and efficiently select the best option. If the iteration converges too quickly, the mutation amplitude can be appropriately increased to broaden the selection range.

[0090] In one scenario, the first screening result can be understood as the set of qualified strategies obtained after screening the first strategy group, wherein the set of qualified strategies includes at least one available parallel strategy. The second screening result can be understood as the set of qualified strategies obtained after screening the second strategy group, wherein the set of qualified strategies includes at least one available parallel strategy.

[0091] In one scenario, for multiple available parallel strategies, the complexity vector corresponding to each strategy is compared with a preset complexity threshold. If the complexity of the parallel strategy in the complexity vector is greater than the preset threshold, or if the sum of the model structure complexity and the heterogeneous topology complexity is greater than the preset threshold, the available parallel strategy can be assigned to a first strategy group. If the complexity of the parallel strategy in the complexity vector is not greater than the preset threshold, and the sum of the model structure complexity and the heterogeneous topology complexity is not greater than the preset threshold, the available parallel strategy can be assigned to a second strategy group. Further, multiple available parallel strategies can be divided into a first strategy group and a second strategy group. Further, a first strategy selection method corresponding to the first strategy group is used to select the first strategy group according to parallel optimization criteria to obtain a first selection result; and a second strategy selection method corresponding to the second strategy group is used to select the second strategy group according to parallel optimization criteria to obtain a second selection result. Further, the first and second selection results can be merged to obtain at least one selectable parallel strategy. S350. Determine at least one policy evaluation metric for optional parallel strategies, and based on at least one policy evaluation metric, determine a target parallel strategy from at least one optional parallel strategy, so as to train the model to be trained based on the target parallel strategy and the target cluster.

[0092] The technical solution of this disclosure quantifies the training difficulty of different parallel strategies through complexity vector quantization, and combines optimization criteria with hierarchical screening strategies to accurately select parallel solutions that are suitable for the current scenario, thereby improving strategy matching and overall training performance.

[0093] Figure 4 This is a schematic diagram of a model training strategy determination device provided in an embodiment of this disclosure. Figure 4 As shown, the model training strategy determination device includes: a first module 410, a second module 420, a third module 430, and a fourth module 440. The first module 410 is used to acquire training configuration information corresponding to the model to be trained, the training configuration information describing the model training configuration corresponding to the model to be trained; the training configuration information includes at least cluster configuration information of the target cluster. The second module 420 is used to obtain, based on the training configuration information, multiple available parallel strategies and parallel optimization criteria corresponding to the model to be trained, the available parallel strategies indicating the parallel training method and its related adjustable parameters for training the model to be trained using the target cluster. The third module 430 is used to determine at least one optional parallel strategy from the multiple available parallel strategies based on the training configuration information and the parallel optimization criteria. The fourth module 440 is used to determine the strategy evaluation index of at least one optional parallel strategy, and based on at least one strategy evaluation index, determine a target parallel strategy from the at least one optional parallel strategy, so as to train the model to be trained based on the target parallel strategy and the target cluster.

[0094] The technical solution of this disclosure collects complete training configuration information, enabling a comprehensive understanding of the basic conditions of the model and cluster. This provides an accurate basis for subsequent parallel strategy generation and selection rules, avoiding mismatches between strategies and the actual environment due to missing information. Furthermore, obtaining a parallel training method suitable for the cluster based on the training configuration information ensures that all available parallel strategies have the foundation for implementation. Moreover, clearly defining parallel optimization criteria provides a unified standard for subsequent selection work, ensuring consistency in the selection direction. Furthermore, combining training configuration information and parallel optimization criteria for preliminary selection can quickly eliminate obviously unsuitable parallel strategies, narrowing the scope of subsequent evaluation, reducing overall processing workload, and improving process efficiency. Furthermore, conducting in-depth analysis of available strategies through dedicated evaluation indicators allows for the selection of the strategy with the best overall performance from the alternatives, making the model training process more aligned with preset requirements and ensuring stable and efficient training. The technical solution provided in this disclosure addresses the limitations of existing model training strategies, which are difficult to adapt to heterogeneous clusters or complex model structures, resulting in poor strategy adaptability, unbalanced resource load, and poor model training performance. It effectively adapts to heterogeneous clusters and complex model structures, improves the overall adaptability of parallel training strategies, balances cluster resource load, and thus improves model training quality. Furthermore, by using hierarchical screening and indicator evaluation to select the best strategy at each level, it simplifies the strategy selection process and selects the target parallel strategy that fits the optimization objective, thereby improving the rationality and operational efficiency of parallel model training.

[0095] In some embodiments of this disclosure, optionally, the target cluster includes multiple different target nodes for training the model to be trained; the first module 410 includes a first unit, a second unit, and a third unit. The first unit is configured to receive a model training task corresponding to the model to be trained, and obtain basic configuration information corresponding to the model to be trained based on the model training task. The basic configuration information includes cluster layout information of the target cluster, model structure information of the model to be trained, and training constraints. The second unit is configured to traverse the target nodes within the target cluster based on the cluster layout information to obtain node configuration parameters of the target nodes, and obtain cluster communication parameters corresponding to the target cluster. The cluster communication parameters include node network topology and communication parameters. The third unit is configured to obtain the cluster configuration information based on the node configuration parameters and the cluster communication parameters, and obtain training configuration information corresponding to the model to be trained based on the model structure information, the training constraints, and the cluster configuration information.

[0096] In some embodiments of this disclosure, optionally, the third module 430 includes a fourth unit and a fifth unit. The fourth unit is used to determine multiple parallel training methods corresponding to the model to be trained based on the training configuration information; the fifth unit is used to determine adjustable parameters related to each of the parallel training methods, and obtain available parallel strategies based on the parallel training method and the adjustable parameters.

[0097] In some embodiments of this disclosure, optionally, the fourth module 440 includes a sixth unit and a seventh unit. The sixth unit is used to determine multiple complexity vectors corresponding to the model to be trained based on the training configuration information and multiple available parallel strategies; the seventh unit is used to determine at least one optional parallel strategy from the multiple available parallel strategies based on the multiple complexity vectors and the parallel optimization criteria.

[0098] In some embodiments of this disclosure, optionally, the training configuration information further includes model structure information; the sixth unit is specifically used to obtain model structure complexity based on the model structure information, parallel strategy complexity based on the available parallel strategies, and heterogeneous topology complexity based on the cluster configuration information for various available parallel strategies; and to obtain a complexity vector corresponding to the model to be trained based on the model structure complexity, the parallel strategy complexity, and the heterogeneous topology complexity.

[0099] In some embodiments of this disclosure, optionally, the seventh unit is specifically configured to divide a variety of available parallel strategies into a first strategy group and a second strategy group based on multiple complexity vectors, wherein the complexity vector of the available parallel strategies included in the first strategy group is higher than that of the available parallel strategies included in the second strategy group; to filter the first strategy group using a first strategy filtering method corresponding to the first strategy group according to the parallel optimization criterion to obtain a first filtering result; and to filter the second strategy group using a second strategy filtering method corresponding to the second strategy group according to the parallel optimization criterion to obtain a second filtering result; and to obtain at least one optional parallel strategy based on the first filtering result and the second filtering result.

[0100] In some embodiments of this disclosure, optionally, the fourth module 440 further includes an eighth unit and a ninth unit. The eighth unit is used to determine evaluation metrics for at least one of the optional parallel strategies across multiple evaluation dimensions; the ninth unit is used to obtain a strategy evaluation metric corresponding to the optional parallel strategy based on the multiple evaluation metrics.

[0101] The model training strategy determination apparatus provided in this disclosure can execute the model training strategy determination method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0102] It is worth noting that the various units and modules included in the above-mentioned model training strategy determination device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0103] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0104] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0105] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0106] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the model training strategy determination method.

[0107] In some embodiments, the model training strategy determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via read-only memory (ROM) 12 and / or communication unit 19. When the computer program is loaded into random access memory (RAM) 13 and executed by processor 11, one or more steps of the model training strategy determination method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the model training strategy determination method by any other suitable means (e.g., by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital circuit systems, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, at least one input device, and at least one output device.

[0109] Computer programs used to implement the model training strategy determination method of this disclosure can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] This disclosure provides a computer-readable storage medium storing computer instructions for causing a processor to execute a model training strategy determination method, comprising: acquiring training configuration information corresponding to a model to be trained, the training configuration information describing the model training configuration corresponding to the model to be trained; the training configuration information including at least cluster configuration information of a target cluster; based on the training configuration information, obtaining multiple available parallel strategies and parallel optimization criteria corresponding to the model to be trained, the available parallel strategies indicating a parallel training method and its related adjustable parameters for training the model to be trained using the target cluster; based on the training configuration information and the parallel optimization criteria, determining at least one optional parallel strategy from the multiple available parallel strategies; determining a strategy evaluation index for the at least one optional parallel strategy, and based on the at least one strategy evaluation index, determining a target parallel strategy from the at least one optional parallel strategy, so as to train the model to be trained based on the target parallel strategy and the target cluster.

[0111] In the context of this disclosure, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, also known as flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode-ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0114] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0115] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of embodiments of this disclosure.

[0116] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements a model training strategy determination method according to any embodiment of this disclosure.

[0117] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining a model training strategy, characterized in that, include: Obtain training configuration information corresponding to the model to be trained, wherein the training configuration information is used to describe the model training configuration corresponding to the model to be trained; The training configuration information includes at least the cluster configuration information of the target cluster; Based on the training configuration information, a variety of available parallel strategies and parallel optimization criteria corresponding to the model to be trained are obtained. The available parallel strategies are used to indicate the parallel training methods that can be used to train the model to be trained based on the target cluster and their related adjustable parameters. Based on the training configuration information and the parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies; Determine a policy evaluation metric for at least one of the optional parallel strategies, and based on at least one of the policy evaluation metric, determine a target parallel strategy from at least one of the optional parallel strategies, so as to train the model to be trained based on the target parallel strategy and the target cluster.

2. The method for determining the model training strategy according to claim 1, characterized in that, The target cluster includes multiple different target nodes used to train the model to be trained; The step of obtaining the training configuration information corresponding to the model to be trained includes: Receive a model training task corresponding to the model to be trained, and obtain basic configuration information corresponding to the model to be trained based on the model training task. The basic configuration information includes cluster layout information of the target cluster, model structure information of the model to be trained, and training constraints. Based on the cluster layout information, the target nodes within the target cluster are traversed to obtain the node configuration parameters of the target nodes, and to obtain the cluster communication parameters corresponding to the target cluster, wherein the cluster communication parameters include node network topology and communication parameters. The cluster configuration information is obtained based on the node configuration parameters and the cluster communication parameters, and the training configuration information corresponding to the model to be trained is obtained based on the model structure information, the training constraints, and the cluster configuration information.

3. The method for determining the model training strategy according to claim 1, characterized in that, Based on the training configuration information, multiple available parallel strategies corresponding to the model to be trained are obtained, including: Based on the training configuration information, multiple parallel training methods corresponding to the model to be trained are determined. For each of the parallel training methods, adjustable parameters related to the parallel training method are determined, and available parallel strategies are obtained based on the parallel training method and the adjustable parameters.

4. The method for determining the model training strategy according to claim 1, characterized in that, Based on the training configuration information and the parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies, including: Based on the training configuration information and various available parallel strategies, multiple complexity vectors corresponding to the model to be trained are determined. Based on the multiple complexity vectors and the parallel optimization criteria, at least one optional parallel strategy is determined from a variety of available parallel strategies.

5. The method for determining the model training strategy according to claim 4, characterized in that, The training configuration information also includes model structure information; the step of determining multiple complexity vectors corresponding to the model to be trained based on the training configuration information and various available parallel strategies includes: For the various available parallel strategies, the model structure complexity is obtained based on the model structure information, the parallel strategy complexity is obtained based on the available parallel strategies, and the heterogeneous topology complexity is obtained based on the cluster configuration information. The complexity vector corresponding to the model to be trained is obtained based on the model structure complexity, the parallel strategy complexity, and the heterogeneous topology complexity.

6. The method for determining the model training strategy according to claim 4, characterized in that, The determination of at least one optional parallel strategy from a variety of available parallel strategies based on multiple complexity vectors and the parallel optimization criteria includes: Based on multiple complexity vectors, the various available parallel strategies are divided into a first strategy group and a second strategy group, wherein the complexity vector of the available parallel strategies included in the first strategy group is higher than that of the available parallel strategies included in the second strategy group. The first strategy group is filtered using a first strategy filtering method corresponding to the first strategy group, and the first strategy group is filtered according to the parallel optimization criterion to obtain a first filtering result; and the second strategy group is filtered using a second strategy filtering method corresponding to the second strategy group, and the second strategy group is filtered according to the parallel optimization criterion to obtain a second filtering result. Based on the first screening result and the second screening result, at least one optional parallel strategy is obtained.

7. The method for determining the model training strategy according to claim 1, characterized in that, The determination of at least one of the optional parallel strategies for strategy evaluation metrics includes: For at least one of the optional parallel strategies, determine the evaluation metrics of the optional parallel strategy on multiple evaluation dimensions; Based on multiple evaluation metrics, a strategy evaluation metric corresponding to the optional parallel strategy is obtained.

8. A model training strategy determination device, characterized in that, include: The first module is used to obtain training configuration information corresponding to the model to be trained, wherein the training configuration information is used to describe the model training configuration corresponding to the model to be trained. The training configuration information includes the cluster configuration information of the target cluster; The second module is used to obtain, based on the training configuration information, multiple available parallel strategies and parallel optimization criteria corresponding to the model to be trained. The available parallel strategies are used to indicate the parallel training methods and their related adjustable parameters that can be used to train the model to be trained based on the target cluster. The third module is used to determine at least one optional parallel strategy from a variety of available parallel strategies based on the training configuration information and the parallel optimization criteria. The fourth module is used to determine a policy evaluation index for at least one of the optional parallel strategies, and based on the at least one policy evaluation index, to determine a target parallel strategy from the at least one of the optional parallel strategies, so as to train the model to be trained based on the target parallel strategy.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the model training strategy determination method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the model training strategy determination method according to any one of claims 1-7.

Images