A model training method, device, equipment, medium, product

Abstract

The present application provides a model training method, device and equipment, medium and product; the method includes: obtaining a network model to be trained, the network model includes M sub-network models; performing strategy decision processing on each sub-network model respectively to obtain the parallel strategy of each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and the execution mode of the parallel training tasks of any sub-network model in the model training process; based on the parallel strategies of the M sub-network models, a training system for the network model is constructed, the training system includes M communication groups, one communication group corresponds to one sub-network model; in the process of model training of the network model, the M communication groups in the training system are called to execute the parallel training tasks of the corresponding sub-network model respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network model to obtain the trained network model. The present application can speed up the model training process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology, and specifically to a model training method, a model training device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] An integrated network model refers to a network model generated by combining multiple different single network models. A single network model refers to an independent neural network model, which is usually composed of one or more neural network layers and is used to perform specific tasks or learn specific data features. A parallel strategy is usually designed for the integrated network model. The parallel strategy is used to indicate the parallel training tasks of the integrated network model and the execution method of the parallel training tasks. Then, the parallel training tasks of the integrated network model are executed according to the execution method indicated by the parallel strategy to accelerate the model training process of the integrated network model. However, the training of integrated network models is still limited by computational efficiency and scalability. How to improve the computational efficiency and scalability of training integrated network models and further accelerate the model training process of integrated network models has become a key issue for technical personnel. Summary of the invention

[0003] The embodiments of the present application provide a model training method, device and equipment, medium, and product, which can improve the computing efficiency and scalability when training an integrated network model and accelerate the model training process of the integrated network model.

[0004] On the one hand, an embodiment of the present application provides a model training method, the method comprising:

[0005] Obtain a network model to be trained, where the network model includes M sub-network models, where M is an integer greater than 1;

[0006] Performing strategy decision processing on each sub-network model respectively to obtain the parallel strategy of each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and the execution method of the parallel training tasks of any sub-network model in the model training process;

[0007] Based on the parallel strategy of M sub-network models, a training system of the network model is constructed, and the training system includes M communication groups; one communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model;

[0008] In the process of model training for the network model, M communication groups in the training system are called to execute the parallel training tasks of the corresponding sub-network models respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network model to obtain the trained network model.

[0009] On the other hand, an embodiment of the present application provides a model training device, the device comprising:

[0010] An acquisition unit, used to acquire a network model to be trained, where the network model includes M sub-network models, where M is an integer greater than 1;

[0011] A processing unit, used to perform strategy decision processing on each sub-network model respectively to obtain a parallel strategy for each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and execution methods of the parallel training tasks of any sub-network model in the model training process;

[0012] The processing unit is further used to construct a training system for the network model based on the parallel strategy of the M sub-network models, the training system including M communication groups; one communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model;

[0013] The processing unit is also used to call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models in accordance with the execution mode indicated by the parallel strategy of the corresponding sub-network models during the process of model training of the network model, so as to obtain the trained network model.

[0014] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model. Any sub-network model is represented as sub-network model M i , i is an integer less than or equal to M;

[0015] The processing unit is used to process the sub-network model M i Perform strategy decision processing to obtain the sub-network model M i When the parallel strategy is used, the following steps are performed:

[0016] Call the acquisition unit to obtain the sub-network model M i Strategy search space, sub-network model M i The strategy search space includes the sub-network model M i Multiple pre-set reference parallel strategies;

[0017] Based on the sub-network model M i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i Training time under each reference parallel strategy;

[0018] The sub-network model M i The reference parallel strategy corresponding to the minimum training time in the strategy search space is the sub-network model Mi Parallel strategies;

[0019] Among them, the parallel parameters include any one or more of the following: data parallel parameters, model parallel parameters; data parallel parameters include one or both of micro-batch size and data parallelism; model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

[0020] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include a data parallel parameter and a model parallel parameter, and the data parallel parameter includes a micro-batch size;

[0021] The processing unit is configured to determine the subnetwork model M based on the reference parallel parameter indicated by the first reference parallel strategy. i During the time-consuming training process under the first reference parallel strategy, the following steps are specifically performed:

[0022] In the first training device, the sub-network model M i Initialized as sub-network model M i Model instance, calling sub-network model M i The model instance processes a training sample and obtains the sub-network model M i Unit training on the first training device takes time;

[0023] The acquisition unit is called to obtain the micro-batch size in the reference parallel parameter indicated by the first reference parallel strategy, which is consistent with the sub-network model M i a first product of unit training times on a first training device;

[0024] The first ratio between the first product and the model parallel parameter indicated by the first reference parallel strategy is determined as the sub-network model M i Training time under the first reference parallel strategy.

[0025] In one embodiment, the processing unit is configured to: i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i Before the training time under each reference parallel strategy, it is also used to perform the following steps:

[0026] Call the acquisition unit to obtain the unit computing amount of M sub-network models on the same training device;

[0027] Determine the target sub-network model with the largest unit computation amount from the M sub-network models;

[0028] If the target sub-network model is not sub-network model M i , then update the sub-network model M according to the parallel strategy of the target sub-network modeli Each reference parallel strategy in the strategy search space.

[0029] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include data parallel parameters, the data parallel parameters include a micro-batch size and a data parallelism;

[0030] The processing unit is used to update the sub-network model M according to the parallel strategy of the target sub-network model i When the first reference parallel strategy in the strategy search space is used, the following steps are specifically performed:

[0031] Calling an acquisition unit to acquire a second product between a micro-batch size and a data parallelism indicated by a parallel strategy of a target sub-network model;

[0032] Using a second ratio between the second product and the data parallelism indicated by the first reference parallel strategy, the micro-batch size indicated by the first reference parallel strategy is updated; or,

[0033] The data parallelism indicated by the first reference parallel strategy is updated using a third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy.

[0034] In one embodiment, the processing unit is used to construct a training system for a network model based on a parallel strategy of M sub-network models, and specifically performs the following steps:

[0035] According to the parallel strategy of each sub-network model, a communication group corresponding to each sub-network model is constructed;

[0036] A component communication topology structure among M communication groups is established to generate a training system for the network model.

[0037] In one embodiment, any sub-network model is represented as sub-network model M i The network model also includes the sub-network model M i Sub-network model M with cascade relationship j , sub-network model M i Corresponding to communication group i, subnetwork model M j corresponding to communication group j; i and j are both integers less than or equal to M, and i is not equal to j;

[0038] When the processing unit is used to establish a component communication topology structure between M communication groups, the following steps are specifically performed:

[0039] Perform group division processing on communication group i and communication group j to obtain P communication groups, where P is a positive integer; each communication group includes at least one sub-communication group obtained by dividing communication group i and at least one sub-communication group obtained by dividing communication group j;

[0040] In the process of establishing the component communication topology structure between communication group i and communication group j, an inter-machine communication bandwidth is allocated to each communication group, and communication channels between sub-communication groups belonging to different communication groups are established in each communication group.

[0041] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model, the parallel parameters include data parallel parameters, and the data parallel parameters include data parallelism;

[0042] The processing unit is used to perform group division processing on the communication group i and the communication group j, and when P communication groups are obtained, the following steps are specifically performed:

[0043] Call the acquisition unit to obtain the sub-network model M i The parallel strategy indicates the data parallelism and sub-network model M j The greatest common divisor between the data parallelisms indicated by the parallel strategies;

[0044] Determine the greatest common divisor as the number of groups P;

[0045] Based on the number of groups P, communication group i and communication group j are divided into groups to obtain P communication groups.

[0046] In one embodiment, each sub-communication group includes one or more training devices;

[0047] The processing unit is used to establish a communication channel between sub-communication groups belonging to different communication groups in each communication group, specifically performing the following steps:

[0048] According to the sub-network model M i With sub-network model M j The cascade relationship between them determines the communication order of components between communication group i and communication group j;

[0049] In each sub-communication group obtained by dividing the communication group with the component communication sequence first, selecting a training device to be determined as a communication transmitting device;

[0050] In each sub-communication group obtained by dividing the communication group whose component communication order is later, selecting a training device to be determined as a communication receiving device;

[0051] A communication channel is established between a communication transmitting device and a communication receiving device.

[0052] In one embodiment, any sub-network model is represented as sub-network model M i , sub-network model M iCorresponding to communication group i, i is an integer less than or equal to M; the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model, the parallel parameters include data parallel parameters and model parallel parameters, and the data parallel parameters include data parallelism;

[0053] The processing unit is used to follow the sub-network model M i When building a communication group i, the following steps are performed:

[0054] Based on the sub-network model M i The data parallelism and model parallelism parameters in the parallelism parameters indicated by the parallel strategy determine the number of training devices in the communication group i;

[0055] Get the device types in the training device cluster used for model training, and follow the sub-network model M i Parallel strategy to determine the sub-network model M i The training time on each equipment type is long;

[0056] Based on the sub-network model M i The training time on the training device of each device type is used to determine the device type of the training device in the communication group i;

[0057] According to the sub-network model M i Based on the number and type of training devices in communication group i, the sub-network model M is initialized in the training device cluster. i , to construct communication group i.

[0058] In one embodiment, the processing unit is configured to: i The training time on each type of training device is used to determine the type of the training device in the communication group i. Specifically, the following steps are performed:

[0059] Based on the sub-network model M i The training time on the training device of each device type is used to determine the target device type corresponding to the minimum training time;

[0060] Obtain the number of training devices of the target device type from the training device cluster;

[0061] When the number of training devices of the target device category is greater than or equal to the number of training devices in the communication group i, the target device category is determined as the device category of the training devices in the communication group i.

[0062] In one embodiment, the data parallel parameters further include a mini-batch size;

[0063] The processing unit is used to follow the sub-network model M iParallel strategy to determine the sub-network model M i During the time-consuming training process on the training device of each device type, the following steps are specifically performed:

[0064] Call the acquisition unit to obtain the sub-network model M i The time it takes to train units on training equipment of each equipment category;

[0065] Based on the sub-network model M i The mini-batch size, model parallel parameters, and sub-network model M indicated by the parallel strategy i The unit training time on each type of training device determines the sub-network model M i Training on training equipment of each equipment category takes time.

[0066] In one embodiment, the processing unit is used to call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network model during the model training of the network model, and specifically perform the following steps:

[0067] In the process of training the network model, the model training process of the network model is divided into M stages in the pipeline, and one stage corresponds to the parallel training task of a sub-network model;

[0068] Execute the pipeline, and in the process of executing the pipeline, determine the target communication group used to execute the current stage, call the target communication group to execute the parallel training task of the sub-network model corresponding to the target communication group according to the execution method indicated by the parallel strategy of the sub-network model corresponding to the target communication group, and the current stage is the stage currently being executed among the M stages.

[0069] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising:

[0070] A processor for loading and executing computer programs;

[0071] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned model training method is implemented.

[0072] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned model training method.

[0073] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned model training method is implemented.

[0074] The technical solution provided in the embodiment of the present application can obtain a network model to be trained (an integrated network model), which includes M sub-network models (a single network model), where M is an integer greater than 1. Perform policy decision processing on each sub-network model to obtain a parallel strategy for each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and the execution method of the parallel training tasks of any sub-network model during the model training process. Compared with the prior art solution, the M sub-network models are taken as a whole and trained using a unified parallel strategy. The technical solution provided by the present application can achieve parallel strategy decoupling, which means that the M sub-network models can be configured with different parallel strategies. In the process of policy decision processing, a suitable parallel strategy can be decided for each sub-network model, thereby improving the computational efficiency when training the network model. Based on the parallel strategies of the M sub-network models, a training system for the network model is constructed, and the training system includes M communication groups; one communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model. On the basis of parallel strategy decoupling, each sub-network model can have its own communication group, so that appropriate hardware resources can be accurately matched for each sub-network model, which can solve the problem of idle resources caused by the uneven computing amount of the M sub-network models when training the M sub-network models as a whole, and improve the scalability of training network models. In the process of model training of the network model, calling the M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network model according to the execution mode indicated by the parallel strategy of the corresponding sub-network model can accelerate the model training process of the network model to obtain the trained network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 is a schematic diagram of hardware resource utilization provided by an embodiment of the present application;

[0076] Figure 2 is an example schematic diagram of an assembly line provided in an embodiment of the present application;

[0077] Figure 3 It is a structural diagram of a large-scale speech model training system provided in an embodiment of the present application;

[0078] Figure 4 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0079] Figure 5 It is a flowchart of a model training method provided in an embodiment of the present application;

[0080] Figure 6It is a flowchart of a strategy decision processing method provided in an embodiment of the present application;

[0081] Figure 7 It is a flowchart of a training system construction method provided in an embodiment of the present application;

[0082] Figure 8 It is a partial structural diagram of a training system provided in an embodiment of the present application;

[0083] Fig. 9 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0084] Fig.10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The Multimodal Large Language Model (MLLM) is an integrated network model consisting of a modal encoder, a large language model (LLM), and a modal decoder. The modal encoder is used to convert multimodal information (such as text, images, audio, video, etc.) into text representations that can be understood by the LLM; the LLM is used to learn the language patterns, contextual information, and semantic associations in the text representation, and generate text sequences with language capabilities (referring to text information that can express language information and contain grammatical structure and semantic information); and the modal decoder is used to translate the text sequences generated by the LLM back to the corresponding modal information.

[0086] Large multimodal language models have shown great application potential in many fields. As an important part of the multimodal field, the speech modality has received widespread attention. This application takes a large multimodal language model as an example, which is a large speech model including an audio encoder (for converting audio into a text representation understandable by LLM), LLM and an audio decoder (for translating the text sequence generated by LLM back to audio modality), to illustrate the technical solution (referred to as a model training solution) provided in this application.

[0087] 1. This application involves parallel strategies:

[0088] The so-called parallel strategy refers to a strategy that uses parallel computing to speed up the model training process. Exemplary parallel strategies include but are not limited to: data parallel strategy, model parallel strategy, and hybrid parallel strategy. Among them:

[0089] ① Data parallel strategy: refers to dividing the training data into multiple micro-batches, each micro-batch contains micro-batch size (MBS) training samples, and assigning these micro-batches to different training devices. A complete model instance is deployed on each training device. The model instance on each training device can be trained using the assigned micro-batches, and the model parameters can be updated by summing or averaging the gradients obtained by training the model instances on each training device. This method can copy the entire model to multiple training devices in a distributed manner, so that each training device can process different micro-batches, thereby effectively utilizing the hardware resources of multiple training devices and speeding up the model training process.

[0090] The micro-batch size refers to the number of training samples processed by each training device in each training iteration. For example, MBS=5 means that in each training iteration, each training device can be allocated 5 training samples, and each training device can simultaneously input the 5 allocated training samples into its own model instance for processing. The so-called training iteration refers to a complete parameter update process that a neural network model undergoes during model training; in a training iteration, the predicted results of the neural network model are usually compared with the actual results, the loss function value (such as mean square error) is calculated, the gradient is calculated using the loss function value, and the model parameters are updated using optimization algorithms such as gradient descent.

[0091] A neural network model is a machine learning model inspired by the neural network structure of the human brain. It consists of multiple neurons (nodes) that form multiple neural network layers (input layer, hidden layer, output layer), and the neural network layers are connected to each other through weights. The neural network model learns the relationship between input data and corresponding output data, so that it can make predictions or classifications without clear rules. Both the integrated network model and the sub-network models that make up the integrated network model are neural network models.

[0092] In the data parallel strategy, the complete model instance will be distributedly deployed on multiple training devices. The number of training devices with the complete model instance deployed can be reflected by the data parallelism (DP). For example, DP=3 means that the complete model instance is deployed on 3 training devices. Neural network models are usually defined by classes, which contain the structure and model parameters of the neural network model. A class is a user-defined reference data type used to implement information encapsulation. Each class contains a data description and a set of functions that operate data or pass messages. An instance of a class is an object created according to the class definition. When the class of the neural network model is initialized (or instantiated, which refers to the process of creating an instance according to the class definition), a model instance of the neural network model can be obtained. Obtaining a model instance of the neural network model means that a neural network model that can be used for learning and reasoning has been prepared, and tasks such as training, testing, and prediction can be performed through the model instance of the neural network model. A complete model instance can be understood as an object that encapsulates the entire structure and model parameters of the neural network model.

[0093] The training device refers to a computing device used to train a neural network model, which may include but is not limited to hardware devices such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and TPU (Tensor Processing Unit). When training a neural network model, GPU is generally considered to be a more common training device. This application uses the training device as an example to illustrate.

[0094] ② Model parallel strategy: refers to the strategy of placing different parts of the neural network model on different training devices for parallel calculation when training the neural network model to speed up the model training process. Model parallel strategies include but are not limited to: pipeline parallel strategy and tensor parallel strategy.

[0095] The pipeline parallel strategy refers to splitting the neural network model into multiple network modules, assigning these network modules to different training devices for training, and letting each training device be responsible for processing the calculation of a specific network module, and passing the intermediate results to the next training device for processing, so that each training device can calculate different network modules in parallel, thereby speeding up the training process of the entire neural network model. A network module refers to a component of a neural network model, which can contain a set of related neural network layers. Pipeline parallelism (PP) is used to reflect the degree of splitting of the neural network model in the pipeline parallel strategy. For example, pipeline parallelism refers to the number of network modules obtained after the neural network model is split. For example, PP=3 means that the neural network model is split into 3 network modules; PP=1 means that the neural network module is not split.

[0096] The tensor parallel strategy refers to splitting the model parameters of the neural network model (such as weight matrix, bias term) into multiple tensors and distributing them to different training devices for training. Each training device is responsible for calculating the gradient of the assigned tensor, so that each training device can independently process the calculation of some parameters, and can update the model parameters by aggregating the gradients calculated by each training device. The so-called tensor refers to a mathematical object representing a multidimensional array, which can contain scalars (0-dimensional tensors), vectors (1-dimensional tensors), matrices (2-dimensional tensors) and arrays of higher dimensions. Exemplarily, the embodiment of the present application takes the tensor parallel strategy to split the model parameters of each neural network layer of the neural network model as an example to illustrate. For example, the neural network model includes an input layer, a hidden layer and an output layer. The weight matrix from the input layer to the hidden layer is: W1, the bias item from the input layer to the hidden layer is: b1, the weight matrix from the hidden layer to the output layer is: W2, and the bias item from the hidden layer to the output layer is: b2. The weight matrix W1 can be split into W1_1 and W1_2 and assigned to GPU1 and GPU2 respectively, the bias item b1 can be split into b1_1 and b1_2 and assigned to GPU1 and GPU2 respectively, the weight matrix W2 can be split into W2_1 and W2_2 and assigned to GPU1 and GPU2 respectively, the bias item b2 can be split into b2_1 and b2_2 and assigned to GPU1 and GPU2 respectively, so that GPU1 can be responsible for the calculation of the tensors: W1_1, b1_1, W2_1, b2_1, and GPU2 can be responsible for the calculation of the tensors: W1_2, b1_2, W2_2, b2_2. Tensor Parallelism (TP) can reflect the degree of splitting of model parameters in the tensor parallel strategy. For example, pipeline parallelism refers to the number of tensors obtained after the model parameters are split. For example, the model parameters of each neural network layer of the neural network model are split into 2 tensors, then TP is equal to 2; when all model parameters remain in their original form and are not split into multiple tensors, TP is equal to 1.

[0097] ③ Hybrid parallel strategy: refers to a strategy that uses data parallel strategy and model parallel strategy at the same time to maximize the use of hardware resources and improve training speed and efficiency. For example, the neural network model is split into network module 1 and network module 2, the model parameters of network module 1 are split into tensors 1 and 2, and the model parameters of network module 2 are split into tensors 3 and 4. Tensor 1 of network module 1 can be deployed on GPU1 and GPU2, tensor 2 of network module 1 can be deployed on GPU3 and GPU4, tensor 3 of network module 2 can be deployed on GPU5 and GPU6, and tensor 4 of network module 2 can be deployed on GPU7 and GPU8. In this way, GPU1, GPU3, GPU5 and GPU7 can realize the function of a complete neural network model (i.e., corresponding to a complete model instance), and GPU2, GPU4, GPU6 and GPU8 can realize the function of a complete neural network model (i.e., corresponding to a complete model instance).

[0098] 2. Technical background involved in this application:

[0099] Taking the audio encoder in the large speech model as an example, which is the open source xeus speech encoder (Xeus speech encoder), and the LLM is the open source 7B moe LLM (7B means one billion model parameters, and moe means mixed expert model), the shortcomings of the existing model training scheme are analyzed.

[0100] ① The problem of idle hardware resources caused by unbalanced computing power: In terms of computing power, when DP=1, TP=1, PP=1, and MBS=1 (i.e., a complete model instance is deployed on a training device, and the micro-batch size processed by the training device is 1), the computing power of the xeus speech encoder is 3.75T FLOPs (i.e., 3.75 trillion floating-point operations), and the computing power of the 7B moe LLM is 180.55T FLOPs (i.e., 180.55 trillion floating-point operations), which is 48 times different from the previous one. The computing power is very unbalanced. Among them, FLOPs refers to the number of floating-point operations, which reflects the amount of computing required by the neural network model during training or inference. Under the existing model training scheme, the audio encoder and audio decoder are regarded as additional layers of the LLM, and a unified parallel strategy is used. Therefore, the MBS of the audio encoder and audio decoder will be limited by the LLM. In other words, the audio encoder and audio decoder need the same MBS as the LLM setting, which will cause a lot of computing resources to be idle when training the xeus speech encoder. See Figure 1The schematic diagram of hardware resource utilization is shown in Figure 11. Part 11 shows the hardware resource utilization when training the xeus speech encoder. It can be seen that the hardware resource utilization is low at this time. This is because the MBS is small for the xeus speech encoder, so the parallelism is low when training the xeus speech encoder, which may lead to a decrease in computing efficiency and slow down the model training process. Hardware resources provide the infrastructure required for computing resources to run, while computing resources refer to the actual ability to perform computing tasks.

[0101] ② Limitation of scalability: Considering the audio encoder, audio decoder, and LLM as a whole will cause the audio encoder, audio decoder, and LLM to use the same set of hardware resources for training. Since the audio encoder and audio decoder occupy additional hardware resources such as video memory, LLM may not be able to fully utilize hardware resources to process larger tasks or data, thus limiting the scalability of LLM. Scalability refers to the ability to effectively adapt to increasing demand or scale.

[0102] ③ Pipeline bubbles: Considering the audio encoder and audio decoder as part of the LLM will make it difficult to evenly split the large speech model into pipelines, resulting in pipeline bubbles, which further reduces the computing efficiency. The so-called pipeline bubble refers to the fact that the execution of some instructions needs to wait for the result of the previous instruction, resulting in the inability to continue to execute subsequent instructions, thus forming an idle time on the pipeline. This idle time is called a pipeline bubble. See Figure 2 An example schematic diagram of the pipeline shown shows that due to the large amount of computation of the LLM, pipeline bubbles are generated on the audio decoder.

[0103] 3. Model training scheme of this application:

[0104] This application can construct a large-scale speech model training system, and use the large-scale speech model training system to train the large-scale speech model. Figure 3 The structural diagram of the training system of the large speech model shown in FIG. Figure 3 It can be seen that the model training scheme provided in the present application can realize the independent decoupling of the parallel strategies of the audio encoder, LLM, and audio decoder. That is to say, the audio encoder, LLM, and audio decoder can each configure their own parallel strategies, which can cleverly eliminate the parallel dependence of the audio encoder, LLM, and audio decoder during the model training process; for example, in Figure 3, the parallel strategy of the audio encoder indicates data parallelism = 2, tensor parallelism = 2, pipeline parallelism = 1, micro-batch size = 6, the parallel strategy of the LLM indicates data parallelism = 4, tensor parallelism = 4, pipeline parallelism = 2, micro-batch size = 3, and the parallel strategy of the audio decoder indicates data parallelism = 1, tensor parallelism = 1, pipeline parallelism = 1, micro-batch size = 12. Exemplarily, a plurality of reference parallel strategies can be pre-set for the audio encoder, and the reference parallel strategy that minimizes the training time of the audio encoder is selected as the parallel strategy of the audio encoder, and the same is true for LLM and audio decoder. Computational efficiency generally refers to the ability to complete a computing task within a unit of time, so the shorter the training time, the higher the computational efficiency. Hardware resource utilization refers to the degree to which hardware resources (such as CPU, GPU, TSU, memory, hard disk, network, etc.) are actually used. The shorter the time it takes to train the model, the higher the hardware resource utilization, because the computational efficiency depends largely on the utilization efficiency of hardware resources. Based on the decoupling of the parallel strategy of the audio encoder, LLM, and audio decoder, in the process of building a large-scale speech model training system, the audio encoder, LLM, and audio decoder can each build their own communication group, which includes one or more training devices. In this way, during the model training process, the audio encoder, LLM, and audio decoder will use independent hardware resources, which is conducive to improving scalability. For example, Figure 3 In the figure, the DP indicated by the parallel strategy of LLM determines that the communication group of LLM contains 4 layers, and the product between TP and PP indicated by the parallel strategy of LLM determines that each layer contains 8 GPU3, which is equivalent to that each layer can realize the function of a complete LLM, and a total of 4 complete LLMs are included. The same is true for the audio encoder and audio decoder.

[0105] In addition, the model training solution is highly flexible and supports different types of hardware resources (i.e., training devices) for audio encoders, LLMs, and audio decoders. This allows the audio encoder, LLM, and audio decoder to be accurately matched with appropriate hardware resources, further optimizing hardware resource utilization. For example, audio encoders usually require lower computing power to achieve audio data compression and encoding, so choosing GPU1 with lower computing power can meet its needs. The audio decoder requires higher computing power for decoding and data reconstruction, so choosing GPU2 with higher computing power can provide better decoding performance. LLM needs to complete data processing within a shorter delay, so choosing GPU3 with the largest computing power provides faster data processing speed.

[0106] The model training solution provided in this application can use more flexible and fine-grained parallel strategy configuration and more efficient hardware resource allocation to achieve more efficient large-scale speech model training and accelerate the model training process of large-scale speech models.

[0107] The following briefly describes the application scenarios to which the model processing solution provided in this application is applicable. It should be noted that the following scenarios are only used to illustrate the embodiments of this application and are not limiting. In specific implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs. Figure 4 , Figure 4 4 is a schematic diagram of an application scenario provided by an embodiment of the present application. The application scenario includes a terminal device 41 and a server 42, and the terminal device 41 and the server 42 can communicate with each other through a communication network.

[0108] The communication network may be a wired network or a wireless network. Therefore, the terminal device 41 and the server 42 may be directly or indirectly connected via a wired or wireless communication. For example, the terminal device 41 may be indirectly connected to the server 42 via a wireless access point, or the terminal device 41 may be directly connected to the server 42 via the Internet, and this application does not limit this.

[0109] The terminal device 41 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, smart home appliances, vehicle-mounted terminals, etc. The server 42 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0110] Exemplarily, the server 42 may include: training data for training a network model to be trained (such as a large speech model), and hardware resources (such as a GPU) for training the network model to be trained. In this way, the server 42 can perform the following model training process according to the model training scheme provided in this application: respectively decide on appropriate parallel strategies for the M sub-network models in the network model to be trained, and build a training system for the network model to be trained based on the parallel strategies of each sub-network model. The training system includes M communication groups, one communication group corresponds to one sub-network model. In the process of model training of the network model, the training data is used and the M communication groups are called to respectively execute the parallel training tasks of the corresponding sub-network models according to the execution mode indicated by the parallel strategy of the corresponding sub-network model to obtain the trained network model.

[0111] In a feasible embodiment, a training framework may be deployed on the server 42. The training framework refers to a software tool or platform for training machine learning models (such as neural network models). The training framework provides a series of tools and interfaces to facilitate developers to design, train, optimize and deploy various machine learning models. The model training process of the model training scheme provided in the embodiment of the present application can be encapsulated into a target function module in the training framework, and the network model to be trained and the training data for training the network model to be trained can be uploaded to the training framework. The training framework calls the target function module to execute the model training process of the model training scheme provided in the present application to obtain the trained network model.

[0112] Optionally, after training the network model to be trained based on the model training scheme in the embodiment of the present application, the trained network model can be directly deployed on the server 42 or the terminal device 41 to provide model services. In one embodiment, the trained network model can be directly deployed on the server 42 to provide model services to the terminal device 41.

[0113] Figure 4 The figure is only an example. In fact, the number of terminal devices 41 and servers 42 is not limited. For example, there may be multiple servers 42, each of which includes GPUs of different device types.

[0114] The collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information requires the knowledge or consent of the individual subject (or the legal basis for obtaining the information), and the subsequent use and processing of data shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as obtaining training data and network models to be trained, it is necessary to obtain the permission or consent of the creators of the training data and the network models to be trained, and the collection, use and processing of relevant data shall comply with the relevant laws, regulations and standards of the relevant regions.

[0115] The following describes specific embodiments of the model training scheme provided in this application.

[0116] See also Figure 5 , Figure 5 is a flow chart of a model training method provided in an embodiment of the present application. The model training method can be executed by a model training device, such as Figure 4 The model training method mainly includes but is not limited to the following steps S101 to S104:

[0117] S101. Obtain a network model to be trained, where the network model includes M sub-network models, where M is an integer greater than 1.

[0118] The network model to be trained is an integrated network model composed of M sub-network models, and the sub-network model can be a single network model. Exemplarily, the network model to be trained can be a multimodal large language model, and the modal encoder, LLM, and modal decoder in the multimodal large language model are three sub-network models included in the multimodal large language model. The present application does not limit the structure and function of the network model to be trained. For example, the network model to be trained can be any network model such as a multimodal large language model, a diffusion model (an image generation model), etc. Exemplarily, the network model to be trained can be determined according to the actual model training requirements. For example, when the server 42 receives a model training instruction, the network model included in the model training instruction can be determined as the network model to be trained.

[0119] S102, perform strategy decision processing on each sub-network model respectively to obtain a parallel strategy for each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and the execution method of the parallel training tasks of any sub-network model during the model training process.

[0120] In this embodiment of the present application, any sub-network model is represented as a sub-network model M i , i is an integer less than or equal to M. i For example, for the sub-network model M i The parallel strategy of sub-network model M is described. i The parallel strategy is: in the model training process (i.e., the process of training the network model to be trained), the sub-network model M i The computing work is distributed to multiple computing resources for parallel processing. i The parallel strategy of M can be a data parallel strategy, a model parallel strategy or a hybrid parallel strategy. The model parallel strategy can include one or both of the pipeline parallel strategy and the tensor parallel strategy. i The parallel strategy can be used to indicate the sub-network model M i Parallel training tasks in the model training process, execution methods of the parallel training tasks, and sub-network models M i The parallel parameters include any one or more of the following: data parallel parameters and model parallel parameters; data parallel parameters include one or both of mini-batch size and data parallelism; model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

[0121] In one implementation, when the sub-network model M i When the parallel strategy includes data parallel strategy, the subnetwork model M iThe parallel training tasks indicated by the parallel strategy include data parallel tasks. Data parallel tasks refer to dividing the training data into multiple micro-batches and running the computing tasks for these micro-batches on multiple training devices at the same time. Subnetwork model M i The data parallel strategy can indicate the execution mode and data parallel parameters of data parallel tasks. For example, when the subnetwork model M i The data parallel strategy is: i When deployed on three training devices, each of which receives five training samples in each training iteration, the subnetwork model M i The execution method of the data parallel task indicated by the data parallel strategy is: i Deployed on 3 training devices, in each round of training iteration, 5 training samples are allocated to each of the 3 training devices; sub-network model M i The data parallel strategy indicates a mini-batch size of 5 and a data parallelism of 3.

[0122] In one implementation, when the sub-network model M i When the parallel strategy includes pipeline parallel strategy, the sub-network model M i The parallel training tasks indicated by the parallel strategy include pipeline parallel tasks. Pipeline parallel tasks refer to i Split into multiple network modules, assign these network modules to multiple training devices, and connect these training devices together according to the execution order of these multiple network modules to form a pipeline structure. In this pipeline structure, each training device performs the computing task of a specific network module. i The pipeline parallel strategy can indicate the execution mode and pipeline parallel degree of pipeline parallel tasks. For example, when the sub-network model M i The pipeline parallel strategy is: sub-network model M i The neural network layer 1 and neural network layer 2 in are deployed to the training device 1, and the sub-network model M i When the neural network layer 3 in is deployed to the training device 2, the sub-network model M i The execution method of the pipeline parallel task indicated by the pipeline parallel strategy is: input the training sample to the training device 1, the training device 1 collects the intermediate results generated by the neural network layer 1 and the neural network layer 2, and sends the intermediate results to the neural network layer 3 in the training device 2 for processing; the sub-network model M i The pipeline parallelism strategy indicates that the pipeline parallelism is 2.

[0123] In one implementation, when the sub-network model M i When the parallel strategy includes the tensor parallel strategy, the subnetwork model Mi The parallel training tasks indicated by the parallel strategy include tensor parallel tasks. Tensor parallel tasks can refer to the sub-network model M i The model parameters of each neural network layer are split into multiple tensors, which are distributed to multiple training devices, which are responsible for performing the calculation tasks of specific tensors. i The tensor parallel strategy can indicate the execution mode and tensor parallel degree of tensor parallel tasks. For example, when the subnetwork model M i The tensor parallel strategy is: sub-network model M i The model parameters of the neural network layer 1 in are split into tensors 1 and 2, and the sub-network model M i The model parameters of neural network layer 2 in are split into tensors 3 and 4. When tensors 1 and 3 are deployed to training device 1, and tensors 2 and 4 are deployed to training device 2, the sub-network model M i The execution method of the tensor parallel task indicated by the tensor parallel strategy is: let training device 1 be responsible for executing the calculation tasks of tensor 1 and tensor 3, and let training device 2 be responsible for executing the calculation tasks of tensor 2 and tensor 4; sub-network model M i The tensor parallelism strategy indicated by is 2.

[0124] In a feasible embodiment, when the sub-network model M i When the parallel strategy does not include the data parallel strategy, set the subnetwork model M i The parallel strategy indicates that the micro-batch size is 1 and the data parallelism is 1; when the sub-network model M i When the parallel strategy does not include the pipeline parallel strategy, set the subnetwork model M i The parallel strategy indicates that the pipeline parallelism is 1; when the sub-network model M i When the parallel strategy does not include the tensor parallel strategy, set the subnetwork model M i The tensor parallelism indicated by the parallel strategy is 1. In this way, the parallel parameters may include data parallel parameters and model parallel parameters; the data parallel parameters may include micro-batch size and data parallelism; the model parallel parameters may include pipeline parallelism and tensor parallelism.

[0125] Next, we take the sub-network model M i As an example, the process of performing policy decision processing on each sub-network model is described. Policy decision processing refers to the process of deciding a suitable parallel strategy for each sub-network model. For example, minimizing the training time of each sub-network model can be used as the execution rule of policy decision processing. i The training time refers to the execution of the sub-network model M in one training iteration. i See Figure 6 , Figure 6 is a flow chart of a policy decision processing method provided in an embodiment of the present application, wherein the policy decision processing method is used to execute a sub-network model M i Perform strategy decision processing to obtain the sub-network model M i The parallel strategy process includes the following steps S201-S203:

[0126] S201, obtain sub-network model M i Strategy search space, sub-network model M i The strategy search space includes the sub-network model M i Pre-set multiple reference parallel strategies.

[0127] Subnetwork model M i The strategy search space includes the sub-network model M i Multiple preset reference parallel strategies can be obtained from the sub-network model M i The strategy search space is the sub-network model M i Decide on a suitable parallel strategy (i.e., subnetwork model M i For example, the reference parallel strategy can be an expert combining model domain knowledge and practical experience, and targeting the sub-network model M i Preset parallel strategies.

[0128] When pre-setting the reference parallel strategy for each sub-network model, it is necessary to consider the problem of data mismatch during the model training process. For example, assuming that in one training iteration, the audio encoder processes 5 training samples per batch (that is, the number of samples processed by the audio encoder per batch is 5), and the LLM processes 4 training samples per batch (that is, the number of samples processed by the LLM per batch is 4), then after the audio encoder processes 5 training samples, it can only pass 4 of them to the LLM, and the remaining 1 training sample cannot continue to flow, resulting in data mismatch. Data mismatch may require additional processing or adjustment to resolve, resulting in a decrease in the model training speed. Even if the data mismatch is not handled correctly, it may cause some data to be lost or processed incorrectly, affecting the training accuracy. The present application can first decide on a suitable parallel strategy (i.e., the parallel strategy of the target sub-network model) for the target sub-network model with the largest unit computational amount (such as the LLM in a large speech model). Since the product of the micro-batch size and the data parallelism indicated by the parallel strategy of the target sub-network model is equal to the number of samples processed per batch by the target sub-network model, the reference parallel strategy of other sub-network models can be limited by making the number of samples processed per batch by other sub-network models consistent with the number of samples processed per batch by the target sub-network model. Other sub-network models refer to network models other than the target sub-network model in the network models to be trained. The unit computational amount of a sub-network model refers to the time required for the sub-network model to process a training sample (i.e., a complete forward calculation) without using parallel computing.

[0129] In one implementation, the unit computation amount of M sub-network models on the same training device can be obtained. For example, the M sub-network models can be deployed on M second training devices respectively, one second training device corresponds to one sub-network model, and each sub-network model can be deployed on the corresponding second training device. By obtaining the computation amount of the sub-network model on each second training device to process a training sample, the unit computation amount of the M sub-network models on the same training device can be obtained. Determine the target sub-network model with the largest unit computation amount from the M sub-network models. The expert can pre-set multiple reference parallel strategies for the target sub-network model, and decide the parallel strategy of the target sub-network model from the multiple reference parallel strategies pre-set for the target sub-network model. The expert can pre-set reference parallel strategies for other sub-network models based on the product between the micro-batch size indicated by the parallel strategy of the target sub-network model and the data parallelism. For example, the product between the micro-batch size indicated by the parallel strategy of the target sub-network model and the data parallelism is 6. When pre-setting reference parallel strategies for other sub-network models, it is necessary to ensure that the product between the micro-batch size indicated by the reference parallel strategy of other sub-network models and the data parallelism is also 6.

[0130] S202, based on sub-network model M i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i Training time under each reference parallel strategy.

[0131] The reference parallel parameter refers to the parallel parameter indicated by the reference parallel strategy. If the problem of data mismatch is not considered and the reference parallel strategy is preset for each sub-network model separately, then i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i Before the training time under each reference parallel strategy, the following steps need to be performed: If the target sub-network model is not the sub-network model M i , then update the sub-network model M according to the parallel strategy of the target sub-network model i Each reference parallel strategy in the strategy search space. Sub-network model M i The multiple reference parallel strategies in the strategy search space include a first reference parallel strategy, and updating the first reference parallel strategy according to the parallel strategy of the target sub-network model is used as an example for explanation. In specific implementation, it is necessary to obtain the second product between the micro-batch size indicated by the parallel strategy of the target sub-network model and the data parallelism, and use the second ratio between the second product and the data parallelism indicated by the first reference parallel strategy to update the micro-batch size indicated by the first reference parallel strategy, that is, determine the second ratio as the micro-batch size indicated by the first reference parallel strategy. For example, assuming that the target sub-network model is LLM, the micro-batch size indicated by the parallel strategy of LLM is expressed as MBS L , data parallelism is represented by DP L , the micro-batch size indicated by the reference parallel strategy of the audio encoder is denoted as MBS e , data parallelism is represented by DP e , the micro-batch size indicated by the reference parallel strategy of the audio decoder is denoted as MBS d , data parallelism is represented by DP d ; According to the formula: MBS e =(MBS L ×DP L )÷DP e Update the micro-batch size indicated by the reference parallel strategy in the strategy search space of the audio encoder, which can be calculated according to the formula: MBS d =(MBS L ×DP L )÷DP dUpdate the micro-batch size indicated by the reference parallel strategy in the strategy search space of the audio decoder. Alternatively, use the third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy to update the data parallelism indicated by the first reference parallel strategy, that is, determine the third ratio as the data parallelism indicated by the first reference parallel strategy. For example, assuming that the target subnetwork model is LLM, the micro-batch size indicated by the parallel strategy of LLM is represented as MBS L , data parallelism is represented by DP L , the micro-batch size indicated by the reference parallel strategy of the audio encoder is denoted as MBS e , data parallelism is represented by DP e , the micro-batch size indicated by the reference parallel strategy of the audio decoder is denoted as MBS d , data parallelism is represented by DP d ; According to the formula: DP e =(MBS L ×DP L )÷MBS e Update the data parallelism indicated by the reference parallel strategy in the strategy search space of the audio encoder, according to the formula: DP d =(MBS L ×DP L )÷MBS d Updates the mini-batch size indicated by a reference parallel strategy in the strategy search space of the audio decoder.

[0132] In one implementation, the parallel parameters include data parallel parameters and model parallel parameters, the data parallel parameters include micro-batch size, and the model parallel parameters include pipeline parallelism and tensor parallelism. Based on the reference parallel parameters indicated by the first reference parallel strategy, determine the subnetwork model M i The training time under the first reference parallel strategy includes: in the first training device, the sub-network model M i Initialized as sub-network model M i Model instance (equivalent to deploying the sub-network model M on the first training device i ), calling the sub-network model M i The model instance processes a training sample (equivalent to calling the sub-network model M i Process a training sample), and get the sub-network model M i The unit training time on the first training device. The first training device can be a training device of any type. The unit training time of the sub-network model refers to the time required for the sub-network model to process a training sample (i.e., a complete forward calculation) without parallel computing. The micro-batch size in the reference parallel parameter indicated by the first reference parallel strategy can be obtained, which is consistent with the sub-network model Mi The first product of the unit training time on the first training device; the first product is equivalent to the sub-network model M when only the data parallel strategy is considered without considering the model parallel strategy. i The first ratio between the first product and the model parallel parameter indicated by the first reference parallel strategy is determined as the sub-network model M i The training time under the first reference parallel strategy is that both the data parallel strategy and the model parallel strategy are considered. i The training is time consuming.

[0133] For example, assume that the micro-batch size indicated by the first reference parallel strategy of LLM is denoted as MBS L , data parallelism is represented by DP L , tensor parallelism is represented by TP L , the pipeline parallelism is expressed as PP L , then the training time of LLM under the first reference parallel strategy is T LLM = , Indicates the unit training time of LLM on the first training device, so that the training time of LLM under each reference parallel strategy preset for LLM can be obtained. By selecting the reference parallel strategy with the smallest training time, a suitable parallel strategy can be determined for LLM. Similarly, assuming that the micro-batch size indicated by the first reference parallel strategy of the audio encoder is denoted as MBS e , data parallelism is represented by DP e , tensor parallelism is represented by TP e , the pipeline parallelism is expressed as PP e , then the training time of the audio encoder under the first reference parallel strategy is T e = , MBS represents the unit training time of the audio encoder on the first training device. It can obtain the training time of the audio encoder under each reference parallel strategy preset for the audio encoder. By selecting the reference parallel strategy with the smallest training time, a suitable parallel strategy can be determined for the audio encoder. Assume that the micro-batch size indicated by the first reference parallel strategy of the audio decoder is denoted as MBS d , data parallelism is represented by DP d , tensor parallelism is represented by TP d , the pipeline parallelism is expressed as PP d , then the training time of the audio decoder under the first reference parallel strategy is T d = , It represents the unit training time of the audio decoder on the first training device, and can obtain the training time of the audio decoder under each reference parallel strategy pre-set for the audio decoder. By selecting the reference parallel strategy with the smallest training time, a suitable parallel strategy can be decided for the audio decoder.

[0134] S203, sub-network model M i The reference parallel strategy corresponding to the minimum training time in the strategy search space is the sub-network model M i That is to say, when the sub-network model M is obtained i In the sub-network model M i After the training time under the preset multiple reference parallel strategies is calculated, the reference parallel strategy corresponding to the minimum training time can be determined as the sub-network model M i parallel strategy.

[0135] It can be seen that the embodiments of the present application, on the one hand, will take into account the problem of data mismatch to ensure that the number of samples processed in each batch of the target sub-network model is consistent with the number of samples processed in each batch of other sub-network models. On the other hand, for a certain sub-network model, it will consider the time required to execute model training using different reference parallel strategies under the same hardware resource conditions, and select the reference parallel strategy with the least time consumption as the parallel strategy of the corresponding sub-network model, which is conducive to improving computing efficiency.

[0136] S103. Based on the parallel strategy of the M sub-network models, a training system for the network model is constructed, where the training system includes M communication groups; one communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model.

[0137] The network model training system is a system for training the network model. The network model training system includes M communication groups, and one communication group corresponds to one sub-network model. Figure 3 The audio encoder, LLM and audio decoder in the communication group each have their own corresponding communication group. A communication group refers to a collection of one or more training devices. By implementing data exchange and communication mechanisms within the communication group, different training devices can transmit data and model parameters to each other to collaboratively execute the parallel training tasks of the corresponding sub-network models and accelerate the calculation process of the corresponding sub-network models.

[0138] See also Figure 7 , Figure 7 1 is a flow chart of a training system construction method provided in an embodiment of the present application. The construction method is used to execute a parallel strategy based on M sub-network models to construct a training system for a network model. The construction method includes the following steps S301-S302:

[0139] S301. According to the parallel strategy of each sub-network model in the M sub-network models, a communication group corresponding to each sub-network model is constructed.

[0140] In one implementation, the sub-network model M i Corresponding to communication group i, the parallel parameters include data parallel parameters and model parallel parameters. The data parallel parameters include data parallelism, and the model parallel parameters include tensor parallelism and pipeline parallelism. i The parallel strategy for building a communication group i includes the following steps 11 to 13:

[0141] Step 11: Based on the sub-network model M i The data parallelism and model parallelism parameters in the parallelism parameters indicated by the parallel strategy determine the number of training devices in the communication group i. i The product of the data parallelism indicated by the parallel strategy of DP and the model parallelism parameter is determined as the number of training devices in communication group i. For example, the data parallelism indicated by the parallel strategy of LLM is expressed as DP L , tensor parallelism is represented by TP L , the pipeline parallelism is expressed as PP L , then the number of training devices in the communication group corresponding to LLM is N LLM =DP L ×TP L ×PP L ; The data parallelism indicated by the parallel strategy of the audio encoder is denoted as DP e , tensor parallelism is represented by TP e , the pipeline parallelism is expressed as PP e , then the number of training devices in the communication group corresponding to the audio encoder is N e =DP e ×TP e ×PP e ; The data parallelism indicated by the parallel strategy of the audio decoder is denoted as DP d , tensor parallelism is represented by TP d , the pipeline parallelism is expressed as PP d , then the number of training devices in the communication group corresponding to the audio decoder is N d =DP d ×TP d ×PP d .

[0142] Step 12: Obtain the device types in the training device cluster used for model training, and i Parallel strategy to determine the sub-network model M iThe training time on each type of training device is based on the sub-network model M i The training time on the training device of each device type is used to determine the device type of the training device in the communication group i.

[0143] For example, assume that the training device cluster is given K (an integer greater than 1) types of training devices (such as K types of GPUs), and the numbers corresponding to the K types of devices are 1, 2, ..., K, and the number of training devices of the K types of devices is N1, N2, ..., N K , then we can minimize the sub-network model M i The principle of training time is used to determine the device type of the training device in communication group i.

[0144] In specific implementation, the device types in the training device cluster can be obtained and the sub-network model M i Parallel strategy to determine the sub-network model M i The training time on each type of training device is as follows. For example, the sub-network model M can be obtained: i The unit training time on the training device of each device type, that is, the sub-network model M i Deploy them on the training devices of each type of equipment respectively, and obtain the sub-network model M on the training devices of each type of equipment i The time required to process a training sample. Based on the sub-network model M i The mini-batch size, model parallel parameters, and sub-network model M indicated by the parallel strategy i The unit training time on each type of training device determines the sub-network model M i The training time on each type of training device is as follows. If any type of device corresponds to a number k, k is an integer less than K, the sub-network model M i Training time on the kth training device = sub-network model M i The parallel strategy indicates the mini-batch size × sub-network model M i Unit training time on the kth training device ÷ (sub-network model M i The parallel strategy indicates the pipeline parallelism × sub-network model M i The parallel strategy of dictates the tensor parallelism.

[0145] Then, based on the sub-network model M i The training time on each type of training device determines the device type of the training device in the communication group i. In specific implementation, it can be based on the sub-network model M iThe training time on the training device of each device type is used to determine the target device type corresponding to the minimum training time. The number of training devices of the target device type is obtained from the training device cluster. When the number of training devices of the target device type is greater than or equal to the number of training devices in the communication group i, the target device type is determined as the device type of the training devices in the communication group i. If the number of training devices of the target device type is less than the number of training devices in the communication group i, the device type with the second smallest training time can be determined as the target device type, and the judgment process is continued. If the number of training devices of the device type with the second smallest training time is less than the number of training devices in the communication group i, the device type with the third smallest training time can be determined as the target device type, and so on.

[0146] For example, the training time of LLM on K types of training devices is expressed as: T 1 LLM , T 2 LLM , ..., T K LLM , the training time of the audio encoder on the training devices of K types of devices is expressed as: T 1 e , T 2 e , ..., T K e , the training time of the audio decoder on the training devices of K types of devices is expressed as: T 1 d , T 2 d , ..., T K d , minimize the objective function: T PPStage =max{T 1 LLM , T 2 LLM , ..., T K LLM ,T 1 e , T 2 e , ..., T K e ,T 1 d , T 2 d , ..., T K d}, we can find T 1 LLM , T 2 LLM , ..., TK LLM The minimum value in T 1 e , T 2 e , ..., T K e The minimum value in T 1 d , T 2 d , ..., T K d The minimum value in , so the device types of training devices included in the communication groups corresponding to LLM, audio encoder, and audio decoder can be determined by minimizing the objective function. In this process, it is necessary to determine whether the quantity constraint conditions are met, such as the number of training devices of a certain device type selected for LLM in the training device cluster must be greater than the number of training devices in the communication group corresponding to LLM; the number of training devices of a certain device type selected for the audio encoder in the training device cluster must be greater than the number of training devices in the communication group corresponding to the audio encoder; the number of training devices of a certain device type selected for the audio decoder in the training device cluster must be greater than the number of training devices in the communication group corresponding to the audio decoder, so as to ensure that there are sufficient hardware resources to establish a communication group.

[0147] Step 13: According to the sub-network model M i Based on the number and type of training devices in communication group i, the sub-network model M is initialized in the training device cluster. i , to build communication group i. Based on the number and type of training devices in communication group i, it is possible to determine which training devices in the training device cluster can be used to build communication group i, and based on the sub-network model M i The parallel strategy can determine the sub-network model M i Initialization method, such as based on the sub-network model M i The data parallel strategy can determine the need for sub-network model M i Initialize and generate several complete model instances based on the sub-network model M i The pipeline parallel strategy can determine the need to convert the sub-network model M i It is divided into several network modules for initialization, etc. In addition, when constructing the communication group i, a communication channel between the training devices in the communication group i is also constructed to realize data interaction between the various training devices in the communication group i.

[0148] It can be seen that the embodiments of the present application can analyze the computing performance of different hardware under the same parallel strategy through the training time of the sub-network model on training devices of different device types, and then allocate more appropriate hardware resources to the sub-network model, which is conducive to optimizing hardware resource utilization and improving scalability.

[0149] S302: Establish a component communication topology structure among M communication groups and generate a training system for a network model.

[0150] The component communication topology refers to an organizational structure in which multiple communication groups are interconnected. Generally, if there is a cascade relationship between two sub-network models, a communication channel will be established between the communication groups corresponding to the two sub-network models. The communication channel can be used to achieve data transmission and communication interaction across different sub-network models. The cascade relationship between two sub-network models means that the output of one sub-network model will become the input of another sub-network model.

[0151] In the following, any sub-network model is represented as sub-network model M i The network model also includes the sub-network model M i Sub-network model M with cascade relationship j , sub-network model M i Corresponding to communication group i, subnetwork model M j The corresponding communication group j is used as an example for explanation, wherein i and j are both integers less than or equal to M, and i is not equal to j. In one embodiment, establishing a component communication topology structure between M communication groups includes the following steps 21-22:

[0152] Step 21: divide the communication group i and the communication group j into groups to obtain P communication groups, where P is a positive integer; each communication group includes at least one sub-communication group obtained by dividing the communication group i and at least one sub-communication group obtained by dividing the communication group j. Each sub-communication group contains one or more training devices, and the number of training devices contained in each sub-communication group is equal to the number of training devices in the sub-network model M. i The parallel strategy indicates the product between pipeline parallelism and tensor parallelism.

[0153] In specific implementation, the sub-network model M can be obtained i The parallel strategy indicates the data parallelism and sub-network model M j The greatest common divisor between the data parallelisms indicated by the parallel strategy. The greatest common divisor is determined as the number of groups P; based on the number of groups P, communication group i and communication group j are grouped to obtain P communication groups. Each communication group includes at least one sub-communication group obtained by dividing communication group i and at least one sub-communication group obtained by dividing communication group j. Among them, each sub-communication group obtained by dividing communication group i can realize the complete sub-network model Mi Each sub-communication group obtained by dividing the communication group j can realize the complete sub-network model M j function.

[0154] For example, see Figure 8 A schematic diagram of a partial structure of a training system is shown in FIG. Figure 8 yes Figure 3 The structural details between LLM and audio encoder in the training system shown. Since the DP of LLM is 4 and the DP of audio encoder is 2, the greatest common divisor of 4 and 2 is 2. Based on the greatest common divisor (i.e. 2), the communication group corresponding to LLM and the communication group corresponding to audio encoder can be divided into two communication groups (i.e. communication group 1 and communication group 2). Each communication group contains a sub-communication group of audio encoder and two sub-communication groups of LLM. As can be seen from the TP=2 of audio encoder, the model parameters of audio encoder are split into two parts (such as tensor 1 and tensor 2). In this way, one GPU1 in communication group 1 can deploy tensor 1 of audio encoder, and one GPU1 can deploy tensor 2 of audio encoder. In this way, two GPU1s in communication group 1 (i.e. sub-communication groups of audio encoder) can realize the function of a complete audio encoder, and the same is true for LLM.

[0155] Step 22: In the process of establishing the component communication topology structure between communication group i and communication group j, an inter-machine communication bandwidth is allocated to each communication group, and communication channels between sub-communication groups belonging to different communication groups are established in each communication group.

[0156] In one embodiment, in the process of establishing the component communication topology between communication group i and communication group j, an inter-machine communication bandwidth can be allocated to each communication group, and the inter-machine communication bandwidth determines the maximum amount of data that can be transmitted within a period of time. In a company or home network, there is usually a total available communication bandwidth resource, and in order to ensure the reasonable allocation and use of network resources, a certain network speed limit can be set for each device. Similarly, in order to ensure the communication efficiency and performance of each communication group, a total communication bandwidth resource can be divided (it can be evenly divided) to obtain the inter-machine communication bandwidth allocated to each communication group, ensuring that each communication group can obtain sufficient communication bandwidth resources for communication and data transmission, thereby effectively utilizing and managing communication bandwidth resources.

[0157] Part of establishing the component communication topology between communication group i and communication group j is to establish a communication channel between communication group i and communication group j, which is a connection path for data transmission and communication interaction between communication group i and communication group j. In the embodiment of the present application, in the process of establishing the component communication topology between communication group i and communication group j, communication channels between sub-communication groups belonging to different communication groups are established in each communication group. In specific implementation, the communication channel can be established according to the sub-network model M. i With sub-network model M j The cascade relationship between them determines the communication order of components between communication group i and communication group j; for example, as shown in the sub-network model M i The output of the sub-network model M j , then the component communication order of communication group i is before communication group j. In each sub-communication group obtained by dividing the communication group whose component communication order is first, a training device can be selected as the communication transmitting device, for example, Figure 8 If the output layer of the audio encoder is deployed on GPU1 in a sub-communication group, the GPU1 can be determined as the communication transmitting device. In each sub-communication group obtained by dividing the communication group with the component communication order later, a training device can be selected to be determined as the communication receiving device; for example Figure 8 If the input layer of LLM is deployed on GPU3 in a sub-communication group, the GPU3 can be determined as a communication receiving device. A communication channel can be established between the communication transmitting device and the communication receiving device, so that in the same communication group, each sub-communication group divided by communication group i can perform data transmission and communication interaction with each sub-communication group divided by communication group j. For example, the sub-communication group of the audio encoder in communication group 1 can process 6 training samples. After processing the 6 training samples, the sub-communication group of the audio encoder can aggregate the calculation results of each training device in the sub-communication group of the audio encoder by the communication transmitting device in the sub-communication group of the audio encoder (such as GPU1 No. 0) to generate two sets of training data, each set of training data contains 3 training samples. The communication transmitting device in the sub-communication group of the audio encoder (such as GPU1 No. 0) can send the two sets of training data to the two sub-communication groups of LLM in communication group 1 respectively through the communication channel with the communication receiving device in each sub-communication group of LLM in communication group 1 (such as GPU3 No. 4 and GPU3 No. 12). In this way, each sub-communication group of LLM can obtain a set of training data to perform computing tasks.

[0158] Figure 8Gather communication is a communication operation in parallel computing. Specifically, Gather communication is a collective operation used to gather data from multiple training devices to a target training device (such as a communication transmitting device) in parallel computing to achieve distributed data collection and integration. Each training device in a sub-communication group can use Gather communication to send the calculation results to the communication transmitting device in the sub-communication group. In one implementation, the communication transmitting device can use send communication in the communication channel to send data to the communication receiving device. Send communication is a communication operation widely used in parallel computing, which is used to send data from one training device to another. Send communication is a point-to-point communication method in which the sender sends data to the receiver and continues to perform other operations after the communication is completed.

[0159] It can be seen that the embodiments of the present application can abstract independent communication groups for each sub-network model, and define communication modes within the communication group and between communication groups, thereby achieving more flexible and fine-grained parallel strategy configuration.

[0160] S104. In the process of training the network model, M communication groups in the training system are called to execute the parallel training tasks of the corresponding sub-network models respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network model to obtain the trained network model.

[0161] In the process of model training for the network model, it is necessary to construct a pipeline for model training for the network model. The pipeline refers to decomposing the model training process of the network model into multiple stages, and connecting these stages together in a certain order to achieve an efficient model training process. In specific implementation, in the process of model training for the network model, the model training process of the network model can be divided into M stages in the pipeline, and one stage corresponds to the parallel training task of a sub-network model. Among them, the M stages in the pipeline are sequentially connected together according to the cascade relationship between the M sub-network models (which may include series connection and parallel connection). For example, in the pipeline, the stage corresponding to the parallel training task of the audio encoder will be connected to the stage corresponding to the parallel training task of the LLM, and the stage corresponding to the parallel training task of the LLM will be connected to the stage corresponding to the parallel training task of the audio decoder. Execute the pipeline. In the process of executing the pipeline, the parallel training tasks corresponding to each stage will be gradually completed according to the sequence of the stages in the pipeline. Therefore, in the process of executing the pipeline, the target communication group for executing the current stage can be determined. The current stage is the stage currently being executed among the M stages. In this way, the target communication group can be called to execute the parallel training task of the sub-network model corresponding to the target communication group according to the execution mode indicated by the parallel strategy of the sub-network model corresponding to the target communication group. For example, the sub-network model M corresponding to the current stage i parallel training tasks, then the target communication group used to execute the current stage is the sub-network model M i The corresponding communication group can call the sub-network model M i The corresponding communication group follows the sub-network model M i The execution mode indicated by the parallel strategy of the execution sub-network model M i Parallel training tasks.

[0162] In the process of model training for the network model to be trained, each time the pipeline is executed, a training iteration will be performed, that is, the predicted result of the network model to be trained will be compared with the actual result, the loss function value will be calculated, the gradient will be calculated using the loss function value, and the model parameters will be updated using optimization algorithms such as gradient descent. The model parameters of the network model can be continuously adjusted by executing the pipeline multiple times to reduce the loss function value generated by the network model, that is, to ensure that the loss function value generated again after each adjustment of the model parameters of the network model is less than the loss function value generated by the network model before adjusting the model parameters. When the network model to be trained reaches the training stop condition after multiple model parameter adjustments, it can be determined that the model training process is over, and the network model trained for the last time can be determined as the trained network model. Among them, the training stop condition includes but is not limited to at least one of the following conditions: the loss function value is less than the set loss threshold, and the number of training iterations reaches the preset number.

[0163] In the embodiment of the present application, the training time of each sub-network model is reduced as the criterion, and a suitable parallel strategy is determined for each sub-network model, and the type of equipment for training equipment in the corresponding communication group is determined for each sub-network model, so that pipeline balancing can be achieved. The so-called pipeline balancing refers to the reasonable adjustment and optimization of the workload and efficiency of each stage in the pipeline, so as to ensure the smooth operation of the entire pipeline and improve the efficiency and quality of model training.

[0164] It can be seen that the embodiment of the present application proposes a parallel strategy decoupling method in the model training scheme, which can create its own parallel strategy for each sub-network model in the network model to be trained, and create an independent communication group for each sub-network model in the network model to be trained, and define the communication mode within the communication group and between communication groups, thereby realizing a more flexible and fine-grained parallel strategy configuration; in addition, a hardware resource allocation method based on performance analysis is proposed, which allocates more appropriate hardware resources and parallel strategies to different sub-network models by analyzing the computing performance of different sub-network models under different hardware and different reference parallel strategies, effectively utilizing the computing power resources of non-high-end training equipment, reducing training costs, improving computing efficiency, and accelerating the model training process.

[0165] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0166] Fig. 9 is a schematic diagram of a model training device provided in an embodiment of the present application; the model training device can be used to perform Figure 5 Some or all of the steps in the method embodiment shown. Fig. 9 The model training device includes the following units: an acquisition unit 901 and a processing unit 902.

[0167] An acquisition unit 901 is used to acquire a network model to be trained, where the network model includes M sub-network models, where M is an integer greater than 1;

[0168] The processing unit 902 is used to perform strategy decision processing on each sub-network model respectively to obtain a parallel strategy for each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and the execution method of the parallel training tasks of any sub-network model in the model training process;

[0169] The processing unit 902 is further used to construct a training system for the network model based on the parallel strategy of the M sub-network models, the training system including M communication groups; one communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model;

[0170] The processing unit 902 is also used to call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models in accordance with the execution method indicated by the parallel strategy of the corresponding sub-network models during the process of model training of the network model, so as to obtain the trained network model.

[0171] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model. Any sub-network model is represented as sub-network model M i , i is an integer less than or equal to M;

[0172] The processing unit 902 is used to process the sub-network model M i Perform strategy decision processing to obtain the sub-network model M i When the parallel strategy is used, the following steps are performed:

[0173] Call acquisition unit 901 to acquire sub-network model M i Strategy search space, sub-network model M i The strategy search space includes the sub-network model M i Multiple pre-set reference parallel strategies;

[0174] Based on the sub-network model M i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i The training time under each reference parallel strategy;

[0175] The sub-network model M i The reference parallel strategy corresponding to the minimum training time in the strategy search space is the sub-network model M i Parallel strategies;

[0176] Among them, the parallel parameters include any one or more of the following: data parallel parameters, model parallel parameters; data parallel parameters include one or both of micro-batch size and data parallelism; model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

[0177] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include a data parallel parameter and a model parallel parameter, and the data parallel parameter includes a micro-batch size;

[0178] The processing unit 902 is configured to determine the subnetwork model M based on the reference parallel parameter indicated by the first reference parallel strategy. i During the time-consuming training process under the first reference parallel strategy, the following steps are specifically performed:

[0179] In the first training device, the sub-network model Mi Initialized as sub-network model M i Model instance, calling sub-network model M i The model instance processes a training sample and obtains the sub-network model M i Unit training on the first training device takes time;

[0180] The calling acquisition unit 901 acquires the micro-batch size in the reference parallel parameter indicated by the first reference parallel strategy, and the micro-batch size in the sub-network model M i a first product of unit training times on a first training device;

[0181] The first ratio between the first product and the model parallel parameter indicated by the first reference parallel strategy is determined as the sub-network model M i Training time under the first reference parallel strategy.

[0182] In one embodiment, the processing unit 902 is configured to: i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i Before the training time under each reference parallel strategy, it is also used to perform the following steps:

[0183] Calling the acquisition unit 901 to acquire the unit computing amount of the M sub-network models on the same training device;

[0184] Determine the target sub-network model with the largest unit computation amount from the M sub-network models;

[0185] If the target sub-network model is not sub-network model M i , then update the sub-network model M according to the parallel strategy of the target sub-network model i Each reference parallel strategy in the strategy search space.

[0186] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include data parallel parameters, the data parallel parameters include a micro-batch size and a data parallelism;

[0187] The processing unit 902 is used to update the sub-network model M according to the parallel strategy of the target sub-network model. i When the first reference parallel strategy in the strategy search space is used, the following steps are specifically performed:

[0188] Calling the acquisition unit 901 to acquire a second product between the micro-batch size indicated by the parallel strategy of the target sub-network model and the data parallelism;

[0189] Using a second ratio between the second product and the data parallelism indicated by the first reference parallel strategy, the micro-batch size indicated by the first reference parallel strategy is updated; or,

[0190] The data parallelism indicated by the first reference parallel strategy is updated using a third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy.

[0191] In one embodiment, the processing unit 902 is used to construct a training system for a network model based on a parallel strategy of M sub-network models, and specifically performs the following steps:

[0192] According to the parallel strategy of each sub-network model, a communication group corresponding to each sub-network model is constructed;

[0193] A component communication topology structure among M communication groups is established to generate a training system for the network model.

[0194] In one embodiment, any sub-network model is represented as sub-network model M i The network model also includes the sub-network model M i Sub-network model M with cascade relationship j , sub-network model M i Corresponding to communication group i, subnetwork model M j corresponding to communication group j; i and j are both integers less than or equal to M, and i is not equal to j;

[0195] When the processing unit 902 is used to establish the component communication topology structure between the M communication groups, it specifically performs the following steps:

[0196] Perform group division processing on communication group i and communication group j to obtain P communication groups, where P is a positive integer; each communication group includes at least one sub-communication group obtained by dividing communication group i and at least one sub-communication group obtained by dividing communication group j;

[0197] In the process of establishing the component communication topology structure between communication group i and communication group j, an inter-machine communication bandwidth is allocated to each communication group, and communication channels between sub-communication groups belonging to different communication groups are established in each communication group.

[0198] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model, the parallel parameters include data parallel parameters, and the data parallel parameters include data parallelism;

[0199] The processing unit 902 is used to perform group division processing on the communication group i and the communication group j, and when P communication groups are obtained, specifically execute the following steps:

[0200] Call acquisition unit 901 to acquire sub-network model Mi The parallel strategy indicates the data parallelism and sub-network model M j The greatest common divisor between the data parallelisms indicated by the parallel strategies;

[0201] Determine the greatest common divisor as the number of groups P;

[0202] Based on the number of groups P, communication group i and communication group j are divided into groups to obtain P communication groups.

[0203] In one embodiment, each sub-communication group includes one or more training devices;

[0204] The processing unit 902 is used to establish a communication channel between sub-communication groups belonging to different communication groups in each communication group, and specifically performs the following steps:

[0205] According to the sub-network model M i With sub-network model M j The cascade relationship between them determines the communication order of components between communication group i and communication group j;

[0206] In each sub-communication group obtained by dividing the communication group with the component communication sequence first, selecting a training device to be determined as a communication transmitting device;

[0207] In each sub-communication group obtained by dividing the communication group whose component communication order is later, selecting a training device as a communication receiving device;

[0208] A communication channel is established between a communication transmitting device and a communication receiving device.

[0209] In one embodiment, any sub-network model is represented as sub-network model M i , sub-network model M i Corresponding to communication group i, i is an integer less than or equal to M; the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model, the parallel parameters include data parallel parameters and model parallel parameters, and the data parallel parameters include data parallelism;

[0210] The processing unit 902 is used to calculate the sub-network model M i When building a communication group i, the following steps are performed:

[0211] Based on the sub-network model M i The data parallelism and the model parallelism parameters in the parallelism parameters indicated by the parallel strategy determine the number of training devices in the communication group i;

[0212] Get the device types in the training device cluster used for model training, and select them according to the sub-network model M i Parallel strategy to determine the sub-network model Mi The training time on each equipment type is long;

[0213] Based on the sub-network model M i The training time on the training device of each device type is used to determine the device type of the training device in the communication group i;

[0214] According to the sub-network model M i Based on the number and type of training devices in communication group i, the sub-network model M is initialized in the training device cluster. i , to construct communication group i.

[0215] In one embodiment, the processing unit 902 is configured to: i The training time on each type of training device is used to determine the type of the training device in the communication group i. Specifically, the following steps are performed:

[0216] Based on the sub-network model M i The training time on the training device of each device type is used to determine the target device type corresponding to the minimum training time;

[0217] Obtain the number of training devices of the target device type from the training device cluster;

[0218] When the number of training devices of the target device category is greater than or equal to the number of training devices in the communication group i, the target device category is determined as the device category of the training devices in the communication group i.

[0219] In one embodiment, the data parallel parameters further include a mini-batch size;

[0220] The processing unit 902 is used to calculate the sub-network model M i Parallel strategy to determine the sub-network model M i During the time-consuming training process on the training device of each device type, the following steps are specifically performed:

[0221] Call acquisition unit 901 to acquire sub-network model M i The time it takes to train units on training equipment of each equipment category;

[0222] Based on the sub-network model M i The mini-batch size, model parallel parameters, and sub-network model M indicated by the parallel strategy i The unit training time on each type of training device determines the sub-network model M i Training on training equipment of each equipment category takes time.

[0223] In one embodiment, the processing unit 902 is used to call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network model during the model training of the network model, and specifically perform the following steps:

[0224] In the process of training the network model, the model training process of the network model is divided into M stages in the pipeline, and one stage corresponds to the parallel training task of a sub-network model;

[0225] Execute the pipeline, and in the process of executing the pipeline, determine the target communication group used to execute the current stage, call the target communication group to execute the parallel training task of the sub-network model corresponding to the target communication group according to the execution method indicated by the parallel strategy of the sub-network model corresponding to the target communication group, and the current stage is the stage currently being executed among the M stages.

[0226] According to one embodiment of the present application, Fig. 9 The various units in the model training device shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 5 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig. 9 The model training device shown in and the model training method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0227] The technical solution provided in the embodiment of the present application can obtain a network model to be trained (an integrated network model), which includes M sub-network models (a single network model), where M is an integer greater than 1. Perform policy decision processing on each sub-network model to obtain a parallel strategy for each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and the execution method of the parallel training tasks of any sub-network model during the model training process. Compared with the prior art solution, the M sub-network models are taken as a whole and trained using a unified parallel strategy. The technical solution provided by the present application can achieve parallel strategy decoupling, which means that the M sub-network models can be configured with different parallel strategies. In the process of policy decision processing, a suitable parallel strategy can be decided for each sub-network model, thereby improving the computational efficiency when training the network model. Based on the parallel strategies of the M sub-network models, a training system for the network model is constructed, and the training system includes M communication groups; one communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model. On the basis of parallel strategy decoupling, each sub-network model can have its own communication group, so that appropriate hardware resources can be accurately matched for each sub-network model, which can solve the problem of idle resources caused by the uneven computing amount of the M sub-network models when training the M sub-network models as a whole, and improve the scalability of training network models. In the process of model training of the network model, calling the M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network model according to the execution mode indicated by the parallel strategy of the corresponding sub-network model can accelerate the model training process of the network model to obtain the trained network model.

[0228] Fig.10 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10 , the computer device includes a processor 1001, a communication interface 1002 and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002 and the computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and send data. The computer-readable storage medium 1003 can be stored in the memory of the computer device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003. The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0229] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more instructions suitable for being loaded and executed by the processor 1001 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0230] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 1001 loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned model training method embodiment; in a specific implementation, the one or more instructions in the computer-readable storage medium are loaded by the processor 1001 and the following steps are executed:

[0231] Obtain a network model to be trained, where the network model includes M sub-network models, where M is an integer greater than 1;

[0232] Performing strategy decision processing on each sub-network model respectively to obtain the parallel strategy of each sub-network model; the parallel strategy of any sub-network model is used to indicate the parallel training tasks and the execution method of the parallel training tasks of any sub-network model in the model training process;

[0233] Based on the parallel strategy of M sub-network models, a training system of the network model is constructed, and the training system includes M communication groups; one communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model;

[0234] In the process of model training for the network model, M communication groups in the training system are called to execute the parallel training tasks of the corresponding sub-network models respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network model to obtain the trained network model.

[0235] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model. Any sub-network model is represented as sub-network model M i , i is an integer less than or equal to M;

[0236] Processor 1001 is used to perform sub-network model M i Perform strategy decision processing to obtain the sub-network model M i When the parallel strategy is used, the following steps are performed:

[0237] Get the subnetwork model M i Strategy search space, sub-network model M i The strategy search space includes the sub-network model M i Multiple pre-set reference parallel strategies;

[0238] Based on the sub-network model M i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i The training time under each reference parallel strategy;

[0239] The sub-network model M i The reference parallel strategy corresponding to the minimum training time in the strategy search space is the sub-network model M i Parallel strategies;

[0240] Among them, the parallel parameters include any one or more of the following: data parallel parameters, model parallel parameters; data parallel parameters include one or both of micro-batch size and data parallelism; model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

[0241] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include a data parallel parameter and a model parallel parameter, and the data parallel parameter includes a micro-batch size;

[0242] The processor 1001 is configured to determine the subnetwork model M based on the reference parallel parameter indicated by the first reference parallel strategy. i During the time-consuming training process under the first reference parallel strategy, the following steps are specifically performed:

[0243] In the first training device, the sub-network model M i Initialized as sub-network model M i Model instance, calling sub-network model M i The model instance processes a training sample and obtains the sub-network model M i Unit training on the first training device takes time;

[0244] Get the mini-batch size in the reference parallel parameter indicated by the first reference parallel strategy, and the sub-network model M i a first product of unit training times on a first training device;

[0245] The first ratio between the first product and the model parallel parameter indicated by the first reference parallel strategy is determined as the sub-network model M i Training time under the first reference parallel strategy.

[0246] In one embodiment, the processor 1001 is configured to: i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space determine the sub-network model M i Before the training time under each reference parallel strategy, it is also used to perform the following steps:

[0247] Get the unit computing amount of M sub-network models on the same training device;

[0248] Determine the target sub-network model with the largest unit computation amount from the M sub-network models;

[0249] If the target sub-network model is not sub-network model M i , then update the sub-network model M according to the parallel strategy of the target sub-network model i Each reference parallel strategy in the strategy search space.

[0250] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include data parallel parameters, the data parallel parameters include a micro-batch size and a data parallelism;

[0251] Processor 1001 is used to update the sub-network model M according to the parallel strategy of the target sub-network model i When the first reference parallel strategy in the strategy search space is used, the following steps are specifically performed:

[0252] Obtain the second product between the mini-batch size and the data parallelism indicated by the parallel strategy of the target sub-network model;

[0253] Using a second ratio between the second product and the data parallelism indicated by the first reference parallel strategy, the micro-batch size indicated by the first reference parallel strategy is updated; or,

[0254] The data parallelism indicated by the first reference parallel strategy is updated using a third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy.

[0255] In one embodiment, the processor 1001 is used to construct a training system for a network model based on a parallel strategy of M sub-network models, and specifically performs the following steps:

[0256] According to the parallel strategy of each sub-network model, a communication group corresponding to each sub-network model is constructed;

[0257] A component communication topology structure among M communication groups is established to generate a training system for the network model.

[0258] In one embodiment, any sub-network model is represented as sub-network model M i The network model also includes the sub-network model M i Sub-network model M with cascade relationship j , sub-network model M i Corresponding to communication group i, subnetwork model M j corresponding to communication group j; i and j are both integers less than or equal to M, and i is not equal to j;

[0259] When the processor 1001 is used to establish a component communication topology structure between the M communication groups, the processor 1001 specifically performs the following steps:

[0260] Perform group division processing on communication group i and communication group j to obtain P communication groups, where P is a positive integer; each communication group includes at least one sub-communication group obtained by dividing communication group i and at least one sub-communication group obtained by dividing communication group j;

[0261] In the process of establishing the component communication topology structure between communication group i and communication group j, an inter-machine communication bandwidth is allocated to each communication group, and communication channels between sub-communication groups belonging to different communication groups are established in each communication group.

[0262] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model, the parallel parameters include data parallel parameters, and the data parallel parameters include data parallelism;

[0263] The processor 1001 is used to perform group division processing on the communication group i and the communication group j, and when P communication groups are obtained, specifically execute the following steps:

[0264] Get the subnetwork model M i The parallel strategy indicates the data parallelism and sub-network model M j The greatest common divisor between the data parallelisms indicated by the parallel strategies;

[0265] Determine the greatest common divisor as the number of groups P;

[0266] Based on the number of groups P, communication group i and communication group j are divided into groups to obtain P communication groups.

[0267] In one embodiment, each sub-communication group includes one or more training devices;

[0268] When the processor 1001 is used to establish a communication channel between sub-communication groups belonging to different communication groups in each communication group, specifically execute the following steps:

[0269] According to the sub-network model M i With sub-network model M j The cascade relationship between them determines the communication order of components between communication group i and communication group j;

[0270] In each sub-communication group obtained by dividing the communication group with the component communication sequence first, selecting a training device to be determined as a communication transmitting device;

[0271] In each sub-communication group obtained by dividing the communication group whose component communication order is later, selecting a training device as a communication receiving device;

[0272] A communication channel is established between a communication transmitting device and a communication receiving device.

[0273] In one embodiment, any sub-network model is represented as sub-network model M i , sub-network model M i Corresponding to communication group i, i is an integer less than or equal to M; the parallel strategy of any sub-network model is also used to indicate the parallel parameters of any sub-network model, the parallel parameters include data parallel parameters and model parallel parameters, and the data parallel parameters include data parallelism;

[0274] The processor 1001 is used to i When building a communication group i, the following steps are performed:

[0275] Based on the sub-network model M i The data parallelism and the model parallelism parameters in the parallelism parameters indicated by the parallel strategy determine the number of training devices in the communication group i;

[0276] Get the device types in the training device cluster used for model training, and select them according to the sub-network model M i Parallel strategy to determine the sub-network model M i The training time on each equipment type is long;

[0277] Based on the sub-network model M i The training time on the training device of each device type is used to determine the device type of the training device in the communication group i;

[0278] According to the sub-network model M i Based on the number and type of training devices in communication group i, the sub-network model M is initialized in the training device cluster. i , to construct communication group i.

[0279] In one embodiment, the processor 1001 is configured to: iThe training time on each type of training device is used to determine the type of the training device in the communication group i. Specifically, the following steps are performed:

[0280] Based on the sub-network model M i The training time on the training device of each device type is used to determine the target device type corresponding to the minimum training time;

[0281] Obtain the number of training devices of the target device type from the training device cluster;

[0282] When the number of training devices of the target device category is greater than or equal to the number of training devices in the communication group i, the target device category is determined as the device category of the training devices in the communication group i.

[0283] In one embodiment, the data parallel parameters further include a mini-batch size;

[0284] The processor 1001 is used to i Parallel strategy to determine the sub-network model M i During the time-consuming training process on the training device of each device type, the following steps are specifically performed:

[0285] Get the subnetwork model M i The time it takes to train units on training equipment of each equipment category;

[0286] Based on the sub-network model M i The mini-batch size, model parallel parameters, and sub-network model M indicated by the parallel strategy i The unit training time on each type of training device determines the sub-network model M i Training on training equipment of each equipment category takes time.

[0287] In one embodiment, the processor 1001 is used to call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models according to the execution mode indicated by the parallel strategy of the corresponding sub-network model during the model training of the network model, and specifically perform the following steps:

[0288] In the process of training the network model, the model training process of the network model is divided into M stages in the pipeline, and one stage corresponds to the parallel training task of a sub-network model;

[0289] Execute the pipeline, and in the process of executing the pipeline, determine the target communication group used to execute the current stage, call the target communication group to execute the parallel training task of the sub-network model corresponding to the target communication group according to the execution method indicated by the parallel strategy of the sub-network model corresponding to the target communication group, and the current stage is the stage currently being executed among the M stages.

[0290] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the computer device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the model training method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0291] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0292] The embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned model training method.

[0293] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0294] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by the computer or a model training device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0295] In this application, elements expressed in the singular are intended to mean "one or more" rather than "one and only one", unless otherwise specified. In this application, "at least one" is intended to mean "one or more", and "more than one" is intended to mean "two or more", unless otherwise specified.

[0296] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: The method comprises: Obtain a network model to be trained, wherein the network model includes M sub-network models, where M is an integer greater than 1; the network model includes a multimodal large language model, and the M sub-network models include a modality encoder, a large language model, and a modality decoder; the modality encoder is used to convert multimodal information input to the multimodal large language model into a text representation, and the multimodal information includes any one of the following: text modality information, image modality information, audio modality information, and video modality information; the large language model is used to learn the language pattern, contextual information, and semantic association in the text representation converted by the modality encoder, and generate a text sequence with language ability; the modality decoder is used to translate the text sequence generated by the large language model back to the corresponding modality information for output; Performing strategy decision processing on each of the sub-network models respectively to obtain a parallel strategy for each of the sub-network models; the parallel strategy of any of the sub-network models is used to indicate the parallel training tasks of any of the sub-network models during the model training process and the execution method of the parallel training tasks; Based on the parallel strategy of the M sub-network models, a training system for the network model is constructed, the training system includes M communication groups; one communication group corresponds to one sub-network model, each communication group is constructed according to the parallel strategy of the corresponding sub-network model, the communication group includes one or more training devices, and the modal encoder, large language model and modal decoder are matched with different types of training devices; In the process of model training of the network model, the M communication groups in the training system are called to execute the parallel training tasks of the corresponding sub-network models respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network model to obtain the trained network model; The parallel strategy of any of the sub-network models is also used to indicate the parallel parameters of any of the sub-network models, and any of the sub-network models is represented as a sub-network model M i , i is an integer less than or equal to M; the sub-network model M i The parallel strategy is to select the sub-network model M i When the plurality of reference parallel strategies include a first reference parallel strategy, the parallel parameters include data parallel parameters, and the data parallel parameters include a micro-batch size and a data parallelism, the method further includes: If the sub-network model M i If the target sub-network model is not a target sub-network model, obtaining a second product between the micro-batch size indicated by the parallel strategy of the target sub-network model and the data parallelism; Using a second ratio between the second product and the data parallelism indicated by the first reference parallel strategy, updating the micro-batch size indicated by the first reference parallel strategy; or, The data parallelism indicated by the first reference parallel strategy is updated by using a third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy.

2. The method according to claim 1, characterized in that For the sub-network model M i Perform strategy decision processing to obtain the sub-network model M i Parallel strategies include: Get the sub-network model M i The strategy search space, the sub-network model M i The strategy search space includes the sub-network model M i Multiple pre-set reference parallel strategies; Based on the sub-network model M i The reference parallel parameters indicated by each reference parallel strategy in the strategy search space are used to determine the sub-network model M i The training time under each of the reference parallel strategies; The sub-network model M i The reference parallel strategy corresponding to the minimum training time in the strategy search space is decided as the sub-network model M i Parallel strategies; Among them, the parallel parameters include any one or more of the following: data parallel parameters, model parallel parameters; the data parallel parameters include one or both of micro-batch size and data parallelism; the model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

3. The method according to claim 2, characterized in that The multiple reference parallel strategies include a first reference parallel strategy; the parallel parameters include a data parallel parameter and a model parallel parameter, and the data parallel parameter includes a micro-batch size; Determine the subnetwork model M based on the reference parallel parameter indicated by the first reference parallel strategy i The training time under the first reference parallel strategy includes: In the first training device, the sub-network model M i Initialize the sub-network model M i Model instance, calling the sub-network model M i The model instance processes a training sample to obtain the sub-network model M i The unit training on the first training device takes time; Obtain the micro-batch size in the reference parallel parameter indicated by the first reference parallel strategy, and i a first product of unit training times on the first training device; The first ratio between the first product and the model parallel parameter indicated by the first reference parallel strategy is determined as the sub-network model M i The training time under the first reference parallel strategy is time-consuming.

4. The method according to claim 2, characterized in that The sub-network model M is determined based on the reference parallel parameters indicated by each reference parallel strategy. i Before the training time under each of the reference parallel strategies is consumed, the method further includes: Obtaining the unit computing amount of the M sub-network models on the same training device; A target sub-network model with the largest unit computation amount is determined from the M sub-network models.

5. The method according to claim 1, characterized in that The parallel strategy based on the M sub-network models is used to construct a training system for the network model, including: According to the parallel strategy of each sub-network model, a communication group corresponding to each sub-network model is constructed; A component communication topology structure is established between the M communication groups to generate a training system for the network model.

6. The method according to claim 5, characterized in that The network model also includes the sub-network model M i Sub-network model M with cascade relationship j , the sub-network model M i Corresponding to communication group i, the sub-network model M j Corresponding to communication group j; j is an integer less than or equal to M, and i is not equal to j; The establishing of the component communication topology structure between the M communication groups includes: Performing group division processing on the communication group i and the communication group j to obtain P communication groups, where P is a positive integer; each of the communication groups includes at least one sub-communication group obtained by dividing the communication group i and at least one sub-communication group obtained by dividing the communication group j; In the process of establishing the component communication topology structure between the communication group i and the communication group j, an inter-machine communication bandwidth is allocated to each communication group, and a communication channel between sub-communication groups belonging to different communication groups is established in each communication group.

7. The method according to claim 6, characterized in that The parallel parameters include data parallel parameters, and the data parallel parameters include data parallelism; The group division processing is performed on the communication group i and the communication group j to obtain P communication groups, including: Get the sub-network model M i The parallel strategy indicates the degree of data parallelism and the subnetwork model M j The greatest common divisor between the data parallelisms indicated by the parallel strategies; Determine the greatest common divisor as the number of groups P; Based on the group quantity P, the communication group i and the communication group j are divided into groups to obtain P communication groups.

8. The method according to claim 6, characterized in that Each of the sub-communication groups includes one or more training devices; The step of establishing a communication channel between sub-communication groups belonging to different communication groups in each communication group comprises: According to the sub-network model M i With the sub-network model M j The cascade relationship between them determines the component communication sequence between the communication group i and the communication group j; In each of the sub-communication groups obtained by dividing the communication group with the component communication sequence first, selecting a training device to be determined as a communication transmitting device; In each of the sub-communication groups obtained by dividing the communication group whose component communication order is later, selecting a training device as a communication receiving device; A communication channel is established between the communication transmitting device and the communication receiving device.

9. The method according to claim 5, characterized in that The sub-network model M i corresponding to communication group i; the parallel parameters include data parallel parameters and model parallel parameters, and the data parallel parameters include data parallelism; According to the sub-network model M i The parallel strategy for constructing the communication group i includes: Based on the sub-network model M i The number of training devices in the communication group i is determined based on the data parallelism and the model parallelism parameters in the parallel parameters indicated by the parallel strategy; Obtain the device types in the training device cluster used for model training, and according to the sub-network model M i The parallel strategy determines the sub-network model M i The training time on each equipment type is long; Based on the sub-network model M i The training time on the training device of each device type is used to determine the device type of the training device in the communication group i; According to the sub-network model M i The parallel strategy is used, and based on the number and type of training devices in the communication group i, the sub-network model M is initialized in the training device cluster. i , to construct the communication group i.

10. The method according to claim 9, characterized in that The sub-network model M i The training time consumed on the training device of each device type, and determining the device type of the training device in the communication group i, include: Based on the sub-network model M i The training time on the training device of each device type is used to determine the target device type corresponding to the minimum training time; Acquire the number of training devices of the target device type from the training device cluster; When the number of training devices of the target device category is greater than or equal to the number of training devices in the communication group i, the target device category is determined as the device category of the training devices in the communication group i.

11. The method according to claim 9, characterized in that The data parallel parameters also include micro-batch size; According to the sub-network model M i The parallel strategy determines the sub-network model M i Time spent training on training equipment of each equipment type, including: Get the sub-network model M i The time it takes to train units on training equipment of each equipment category; Based on the sub-network model M i The parallel strategy indicates the mini-batch size, model parallel parameters, and the sub-network model M i The unit training time on the training device of each device type determines the sub-network model M i Training on training equipment of each equipment category takes time.

12. The method according to claim 1, characterized in that In the process of model training the network model, calling the M communication groups in the training system to respectively execute the parallel training tasks of the corresponding sub-network models according to the execution mode indicated by the parallel strategy of the corresponding sub-network model includes: In the process of model training the network model, the model training process of the network model is divided into M stages in the pipeline, and one stage corresponds to a parallel training task of one sub-network model; Execute the pipeline, and in the process of executing the pipeline, determine the target communication group used to execute the current stage, call the target communication group to execute the parallel training task of the sub-network model corresponding to the target communication group according to the execution mode indicated by the parallel strategy of the sub-network model corresponding to the target communication group, and the current stage is the stage currently being executed among the M stages.

13. A model training device, characterized in that: The device comprises: An acquisition unit is used to acquire a network model to be trained, wherein the network model includes M sub-network models, where M is an integer greater than 1; the network model includes a multimodal large language model, and the M sub-network models include a modality encoder, a large language model, and a modality decoder; the modality encoder is used to convert multimodal information input to the multimodal large language model into a text representation, and the multimodal information includes any one of the following: text modality information, image modality information, audio modality information, and video modality information; the large language model is used to learn the language pattern, contextual information, and semantic association in the text representation converted by the modality encoder, and generate a text sequence with language ability; the modality decoder is used to translate the text sequence generated by the large language model back to the corresponding modality information for output; A processing unit, used to perform strategy decision processing on each of the sub-network models respectively to obtain a parallel strategy for each of the sub-network models; the parallel strategy of any of the sub-network models is used to indicate the parallel training tasks of any of the sub-network models during the model training process and the execution method of the parallel training tasks; The processing unit is further used to construct a training system for the network model based on the parallel strategy of the M sub-network models, the training system comprising M communication groups; one communication group corresponds to one sub-network model, each communication group is constructed according to the parallel strategy of the corresponding sub-network model, the communication group comprises one or more training devices, and the modal encoder, large language model and modal decoder are matched with different types of training devices; The processing unit is further used to call the M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models respectively according to the execution mode indicated by the parallel strategy of the corresponding sub-network models during the model training of the network model, so as to obtain the trained network model; The parallel strategy of any of the sub-network models is also used to indicate the parallel parameters of any of the sub-network models, and any of the sub-network models is represented as a sub-network model M i , i is an integer less than or equal to M; the sub-network model M i The parallel strategy is to select the sub-network model M i When the plurality of reference parallel strategies include a first reference parallel strategy, the parallel parameters include data parallel parameters, the data parallel parameters include a micro-batch size and a data parallelism, and the processing unit is further configured to: If the sub-network model M i If the target sub-network model is not a target sub-network model, obtaining a second product between the micro-batch size indicated by the parallel strategy of the target sub-network model and the data parallelism; Using a second ratio between the second product and the data parallelism indicated by the first reference parallel strategy, updating the micro-batch size indicated by the first reference parallel strategy; or, The data parallelism indicated by the first reference parallel strategy is updated by using a third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy.

14. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the model training method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the model training method as described in any one of claims 1-12.

16. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the model training method as described in any one of claims 1 to 12.

Images