Model training method, communication device and storage medium

Abstract

The invention provides a model training method, a communication device and a storage medium, relates to the technical field of communication, and aims to reduce the occurrence frequency of hash conflicts in a communication process between computing cards so as to guarantee the communication efficiency between the computing cards. The method comprises the steps of obtaining a training result of each sub-model training task, and determining a training result of a model training task according to the training result of each sub-model training task, the training result of the sub-model training task is used for indicating an execution result obtained by executing the sub-model training task in parallel by the computing card obtained by mapping the sub-model training task.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a model training method, a communication device, and a storage medium. Background Technology

[0002] As the parameter size of large models continues to grow, the amount of data required for their training also increases, leading to significant computational and storage resource consumption during training. To meet the computational and storage demands of large model training, a parallel training strategy can be employed. This strategy directs the parallel execution of at least one sub-model training task required for training the large model. Before using this strategy, multiple computing cards involved in the large model training can be sequentially mapped to at least one sub-model training task. This allows for subsequent control over the computing cards mapped to at least one sub-model training task during the large model training process.

[0003] The current method for mapping computation cards to sub-model training tasks only considers the prediction traffic between the computation cards mapped to the sub-training tasks and / or whether the computation cards are located on the same server. The method described above for mapping computation cards to sub-model training tasks is too one-sided. As a result, in the actual communication process of the computation cards, different computation cards may communicate through a single switch at the same time. This means that hash collisions may occur during the communication between the computation cards, which may cause significant losses, such as network congestion, and thus reduce the communication efficiency between the computation cards. Summary of the Invention

[0004] To address the aforementioned technical issues, embodiments of this application provide a model training method, a communication device, and a storage medium to reduce losses caused by hash collisions during communication between computing cards, thereby ensuring the communication efficiency between computing cards.

[0005] Firstly, a model training method is provided. This method is used to execute a model training task, which is divided into at least one sub-model training task. Each sub-model training task is mapped to multiple computing cards. The computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and / or the communication distance between any two computing cards. The predicted communication volume of the sub-model training task indicates the sum of the predicted data transmission amounts between the computing cards mapped to the sub-model training task. The communication distance between two computing cards characterizes the number of switches traversed by the data transmitted between the two computing cards. This method can be executed by a first device, or by a component of the first device, such as its processor, chip, or chip system, or by a logic module or software capable of implementing all or part of the first device. The following description uses the execution of this method by a first device as an example. The model training method includes: obtaining the training result of each sub-model training task, and determining the training result of the model training task based on the training result of each sub-model training task. The training result of the sub-model training task is used to indicate the execution result obtained by the computing cards mapped to the sub-model training task executing the sub-model training task in parallel.

[0006] In this embodiment, the first device can acquire the training results of each sub-model training task in at least one sub-model training task obtained based on the model training task division, and determine the training result of the model training task based on the training results of each sub-model training task. Specifically, at least one sub-model training task is mapped to multiple computing cards, and the computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computing cards.

[0007] Furthermore, the communication distance between two computing cards is used to characterize the number of switches through which data is transmitted between the two computing cards, rather than simply considering whether they are located within the same training server. If only the location within the same training server is considered, the communication distance between the two computing cards would be 0 if they are located within the same training server, and 1 if they are not located within the same training server. However, the communication distance between the two computing cards described in this application is not limited to 0 or 1; it is the actual number of switches through which data is transmitted between the two computing cards. In other words, the communication distance between the two computing cards described in this application is a more precise value.

[0008] The predicted communication volume of the sub-model training task is used to indicate the sum of the data transmitted between the computing cards to which the predicted sub-model training task is mapped, rather than simply considering the data transmitted by a single computing card in a single transaction. This allows the predicted communication volume of the sub-model training task described in the embodiments of this application to more comprehensively reflect the actual data transmitted between the computing cards to which the sub-model training task is mapped.

[0009] Therefore, since the communication distance described in the embodiments of this application can reflect the communication distance between two computing cards in a more refined manner, and the predicted communication volume of the sub-model training task described in the embodiments of this application can more comprehensively reflect the actual amount of data transmitted between the computing cards mapped to the sub-model training task, the computing cards mapped to the sub-training task based on the predicted communication volume of each sub-model training task and the communication distance between each pair of computing cards can better fit the actual communication needs of the computing cards, thereby minimizing the loss caused by hash collisions during communication between computing cards, and thus ensuring the communication efficiency between computing cards.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the greater the prediction communication volume of the sub-model training task, the smaller the sum of the communication distances between any two computing cards in the computing cards mapped to the sub-model training task.

[0011] In this implementation, a smaller communication distance between computing cards can be interpreted as a stronger network affinity between them, while a larger communication distance can be interpreted as a weaker network affinity. In this case, the greater the prediction communication volume of the sub-model training task, the stronger the affinity between the computing cards mapped to the sub-model training task, which means a smaller communication distance between the computing cards mapped to the sub-model training task.

[0012] In conjunction with the first aspect above, in one possible implementation, the method provided in this application further includes: taking the sub-model training task with the largest predicted communication volume among at least one sub-model training task as the starting sub-model training task, and performing the following mapping process for each sub-model training task in the at least one sub-model training task: based on the parallelism value of the sub-model training task and the communication distance between every two computing cards in the multiple computing cards, performing clustering processing on the multiple computing cards to obtain the clustering results corresponding to the sub-model training task, and selecting one cluster from the clustering results corresponding to the sub-model training task as the computing card mapped by the sub-model training task, wherein the clustering result includes at least one cluster, and one cluster includes M computing cards; the parallelism value is used to indicate the number M of computing cards mapped by the sub-model training task.

[0013] In this implementation, the first device can perform multiple mapping processes based on the above method until each sub-model training task is mapped to a computing card. This ensures that no sub-model training task remains unmapped. Computing cards are mapped sequentially for each sub-model training task, prioritizing the sub-model training task with the highest prediction communication volume to ensure the communication performance of the computing cards mapped to these tasks. Furthermore, the first device can cluster the multiple computing cards based on the parallelism value M of the sub-model training task and the communication distance between each pair of computing cards to select suitable M computing cards for each sub-training task. Since the clustering results determined by the clustering algorithm accurately meet the clustering requirements related to parallelism value and communication distance, the computing cards mapped to the sub-model training tasks based on the clustering results also accurately meet these requirements, further ensuring the communication performance of the computing cards mapped to the sub-model training tasks.

[0014] In conjunction with the first aspect above, in one possible implementation, selecting a cluster from the clustering results corresponding to the sub-model training task as the computation card for the sub-model training task mapping includes: selecting a first cluster with the smallest sum of communication distances from at least one cluster, and using the computation cards included in the first cluster as the computation cards for the sub-model training task mapping, wherein the sum of communication distances is the sum of the communication distances between every two computation cards included in the cluster.

[0015] In this implementation, the first device can use the computing cards in the first cluster with the smallest sum of communication distances as the computing cards for the sub-model training task mapping. This ensures that the sum of communication distances between any two computing cards in the first cluster with the smallest sum of communication distances in the clustering results of each sub-model training task mapping is minimized. Since a larger communication distance between computing cards increases the probability of hash collisions, reducing the sum of communication distances between any two computing cards in the sub-model training task mapping reduces the probability of hash collisions.

[0016] In conjunction with the first aspect above, in one possible implementation, the sub-model training task with the largest predicted communication volume in at least one sub-model training task is taken as the starting sub-model training task, and each sub-model training task in at least one sub-model training task is traversed, including: sorting at least one sub-model training task in descending order of predicted communication volume to obtain a sub-model training task sequence, and traversing each sub-model training task in at least one sub-model training task based on the sub-model training task sequence.

[0017] In this implementation, the first device maps computation cards to sub-model training tasks in descending order of predicted communication volume. This allows the first device to prioritize mapping computation cards to sub-model training tasks with higher predicted communication volume, and to prioritize mapping computation cards with shorter communication distances to sub-model training tasks with higher predicted communication volume. This ensures that the distance between the computation cards mapped to sub-model training tasks with higher predicted communication volume is as small as possible, thereby making the network affinity of the determined group information stronger, so as to minimize the probability of hash collisions in the future.

[0018] In conjunction with the first aspect above, in one possible implementation, the method provided in this application embodiment further includes: obtaining the parallelism value of each sub-model training task, and determining the prediction communication volume of each sub-model training task based on the parallelism value corresponding to each sub-model training task, wherein the parallelism value is used to indicate the number of computing cards mapped to the sub-model training task.

[0019] In this implementation, since the parallelism value can be used to limit at least one of the following: the number of sub-training datasets, the number of sub-model training tasks obtained by partitioning models at different levels, the number of sub-model training tasks obtained by partitioning models at the same level, and the number of computing cards mapped to a sub-model training task, that is, the parallelism value can be related to multiple parameters related to the sub-model training task. Therefore, the predicted communication volume of each sub-model training task determined based on the parallelism value corresponding to each sub-model training task can accurately reflect the communication situation of each sub-model training task, providing a more accurate data foundation for determining grouping information based on the predicted communication volume of each sub-model training task.

[0020] In conjunction with the first aspect above, in one possible implementation, determining the predicted communication volume of each sub-model training task based on the parallelism value corresponding to each sub-model training task includes: determining the predicted traffic and predicted communication count of the second sub-model training task based on the parallelism value of the second sub-model training task, and determining the predicted communication volume of the second sub-model training task based on the predicted traffic and predicted communication count of the second sub-model training task, wherein the second sub-model training task is any one of the sub-model training tasks in the set of sub-model training tasks, and the predicted traffic is the sum of the data volume transmitted in a single transmission between every two computing cards in the computing card mapped to the second sub-model training task.

[0021] In this implementation, when the first device determines the predicted communication volume of each sub-model training task based on the parallelism value corresponding to each sub-model training task, it considers not only the amount of data transmitted in a single transmission (i.e., the predicted traffic) but also the number of predicted communications. Compared with the prior art, which determines the predicted communication volume of each sub-model training task based on the predicted traffic of each sub-model training task, the predicted communication volume of each sub-model training task described in this application embodiment can further and more accurately reflect the communication situation of each sub-model training task, providing a more accurate data foundation for determining grouping information based on the predicted communication volume of each sub-model training task.

[0022] In conjunction with the first aspect above, in one possible implementation, the method provided in this application embodiment further includes: obtaining network topology information of the training cluster, and based on the network topology information, calculating the communication distance between every two computing cards in the multiple computing cards, wherein the training cluster includes at least one switch and at least one training server, and multiple computing cards are deployed in the at least one training server, wherein computing cards within the same training server are directly connected, or computing cards within different training servers are connected through a switch.

[0023] In this implementation, the communication distance between the two computing cards described in this application embodiment is used to characterize the number of switches through which data transmitted between the two computing cards passes, rather than simply considering whether they are located within the same training server. If only whether they are located within the same training server is considered, then the communication distance between the two computing cards is 0 when they are located within the same training server, and 1 when they are not located within the same training server. However, the communication distance between the two computing cards described in this application is not limited to 0 or 1; it is the actual number of switches through which data is transmitted between the two computing cards. In other words, the communication distance between the two computing cards described in this application is a more precise value.

[0024] In conjunction with the first aspect above, in one possible implementation, the method provided in this application embodiment further includes: sending grouping information, the grouping information being used to indicate the computation card mapped to each sub-model training task.

[0025] In this implementation, the first device can send the computing card mapped to each sub-model training task so that the subsequent training server can control at least one computing card mapped to a sub-model training task to train the large model.

[0026] In conjunction with the first aspect above, in one possible implementation, sending packet information includes: sending packet information to each training server in the training cluster, the training cluster including a controller for controlling the training cluster, at least one switch, and at least one training server, wherein multiple computing cards are deployed in the at least one training server, wherein computing cards within the same training server are directly connected, or computing cards within different training servers are connected through a switch.

[0027] In this implementation, if the training cluster includes a controller for controlling the training cluster, the controller can execute the relevant steps of the model training method described in the embodiments of this application, and then the controller can send group information to each training server in the training cluster so that the subsequent training server can control its multiple computing cards to perform model training tasks based on the group information.

[0028] In conjunction with the first aspect above, in one possible implementation, sending packet information includes: sending packet information so that a second training server can obtain the packet information; the training cluster includes at least one switch and at least one training server; multiple computing cards are deployed in the at least one training server; computing cards within the same training server are directly connected, or computing cards within different training servers are connected through a switch; wherein the second training server is a training server other than the first training server among the at least one training server; and the first training server is used to determine the packet information.

[0029] In this implementation, if the training cluster does not include a controller for controlling the training cluster, the training server in the training cluster (referred to as the first training server) can execute the relevant steps of the model training method described in the embodiments of this application. Then, the first training server sends group information to other training servers in the training cluster (referred to as the second training server) so that each training server can control its multiple computing cards to perform model training tasks based on the group information.

[0030] Secondly, a communication device is provided for implementing the various methods described above. This communication device can be the first device described in the first aspect, or any implementation thereof, or a device including the first device, or a device included in the first device, such as a chip. The communication device includes modules, units, or means corresponding to the methods described above. These modules, units, or means can be implemented in hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the functions described above.

[0031] In some possible designs, the communication device may include a processing module and a transceiver module. The transceiver module, also referred to as a transceiver unit, is used to implement the transmission and / or reception functions in any of the above aspects and their possible implementations. The transceiver module may consist of transceiver circuits, transceivers, transceivers, or communication interfaces. The processing module can be used to implement the processing functions in any of the above aspects and their possible implementations.

[0032] In some possible designs, the transceiver module includes a sending module and a receiving module, which are used to implement the sending and receiving functions in any of the above aspects and any possible implementation methods.

[0033] Thirdly, a communication device is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the communication device to perform the methods of any of the above aspects. The communication device may be the first device described in the first aspect, or any implementation thereof, or a device including the first device, or a device included in the first device, such as a chip.

[0034] Fourthly, a communication device is provided, comprising: a processor and a communication interface; the communication interface being used to communicate with a module outside the communication device; the processor being used to execute computer programs or instructions such that the communication device can be a first device as described in the first aspect above, or any implementation thereof, or a device including the first device, or a device included in the first device, such as a chip.

[0035] Fifthly, a communication device is provided, comprising: at least one processor; the processor being configured to execute a computer program or instructions stored in a memory to cause the communication device to perform the methods of any of the above aspects. The memory may be coupled to the processor, or may be independent of the processor. The communication device may be a first device according to the first aspect, or any implementation thereof, or a device including the first device, or a device included in the first device, such as a chip.

[0036] In a sixth aspect, a computer-readable storage medium is provided, which stores a computer program or instructions that, when executed on a communication device, enable the communication device to perform the methods of any of the above aspects or any implementation thereof.

[0037] In a seventh aspect, a computer program product containing instructions is provided, which, when run on a communication device, enables the communication device to execute any of the above aspects or any implementation thereof.

[0038] Eighthly, a communication device (e.g., a chip or chip system) is provided, the communication device including a processor for implementing the functions involved in any of the above aspects or any implementation thereof.

[0039] In some possible designs, the communication device includes a memory for storing necessary program instructions and data.

[0040] In some possible designs, when the device is a chip system, it can be composed of chips or contain chips and other discrete components.

[0041] It is understood that when the communication device provided by any of the third to sixth aspects is a chip, the aforementioned sending action / function can be understood as an output, and the aforementioned receiving action / function can be understood as an input.

[0042] Ninthly, a model training method is provided, which includes the method of the first aspect or any implementation thereof, and the method of the second aspect or any implementation thereof.

[0043] In a tenth aspect, a communication system is provided, the communication system including the first device described above.

[0044] Eleventhly, a computer program product is provided that, when run on a communication device, enables the communication device to execute the method of any of the above aspects or any implementation thereof.

[0045] In a twelfth aspect, a training cluster is provided, the training cluster being used to indicate a model training method. The training cluster includes at least one switch and at least one training server, or the training cluster includes a controller for controlling the training cluster, at least one switch, and at least one training server, wherein multiple computing cards are deployed in the at least one training server; the multiple computing cards map at least one sub-model training task, the at least one sub-model training task being obtained by partitioning a model training task, and the computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and the communication distance between every two computing cards; the predicted communication volume of the sub-model training task is used to indicate the sum of the predicted data volume transmitted between the computing cards to which the sub-model training task is mapped, and the communication distance between two computing cards is used to characterize the number of switches through which the data transmitted between the two computing cards passes; the computing card mapped to each sub-model training task is used to execute the corresponding sub-model training task in parallel to obtain the training result; the controller or any one of the training servers is used to obtain the training result of each sub-model training task and determine the training result of the model training task based on the training result of each sub-model training task.

[0046] The technical effects of any of the implementation methods in aspects two through twelfth can be found in the technical effects of the corresponding implementation method in aspect one, and will not be repeated here.

[0047] Among these, any possible implementation methods of any one of the above aspects can be combined, provided that the solutions do not contradict each other. Attached Figure Description

[0048] Figure 1 A schematic diagram of the topology of a training cluster provided in an embodiment of this application;

[0049] Figure 2 A flowchart of a computation card for mapping training tasks for at least one sub-model is provided as an embodiment of this application;

[0050] Figure 3 A schematic diagram of another training cluster topology provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0052] Figures 5-11 A schematic flowchart illustrating the model training method provided in this application embodiment;

[0053] Figure 12 This is a schematic diagram of another communication device provided in an embodiment of this application. Detailed Implementation

[0054] To facilitate understanding of the technical solutions provided in the embodiments of this application, a brief introduction to the relevant technologies of this application is given first. The brief introduction is as follows:

[0055] 1. Large Model

[0056] Large models refer to deep learning models with billions or more parameters.

[0057] 2. Model Training

[0058] Model training refers to adjusting the parameters of a model so that it can learn patterns from training data, enabling the model to predict or classify unknown data based on these learned patterns.

[0059] 3. Parallel Training

[0060] As described above regarding "large models," these models typically have billions or more parameters. Consequently, the amount of data required for training these models also grows exponentially, consuming enormous computing and storage resources. However, a single computing card is far from sufficient to meet the computational and storage demands of training large models. To address these resource requirements, parallel training can be implemented.

[0061] Parallel training is a method to improve model training efficiency, which refers to using multiple training servers or multiple computing cards to execute a model training task simultaneously. Typically, a training server deploys at least one computing card. A training server can refer to a server capable of training large models.

[0062] Furthermore, the parallel training process can be implemented based on parallel training strategies. The following provides a detailed explanation of these parallel training strategies.

[0063] Parallel training strategies can be used to indicate the computational cards mapped to at least one sub-model training task. A sub-model training task is mapped to one or more computational cards, which then execute the training task corresponding to the sub-model training task. Optionally, parallel training strategies can be categorized into data parallelism (DP) training strategies and / or model parallel training strategies based on the way the training tasks are divided. Typically, data parallelism training strategies and model parallelism training strategies are combined for model training to achieve efficient parallel training.

[0064] The method of dividing the sub-model training tasks can include dividing the training dataset or dividing the large model. If at least one sub-model training task indicated by the parallel training strategy is obtained by dividing the training dataset, then the parallel training strategy is a data parallel training strategy; if at least one sub-model training task indicated by the parallel training strategy is obtained by dividing the large model, then the parallel training strategy is a model parallel training strategy.

[0065] For example, a data-parallel training strategy specifies at least one sub-model training task, which is determined by partitioning the training dataset. Specifically, the controller can divide the training dataset into at least one sub-training dataset, determine at least one sub-model training task based on this sub-training dataset, and map these sub-model training tasks to multiple computing cards. This allows each computing card mapped to a sub-model training task to execute the sub-model training task in parallel based on its corresponding sub-training dataset. Data-parallel training strategies are primarily applied in scenarios where the computing resources of a single computing card are insufficient (or the computing resource requirements cannot be met).

[0066] Furthermore, when using a data-parallel training strategy, it is necessary to specify the DP parallelism (hereinafter referred to as DP). DP can be used to limit the number of sub-training datasets and the number of computation cards mapped to a sub-model training task. In addition, the sub-model training task can also be called a DP sub-model training task, and the computation cards mapped to the DP sub-model training task can also form a DP communication group.

[0067] Taking a DP sub-model training task as an example: The controller can divide the training dataset into DP sub-training datasets, determine DP sub-model training tasks based on the DP sub-training datasets, and map the DP sub-model training tasks to DP computing cards, so that the DP computing cards can execute the DP sub-model training tasks in parallel based on the corresponding sub-training datasets. Each of the DP computing cards processes a portion of the training data (i.e., a sub-training dataset), and each computing card stores a complete model.

[0068] Furthermore, for a large model training task, if the parallel training strategy corresponding to the model training task instructs the partitioning of the training dataset to divide the model training task into at least one sub-model training task, and executes at least one sub-model training task in parallel to complete the large model training task, then the controller can establish a mapping relationship between at least one DP sub-model training task set and computing cards based on the above method. Each DP sub-model training task set includes DP sub-model training tasks, and these DP sub-model training tasks map to DP computing cards. Moreover, different DP sub-model training task sets may include DP sub-model training tasks mapped to different DP computing cards.

[0069] Understandably, data-parallel training strategies have lower computational power requirements for individual computing cards. However, in the process of updating the parameters of a large model, it is necessary to aggregate the gradients of the training results from all computing cards, which introduces additional communication overhead.

[0070] For example, a model parallel training strategy can instruct at least one computational card to be mapped to a sub-model training task, where the sub-model training task is determined by partitioning a large model. Specifically, the controller can divide the large model into at least one sub-large model, determine at least one sub-model training task based on this sub-large model, and map this sub-model training task to multiple computational cards, so that each computational card mapped to a sub-model training task can execute the corresponding sub-model training task in parallel. Model parallel training strategies are primarily applied in scenarios where the storage resources of a single computational card are insufficient (or storage resource requirements cannot be met).

[0071] Furthermore, model parallel training strategies may include pipeline parallelism (PP) training strategies and / or tensor parallelism (TP) training strategies.

[0072] The pipelined parallel training strategy specifies at least one sub-model training task, which is determined by partitioning different layers of the large model. Specifically, the controller can divide the large model into at least one sub-large model according to the layers of the neural network model, determine at least one sub-model training task based on this sub-large model, and map this sub-model training task onto multiple computing cards so that these multiple computing cards can execute the corresponding sub-model training tasks in parallel. In other words, different layers of the large model are allocated to different computing cards, allowing these different computing cards to execute the sub-model training tasks corresponding to different layers in parallel.

[0073] Furthermore, when using a pipelined parallel training strategy, it is necessary to specify the PP parallelism (hereinafter referred to as PP). PP can be used to limit the number of sub-models and the number of compute cards mapped to a sub-model training task. In addition, the sub-model training task can also be called a PP sub-model training task, and the compute cards mapped to the PP sub-model training task can also form a PP communication group.

[0074] To illustrate with an example of a PP sub-model training task: the controller can divide the large model into PP sub-large models according to the hierarchy of the neural model, determine the PP sub-model training tasks based on the PP sub-large models, and map the PP sub-model training tasks to PP computing cards so that the PP computing cards can execute the PP sub-model training tasks in parallel.

[0075] Furthermore, for a model training task, if the parallel training strategy corresponding to the task instructs the partitioning of a large model into different layers to divide the training task into at least one sub-model training task, and executes these sub-model training tasks in parallel to complete the training of the large model, then the controller can establish a mapping relationship between at least one PP sub-model training task set and computing cards based on the above method. Each PP sub-model training task set includes PP sub-model training tasks, and these PP sub-model training tasks map to PP computing cards. Moreover, different PP sub-model training task sets may have different PP computing cards mapped to their respective PP sub-model training tasks.

[0076] The tensor parallel training strategy instructs at least one sub-model training task, which is determined by partitioning the same level of the large model. Specifically, the controller can divide a large model at one level into at least one sub-large model, determine at least one sub-model training task based on this sub-large model, and map these at least one sub-model training tasks onto multiple computing cards, enabling these multiple computing cards to execute the corresponding sub-model training tasks in parallel. In other words, different parts of the same level of the large model are allocated to different computing cards, allowing these different computing cards to execute the sub-model training tasks corresponding to different parts of the same level in parallel.

[0077] Furthermore, when using the tensor parallel training strategy, it is necessary to specify the TP parallelism (hereinafter referred to as TP). TP can be used to limit the number of sub-models and the number of computation cards mapped to a sub-model training task. In addition, the sub-model training task can also be called a TP sub-model training task, and the computation cards mapped to the TP sub-model training task can also form a TP communication group.

[0078] To illustrate with an example of a TP sub-model training task: the controller can divide a layer of a large model into TP sub-large models, determine TP sub-model training tasks based on the TP sub-large models, and map the TP sub-model training tasks to TP computing cards so that the TP computing cards can execute the TP sub-model training tasks in parallel.

[0079] Furthermore, for a large model training task, if the parallel training strategy corresponding to the model training task instructs the same level of the large model to be divided into at least one sub-model training task, and at least one sub-model training task is executed in parallel to complete the large model training task, then the controller can establish a mapping relationship between at least one TP sub-model training task set and computing cards based on the above method. Each TP sub-model training task set includes TP sub-model training tasks, and each TP sub-model training task maps to TP computing cards. Moreover, different TP sub-model training task sets include different TP sub-model training tasks mapped to different TP computing cards.

[0080] 4. Hash collision

[0081] Hash collisions occur when different inputs are mapped to the same hash value (i.e., the same index in an array) by a hash function, leading to conflicts when processing those inputs. However, in a model training scenario, a training cluster used to perform model training tasks can include at least one of the following: a controller, at least one switch, or at least one training server. The controller is primarily used to schedule the training cluster to execute training tasks. The switch is primarily used to connect different training servers, enabling communication between them. The training server is primarily used to execute training tasks, and the computing cards within the training server can execute training tasks in parallel.

[0082] Furthermore, when multiple training servers transmit data simultaneously within a training cluster, the switches through which different training servers transmit data overlap. That is, the switches through which different training servers transmit data are mapped to the same hash value by the hash function, which can be considered as a hash collision occurring within the training cluster.

[0083] For example, Figure 1 A schematic diagram of the training cluster topology is shown. (For example...) Figure 1 As shown, the training cluster includes controller 1, switches 1 to 4, and training servers 1 to 4. Each of the four training servers may include at least one computing card.

[0084] When training server 1 transmits data to training server 3, training server 1 can transmit data to training server 3 via switch 3-switch 1-switch 4, or via switch 3-switch 2-switch 4. Similarly, when training server 2 transmits data to training server 4, training server 2 can transmit data to training server 4 via switch 3-switch 1-switch 4, or via switch 3-switch 2-switch 4.

[0085] If training server 1 is transmitting data to training server 3 while training server 2 is also transmitting data to training server 4, then both training server 1 and training server 2 may transmit data to and from training server 3 and training server 4 via switch 3-switch 1-switch 4. Given this, it can be seen that both training server 1 and training server 2 need to pass through switch 1 when transmitting data, which will cause hash collisions on switch 1. This will lead to switch 1 being overloaded while switch 2 is idle, resulting in an unbalanced load across the switches. In other words, switch 1 is likely to experience network congestion, while switch 2 will suffer from resource waste. Furthermore, both training server 1 and training server 2 also need to pass through switches 3 and 4 when transmitting data, which will also cause hash collisions on switches 3 and 4.

[0086] Furthermore, it is understandable that if the number of switches through which data is transmitted between two computing cards is large, that is, the number of switches through which hash collisions can occur, the probability of sending hash collisions during communication between computing cards is higher.

[0087] The above is a brief introduction to the relevant technologies of this application.

[0088] As mentioned above regarding "parallel training strategies," parallel training strategies can include data parallel training strategies and / or model parallel training strategies. Model parallel training strategies can include pipeline parallel training strategies and / or tensor parallel training strategies. In other words, a parallel training strategy can be a combination of data parallel training strategies, pipeline parallel training strategies, and tensor parallel training strategies. In this case, the parallel training strategy can be called a 3D parallel training strategy.

[0089] Since the parallel training strategy can be used to indicate the training tasks corresponding to at least one sub-model training task required for parallel execution of model training, before using the parallel training strategy for model training, multiple computing cards participating in model training can be sequentially mapped to the above-mentioned at least one sub-model training task, so that the computing cards mapped to at least one sub-model training task can be controlled to perform model training based on the parallel training strategy.

[0090] Furthermore, Figure 2 The implementation process of a computation card for training task mappings for at least one sub-model is shown. For example... Figure 2 As shown, taking the parallel training strategy as an example of the 3D parallel training strategy, the implementation process of the computing card for the training task mapping of at least one sub-model can be achieved through the following S201 to S204.

[0091] S201. The controller responds to user input operations and obtains PP, TP, and DP corresponding to the model training task input by the user.

[0092] For example, the controller described above can be a Megatron. Of course, the above is merely an exemplary description of a controller, and the controller can also be other devices; this application embodiment does not impose any limitations on this.

[0093] It is understandable that if the user inputs PP, TP, and DP for the model training task, it indicates that the parallel training strategy used for the model training task is a 3D parallel training strategy.

[0094] S202, the controller determines, based on the above PP, TP, and DP, at least one sub-model training task for parallel execution of the model training task, and the prediction flow between computing cards mapped to the sub-model training task.

[0095] The predicted traffic is the amount of data transferred between the two computing cards in a single transaction.

[0096] Optionally, the process by which the controller determines at least one sub-model training task for parallel execution of the model training task based on the above-mentioned PP, TP, and DP can be as follows: the controller adaptively determines PP, TP, and DP as the number of computing cards mapped to each sub-model training task in at least one sub-model training task, divides at least one computing card based on the number of computing cards mapped to each sub-model training task to obtain the number of the above-mentioned at least one sub-model training tasks, and determines the above-mentioned at least one sub-model training task based on the number of the above-mentioned at least one sub-model training tasks and the number of computing cards mapped to each sub-model training task in at least one sub-model training task, according to the sub-model task division method described in the above-mentioned parallel training strategy.

[0097] Optionally, the process by which the controller determines the prediction flow between computing cards mapping sub-model training tasks based on PP, TP, and DP can be as follows: The controller can partition the training dataset or model based on PP, TP, and DP to obtain at least one sub-training task, and determine the amount of data that the computing card needs to generate to execute each sub-training task based on the amount of data in the training dataset and the amount of data in the model. The amount of data that the computing card needs to generate to execute each sub-training task is the amount of data that the computing card needs to transmit, which is also the prediction flow between computing cards mapping sub-model training tasks.

[0098] S203. The controller determines the mapping relationship between the sub-model training task and the computing card based on the predicted flow between the computing cards mapped to the sub-model training task.

[0099] Among them, sub-model training tasks with large prediction traffic between computing cards can be mapped to computing cards within the same server as much as possible, while sub-model training tasks with small prediction traffic between computing cards can be mapped to computing cards across servers.

[0100] S204. The controller sends the mapping relationship between the sub-model training tasks and the computing cards to the training server. Correspondingly, the training server receives the mapping relationship between the sub-model training tasks and the computing cards from the controller.

[0101] As mentioned above, the current method for mapping computation cards to sub-model training tasks only considers the prediction traffic between the computation cards mapped to the sub-training tasks and / or whether the computation cards are located on the same server. The method described above for mapping computation cards to sub-model training tasks is too one-sided. As a result, in the actual communication process of the computation cards, different computation cards may communicate through a single switch at the same time. This means that there may be a high number of hash collisions during the communication between the computation cards. A high number of hash collisions may cause network congestion and other problems, which will reduce the communication efficiency between the computation cards.

[0102] Therefore, the first device can acquire the training results of each sub-model training task in at least one sub-model training task obtained based on the model training task partitioning, and determine the training result of the model training task based on the training results of each sub-model training task. Specifically, at least one sub-model training task is mapped to multiple computing cards, and the computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computing cards.

[0103] Furthermore, the communication distance between two computing cards is used to characterize the number of switches through which data is transmitted between the two computing cards, rather than simply considering whether they are located within the same training server. If only the location within the same training server is considered, the communication distance between the two computing cards would be 0 if they are located within the same training server, and 1 if they are not located within the same training server. However, the communication distance between the two computing cards described in this application is not limited to 0 or 1; it is the actual number of switches through which data is transmitted between the two computing cards. In other words, the communication distance between the two computing cards described in this application is a more precise value.

[0104] The predicted communication volume of the sub-model training task is used to indicate the sum of the data transmitted between the computing cards to which the predicted sub-model training task is mapped, rather than simply considering the data transmitted by a single computing card in a single transaction. This allows the predicted communication volume of the sub-model training task described in the embodiments of this application to more comprehensively reflect the actual data transmitted between the computing cards to which the sub-model training task is mapped.

[0105] Therefore, since the communication distance described in the embodiments of this application can reflect the communication distance between two computing cards in a more refined manner, and the predicted communication volume of the sub-model training task described in the embodiments of this application can more comprehensively reflect the actual amount of data transmitted between the computing cards mapped to the sub-model training task, the computing cards mapped to the sub-training task based on the predicted communication volume of each sub-model training task and the communication distance between each pair of computing cards can better fit the actual communication needs of the computing cards, thereby minimizing the loss caused by hash collisions during communication between computing cards, and thus ensuring the communication efficiency between computing cards.

[0106] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0107] To facilitate understanding of the embodiments of this application, the following points will be explained before introducing the embodiments of this application.

[0108] 1. In the embodiments of this application, for ease of description, when numbering is involved, it can start from 1 and be numbered consecutively, or it can start from 0 and be numbered from any parameter. It should be understood that the above are settings made for the convenience of describing the technical solutions provided in the embodiments of this application, and are not intended to limit the scope of the embodiments of this application.

[0109] 2. The “protocol” involved in the embodiments of this application may refer to standard protocols in the field of communication, such as the Long Term Evolution (LTE) protocol, the New Radio (NR) protocol, and related protocols applied to future communication systems. The embodiments of this application do not limit this.

[0110] 3. In the embodiments of this application, the descriptions such as "when," "under the circumstances," "if," and "if" all refer to the fact that the device (e.g., the controller or training server) will make corresponding processing under certain objective circumstances. They are not time limits, nor do they require the device (e.g., the controller or training server) to have a judgment action when implementing it, nor do they mean that there are other limitations.

[0111] 4. In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "and / or" in the embodiments of this application is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a to b, a to c, b to c, or a to b to c, where a, b, and c can be single or multiple. Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0112] 5. In this application, "at least one" means one or more. "More" means two or more. "At least two" means two or three or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0113] 6. In this application, "instruction" may include: direct instruction, or, indirect instruction, or, explicit instruction, or, implicit instruction.

[0114] In this application, "including" can include: direct inclusion, indirect inclusion, explicit inclusion, or implicit inclusion.

[0115] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0116] It should be understood that existing technologies may change as technical solutions evolve, and the technical solutions provided in this application are not limited to the existing technologies provided.

[0117] In this application, different embodiments or parts of steps (e.g., any one or more steps) can be combined to form new embodiments. The scope of steps in different embodiments is not limited; they may include optional steps, mandatory steps, or both. Unless otherwise specified or logically conflicting, the terminology and / or descriptions in different embodiments are consistent and can be mutually referenced. The order of steps in the embodiments of this application is not limited. The order of judgments of different conditions in the embodiments of this application is not limited. The terms "after" and "at" in this application are not strictly limited to specific time points.

[0118] The nouns and terms used in this application are merely examples and may be other names. This application does not impose any restrictions on their use.

[0119] Figure 3 A schematic diagram of a possible, unrestricted training cluster is shown. (e.g.) Figure 3As shown, the training cluster 3000 includes at least one of the following: a controller, N switches, and L training servers, where N and L are both positive integers. Each of the at least one training server may include at least one computing card. The controller is mainly used to schedule the training cluster to execute training tasks. The switches are mainly used to connect different training servers so that different training servers can communicate with each other through the switches. The training servers are mainly used to execute training tasks. For example, computing cards can be deployed in the training servers, and the computing cards in the training servers execute training tasks. The computing cards in different training servers or in the same training server can execute training tasks simultaneously / in parallel.

[0120] In addition to the above, Figure 3 The diagram also illustrates the hierarchical relationships between computing cards and training servers, the connections between switches and training servers, and the connections between switches. These relationships between devices, such as the hierarchical relationships between computing cards and training servers, the connections between switches and training servers, and the connections between switches, characterize the network topology of the training cluster.

[0121] Optionally, the network topology of the training cluster can be provided by the user initiating the training task, and the network topology can also be expressed in other formats such as Excel. That is, when a user first uses the training cluster for model training, the user needs to provide the network topology of the training cluster. Furthermore, the controller can parse the network topology of the training cluster and store the information indicating the network topology and the parsing results in the database. The parsing results indicate the outcome of parsing the network topology of the training cluster.

[0122] It should be understood that Figure 3 The number and type of devices in the communication system shown are for illustrative purposes only. This application is not limited to these. In actual applications, the communication system may include more switches, more training servers, or more computing cards, as well as other devices, such as other devices used to implement artificial intelligence functions.

[0123] In one possible implementation, the controller and training server in the embodiments of this application can also be referred to as communication devices, which can be a general-purpose device or a special-purpose device. The embodiments of this application do not specifically limit this.

[0124] In one possible implementation, the related functions of the controller or training server in this application embodiment can be implemented by one device, multiple devices working together, or one or more functional modules within a single device. This application embodiment does not specifically limit this. It is understood that the above functions can be network elements in hardware devices, software functions running on dedicated hardware, a combination of hardware and software, or virtualization functions instantiated on a platform (e.g., a cloud platform).

[0125] For example, the relevant functions of the controller or training server in the embodiments of this application can be achieved through... Figure 4 This is achieved through the communication device 410. Figure 4 A schematic diagram of a possible communication device is shown. It will be understood that the communication device 410 includes means of the necessary form, such as modules, units, elements, circuits, or interfaces, to be appropriately configured together to perform this solution. The communication device 410 can be... Figure 3 The controller or training server in the communication device, or a component (e.g., a chip) within these devices, is used to implement the methods described in the following method embodiments. The communication device 410 includes one or more processors 411. The processor 411 can be a general-purpose processor or a dedicated processor, such as a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, while the central processing unit can be used to control the communication device (e.g., a controller, training server, or chip), execute software programs, and process data from the software programs.

[0126] Optionally, in one design, the processor 411 may include a program 413 (sometimes also referred to as code or instructions), which can be executed on the processor 411 to cause the communication device 410 to perform the methods described in the embodiments below. In yet another possible design, the communication device 410 includes circuitry (…). Figure 4 (Not shown), the circuit is used to implement the communication function in the following embodiments.

[0127] Optionally, the communication device 410 may include one or more memories 412 storing a program 413 (sometimes referred to as code or instructions), which can be run on the processor 411 to cause the communication device 410 to perform the methods described in the following method embodiments.

[0128] Optionally, the processor 411 and / or memory 412 may include artificial intelligence (AI) modules 417 and 418, which are used to implement AI-related functions. AI modules 417 or 418 can be implemented through software, hardware, or a combination of both. For example, AI modules 417 or 418 may include a radio intelligent controller (RIC) module. For example, AI modules 417 or 418 can be near real-time RICs or non-real-time RICs. Furthermore, AI modules 417 or 418 can function as modules using large models.

[0129] Optionally, data may also be stored in the processor 411 and / or the memory 412. The processor and memory may be configured separately or integrated together.

[0130] Optionally, the communication device 410 may also include a transceiver 415 and / or an antenna 416. The processor 411, sometimes referred to as a processing unit, controls the communication device (e.g., a training server or controller). The transceiver 415, sometimes referred to as a transceiver unit, transceiver, transceiver circuit, or transceiver, is used to implement the transmission and reception functions of the communication device via the antenna 416.

[0131] The following will combine Figure 4 The model training method provided in the embodiments of this application will be described in detail.

[0132] In the following embodiments of this application, the message names, parameter names, or information names between network elements are merely examples, and other names may be used in other embodiments. The methods provided in the embodiments of this application are not specifically limited in this regard. It is understood that in the embodiments of this application, each network element may execute some or all of the steps in the embodiments of this application. These steps or operations are examples, and the embodiments of this application may also execute other operations or variations of various operations. Furthermore, the steps may be executed in different orders as presented in the embodiments of this application, and it is not necessary to execute all the operations in the embodiments of this application.

[0133] Figure 5This is an example of the model training method provided in this application. The method is described using a first device as an example. The first device can be a controller or a training server. Of course, the entity executing the controller action in this method can also be a device / module within the controller, such as a chip, processor, or processing unit within the controller; or, the entity executing the training server action in this method can also be a device / module within the training server, such as a chip, processor, or processing unit within the training server. This application does not specifically limit this. The model training method described in this application is used to execute a model training task. The model training task is divided into at least one sub-model training task, and at least one sub-model training task is mapped to multiple computing cards. The computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computing cards. The predicted communication volume of the sub-model training task is used to indicate the sum of the predicted data transmitted between the computing cards mapped to the sub-model training task, and the communication distance between two computing cards is used to characterize the number of switches through which the data transmitted between the two computing cards passes. For example, as shown... Figure 5 As shown, the model training method includes the following steps:

[0134] S501, The first device acquires the training results of each sub-model training task.

[0135] The training results of the sub-model training task are used to indicate the execution results of the sub-model training task obtained by parallel execution of the sub-model training task on the computing card mapped by the sub-model training task.

[0136] S502. The first device determines the training result of the model training task based on the training result of each sub-model training task.

[0137] Optionally, the implementation process of S502 above can be as follows: the first device can aggregate the training results of each sub-model training task based on the type of the sub-model training task to obtain the training results of the model training task. For example, if the sub-model training task is a DP sub-model training task, the first device can aggregate the training results by sub-training dataset; or, for example, if the sub-model training task is a PP sub-model training task, the first device can aggregate the training results by different levels of models.

[0138] In this embodiment, the first device can acquire the training results of each sub-model training task in at least one sub-model training task obtained based on the model training task division, and determine the training result of the model training task based on the training results of each sub-model training task. Specifically, at least one sub-model training task is mapped to multiple computing cards, and the computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computing cards.

[0139] Furthermore, the communication distance between two computing cards is used to characterize the number of switches through which data is transmitted between the two computing cards, rather than simply considering whether they are located within the same training server. If only the location within the same training server is considered, the communication distance between the two computing cards would be 0 if they are located within the same training server, and 1 if they are not located within the same training server. However, the communication distance between the two computing cards described in this application is not limited to 0 or 1; it is the actual number of switches through which data is transmitted between the two computing cards. In other words, the communication distance between the two computing cards described in this application is a more precise value.

[0140] The predicted communication volume of the sub-model training task is used to indicate the sum of the data transmitted between the computing cards to which the predicted sub-model training task is mapped, rather than simply considering the data transmitted by a single computing card in a single transaction. This allows the predicted communication volume of the sub-model training task described in the embodiments of this application to more comprehensively reflect the actual data transmitted between the computing cards to which the sub-model training task is mapped.

[0141] Therefore, since the communication distance described in the embodiments of this application can reflect the communication distance between two computing cards in a more refined manner, and the predicted communication volume of the sub-model training task described in the embodiments of this application can more comprehensively reflect the actual amount of data transmitted between the computing cards mapped to the sub-model training task, the computing cards mapped to the sub-training task based on the predicted communication volume of each sub-model training task and the communication distance between each pair of computing cards can better fit the actual communication needs of the computing cards, thereby minimizing the loss caused by hash collisions during communication between computing cards, and thus ensuring the communication efficiency between computing cards.

[0142] As can be seen from the aforementioned description of "computation cards mapped to each sub-training task", the computation cards mapped to each sub-model training task are determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computation cards in the multiple computation cards.

[0143] Furthermore, optionally, the greater the prediction communication volume of the sub-model training task, the smaller the sum of communication distances between any two computing cards in the computing cards mapped to the sub-model training task. Conversely, the smaller the prediction communication volume of the sub-model training task, the greater the sum of communication distances between any two computing cards in the computing cards mapped to the sub-model training task.

[0144] It's understandable that a smaller communication distance between computing cards indicates stronger network affinity, while a larger communication distance indicates weaker network affinity. In this case, the greater the prediction communication volume of the sub-model training task, the stronger the affinity between the computing cards mapped to the sub-model training task, meaning a smaller communication distance between them.

[0145] In this embodiment, the greater the communication distance between computing cards, the higher the probability of hash collisions, and vice versa. The greater the prediction communication volume of the sub-model training task, the longer the processing time of the switch. Consequently, if the computing card mapped to this sub-model training task experiences a hash collision with other computing cards, network congestion becomes more severe, for example, the network congestion time increases. Therefore, sub-model training tasks with higher prediction communication volumes can be mapped to computing cards with shorter communication distances. This allows sub-model training tasks that incur greater losses when hash collisions occur to be mapped to at least one computing card with a lower probability of hash collisions, thus minimizing the losses caused by hash collisions and improving communication efficiency.

[0146] As described above regarding the "computation cards mapped for each sub-training task," the computation cards mapped for each sub-model training task are determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computation cards. Therefore, before mapping computation cards for each sub-model training task, the first device needs to determine the predicted communication volume for each sub-model training task. The following details the implementation process of the first device for determining the predicted communication volume for each sub-model training task. Figure 6 As shown, the model training method described in the embodiments of this application may further include the following S601 to S602.

[0147] S601, The first device obtains the parallelism value of each sub-model training task.

[0148] The parallelism value indicates the number of computation cards mapped to the sub-model training task.

[0149] Optionally, the first device can obtain the parallelism value of each sub-model training task through user configuration. Specifically, the implementation process of S601 above can be as follows: the first device can respond to user input operations, obtain configuration information input by the user, and obtain the parallelism value of each sub-model training task from the configuration information. Of course, the above is only an exemplary description of the implementation process of S601 above, and there may be other implementation processes for S601 above, which are not limited in this application embodiment.

[0150] As described above regarding "parallel training strategies," these strategies can include data-parallel training strategies and / or model-parallel training strategies. Model-parallel training strategies can include pipelined parallel training strategies and / or tensor parallel training strategies. There is a correlation between parallel training strategies and parallelism values; that is, different types of parallel training strategies result in different parallelism values ​​for the sub-model training tasks. The following examples illustrate the relationship between the type of parallel training strategy and the parallelism value of the sub-model training task.

[0151] In one example, if the parallel training strategy corresponding to the model training task is a 3D parallel training strategy, that is, a parallel training strategy combining data parallel training strategy, pipeline parallel training strategy and tensor parallel training strategy, then the parallelism value recorded in the embodiments of this application may include DP, TP and PP, that is, the parallelism value of each sub-model training task is any one of the above DP, TP and PP.

[0152] Furthermore, in this example, the at least one sub-model training task described in this application embodiment may include: at least one DP sub-model training task, at least one TP sub-model training task, and at least one PP sub-model training task. Wherein, the parallelism value of each DP sub-model training task in the at least one DP sub-model training task is DP, the parallelism value of each TP sub-model training task in the at least one TP sub-model training task is TP, and the parallelism value of each PP sub-model training task in the at least one PP sub-model training task is PP.

[0153] In another example, if the parallel training strategy corresponding to the model training task is a combination of data parallel training strategy and tensor parallel training strategy, then the parallelism value involved in the embodiments of this application may include DP and TP, that is, the parallelism value of each sub-model training task is the above-mentioned DP or TP.

[0154] Furthermore, in this example, the at least one sub-model training task described in this application embodiment may include: at least one DP sub-model training task and at least one TP sub-model training task. Wherein, the parallelism value of each DP sub-model training task in the at least one DP sub-model training task is DP, and the parallelism value of each TP sub-model training task in the at least one TP sub-model training task is TP.

[0155] In another example, if the parallel training strategy corresponding to the model training task is a pipelined parallel training strategy, then the parallelism value involved in the embodiments of this application may include PP, that is, the parallelism value of each sub-model training task is PP.

[0156] Furthermore, in this example, the at least one sub-model training task described in this application embodiment may include at least one PP sub-model training task. The parallelism value of each PP sub-model training task in the at least one PP sub-model training task is PP.

[0157] In addition to data parallel training strategies, pipeline parallel training strategies, and tensor parallel training strategies, the parallel training strategies described in the embodiments of this application may also include other parallel training strategies, or be combined with other parallel training strategies, such as 4D parallel training strategies, etc. The embodiments of this application do not impose any limitations on this.

[0158] S602, The first device determines the predicted communication volume of each sub-model training task based on the parallelism value corresponding to each sub-model training task.

[0159] Based on the above example, if the parallel training strategy corresponding to the model training task is a 3D parallel training strategy, at least one sub-model training task may include: at least one DP sub-model training task, at least one TP sub-model training task, and at least one PP sub-model training task.

[0160] In this application, the predicted communication volume of the TP sub-model training task is greater than that of the DP sub-model training task, and the predicted communication volume of the DP sub-model training task is greater than that of the PP sub-model training task.

[0161] As described above regarding "sub-model training tasks," the TP sub-model training task is obtained by partitioning the model into different parts of the same level, the PP sub-model training task is obtained by partitioning the model into different parts of different levels, and the DP sub-model training task does not partition the model but rather the training dataset. In other words, the TP sub-model training task has the smallest partition granularity, followed by the PP sub-model training task, and the DP sub-model training task has the largest partition granularity. The smaller the partition granularity of a sub-model training task, the larger the amount of data that needs to be transmitted for the mapping computation. Therefore, the prediction communication volume of the TP sub-model training task is greater than that of the DP sub-model training task, and the prediction communication volume of the DP sub-model training task is greater than that of the PP sub-model training task.

[0162] Understandably, since the parallelism value can be used to limit at least one of the following: the number of sub-training datasets, the number of sub-model training tasks obtained from model partitioning at different levels, the number of sub-model training tasks obtained from model partitioning at the same level, and the number of computing cards mapped to a sub-model training task, that is, the parallelism value can be related to multiple parameters related to the sub-model training task. Therefore, the predicted communication volume of each sub-model training task determined based on the parallelism value corresponding to each sub-model training task can more accurately reflect the communication situation of each sub-model training task, providing a more accurate data foundation for determining grouping information based on the predicted communication volume of each sub-model training task.

[0163] The following uses any one of the sub-model training tasks (such as the second sub-model training task) in the set of sub-model training tasks as an example to illustrate the process of determining the predicted communication volume of the second sub-model training task: The first device determines the predicted traffic and the predicted number of communications for the second sub-model training task based on its parallelism value, and determines the predicted communication volume for each sub-model training task based on its corresponding parallelism value. The predicted traffic is the sum of the data volume transmitted in a single transmission between every two computing cards mapped to the second sub-model training task.

[0164] In one example, the first device can multiply the predicted traffic of the second sub-model training task and the predicted number of communications of the second sub-model training task to obtain the predicted communication volume of the second sub-model training task. Alternatively, the first device can multiply the predicted traffic of the second sub-model training task and the predicted number of communications of the second sub-model training task to obtain an initial calculation result, and then perform weighted processing on the initial calculation result to obtain the predicted communication volume of the second sub-model training task.

[0165] In another example, bitwise operations are performed on the predicted traffic of the second sub-model training task and the predicted number of communications of the second sub-model training task to obtain the operation result, and the above operation result is determined as the predicted communication volume of the second sub-model training task.

[0166] Of course, the above-described implementation process for calculating the predicted communication volume of the second sub-model training task is an illustrative example, and other implementation processes may also exist. For example, the first device determines the predicted traffic volume of the second sub-model training task and the predicted number of communications of the second sub-model training task based on the parallelism value of the second sub-model training task, and determines the ratio of the predicted traffic volume of the second sub-model training task and the predicted number of communications of the second sub-model training task as the predicted communication volume of the second sub-model training task. This application embodiment does not impose any limitations on this.

[0167] Optionally, the process by which the controller determines the predicted throughput and predicted communication count of the second sub-model training task based on the parallelism value of the second sub-model training task can be as follows: The controller can partition the training dataset or model based on the parallelism value of the second sub-model training task to obtain at least one sub-training task, and determine the amount of data that the computing card needs to generate and the number of data transmissions required to execute the second sub-training task based on the amount of data in the training dataset or the amount of data in the model. The sum of the amount of data that the computing card needs to generate to execute the second sub-training task is the predicted throughput of the second sub-model training task. The sum of the number of data transmissions required by the computing card to execute the second sub-training task is the predicted communication count of the second sub-model training task.

[0168] It is understandable that, in the process of determining the predicted communication volume of each sub-model training task based on the parallelism value corresponding to each sub-model training task, the first device not only considers the amount of data transmitted in a single transmission (i.e., the predicted traffic) but also the number of predicted communications. Compared with the prior art, which determines the predicted communication volume of each sub-model training task based on the predicted traffic of each sub-model training task, the predicted communication volume of each sub-model training task described in this application embodiment can further and more accurately reflect the communication situation of each sub-model training task, providing a more accurate data foundation for determining grouping information based on the predicted communication volume of each sub-model training task.

[0169] As described above regarding the "computation cards mapped for each sub-training task," the computation cards mapped for each sub-model training task are determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computation cards. Therefore, before mapping computation cards for each sub-model training task, the first device needs to determine the communication distance between every two computation cards. The following, combined with... Figure 7The process of determining the communication distance between every two computing cards in the first device is described in detail. For example... Figure 7 As shown, the model training method described in the embodiments of this application may further include the following S701 to S702.

[0170] S701, The first device acquires the network topology information of the training cluster.

[0171] The training cluster includes at least one switch and at least one training server. Multiple computing cards are deployed in the at least one training server. Computing cards within the same training server are directly connected, or computing cards in different training servers are connected through a switch.

[0172] Furthermore, for example, the network topology information of the training cluster may include at least one of the following: information about each training server in the at least one training server included in the training cluster, information about each switch in the at least one switch included in the training cluster, or the connection relationship between the servers and the switches. The information about the training servers may include at least one of the following: the internet protocol (IP) used to interconnect the networks of the training servers, the identifier of the training server, the model of the training server, or the number of computing cards deployed in the training server. The information about the switches may include the identifier of the switches and / or the model of the switches.

[0173] Of course, the above is only an exemplary description of the network topology information of the training cluster, the information of the training server, and the information of the switch. The network topology information of the training cluster, the information of the training server, and the information of the switch may also include other information, and this application embodiment does not impose any limitations on this.

[0174] Optionally, the network topology information of the training cluster can be included in the model training task information and configured for the first device, so that the first device can obtain the network topology information of the training cluster related to the model training task from the model training task information. To reduce signaling overhead, the network topology information of the training cluster can be included in the training cluster model training task information initially configured / first-time configured for the training cluster. After obtaining the network topology information of the training cluster from the model training task information, the first device can store the training cluster and its network topology information accordingly. Subsequently, if the user needs the training cluster to perform a model training task again, the configured model training task information can indicate the model training task without indicating the network topology information of the training cluster. In this way, the amount of information in the model training task information is reduced, thereby reducing communication overhead.

[0175] S702, The first device uses the network topology information of the training cluster to calculate the communication distance between every two computing cards in multiple computing cards.

[0176] Optionally, the first device may also count the communication distance between every two computing cards in a portion of the multiple computing cards. These computing cards may be specified by the first device according to the actual needs of the model training task, or they may be any part of the multiple computing cards. This application embodiment does not impose any restrictions on this.

[0177] It is understandable that if multiple computing cards are directly connected, it means that the computing cards do not need to be connected through a switch, and the communication distance between the computing cards can be determined as 0. If multiple computing cards are connected through a switch, it means that the computing cards need to be connected through a switch, and the number of switches through which the data transmitted between the computing cards passes can be used to determine the communication distance between the computing cards.

[0178] For example, if computing card 1 can transmit data to computing card 2 via switch 3-switch 1-switch 4, the communication distance between computing card 1 and computing card 2 is 3. If computing card 3 can transmit data to computing card 4 via switch 4-switch 2, the communication distance between computing card 3 and computing card 4 is 2.

[0179] It is understood that the communication distance between the two computing cards described in the embodiments of this application is used to characterize the number of switches through which data is transmitted between the two computing cards, and does not simply consider whether they are located in the same training server. If only whether they are located in the same training server is considered, then the communication distance between the two computing cards is 0 when they are in the same training server, and 1 when they are not in the same training server. However, the communication distance between the two computing cards described in this application is not limited to 0 or 1, but is the actual number of switches through which data is transmitted between the two computing cards. In other words, the communication distance between the two computing cards described in this application is a more accurate value.

[0180] As described above regarding the "computation cards mapped for each sub-training task," the computation cards mapped for each sub-model training task are determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computation cards. The following details the implementation process of the first device mapping computation cards for each sub-model training task. Figure 8 As shown, the model training method described in the embodiments of this application may further include the following S801 to S802.

[0181] S801, the first device takes the sub-model training task with the largest predicted communication volume in at least one sub-model training task as the starting sub-model training task, and performs the following mapping process (i.e., S801A to S802B) for each sub-model training task in at least one sub-model training task:

[0182] S801A and the first device perform clustering processing on multiple computing cards based on the parallelism value of the sub-model training task and the communication distance between every two computing cards, to obtain the clustering results corresponding to the sub-model training task.

[0183] The clustering results include at least one cluster, and each cluster includes M computing cards. The parallelism value is used to indicate the number of computing cards M mapped to the sub-model training task.

[0184] Optionally, the implementation process of the above S801A can be as follows: the first device can input the parallelism value of the sub-model training task and the communication distance between each pair of computing cards in the multiple computing cards into the greedy algorithm model to obtain the model output result, that is, the clustering result.

[0185] Alternatively, the implementation process of the above S801A can be as follows: the first device can input the parallelism value of the sub-model training task and the communication distance between each pair of computing cards in the multiple computing cards into the clustering model based on the K-means algorithm to obtain the model output result, that is, the clustering result.

[0186] Of course, the above is only an exemplary description of the implementation process of S801A. Other implementation processes may also exist for S801A, and this application embodiment does not impose any restrictions on them.

[0187] S801B: The first device selects a cluster from the clustering results corresponding to the sub-model training task as the computing card for the sub-model training task mapping.

[0188] It is understandable that the first device can be based on the above. Figure 8The method described involves multiple mapping processes until each sub-model training task is mapped to a computing card. This ensures that no sub-model training task remains unmapped. Computing cards are mapped sequentially for each sub-model training task, prioritizing those with the highest prediction communication volume to guarantee their communication performance. Furthermore, the first device can cluster multiple computing cards based on the parallelism value M of the sub-model training task and the communication distance between any two computing cards, selecting suitable M computing cards for each sub-training task. Since the clustering results determined by the clustering algorithm accurately meet the requirements related to parallelism value and communication distance, the computing cards mapped to the sub-model training tasks based on these clustering results also accurately meet these requirements, further enhancing the communication performance of the computing cards mapped to the sub-model training tasks.

[0189] As described above regarding "S801B", the first device selects a cluster from the clustering results corresponding to the sub-model training task as the computation card for the sub-model training task mapping. Figure 9 As shown, S801B can be replaced by S901 to S902.

[0190] S901, The first device selects the first cluster with the smallest sum of communication distances from at least one cluster.

[0191] The sum of communication distances is the sum of the communication distances between any two computing cards in the cluster.

[0192] S902, The first device uses the computing cards included in the first cluster as computing cards for sub-model training task mapping.

[0193] Understandably, the first device can use the computing cards in the first cluster with the smallest sum of communication distances as the computing cards for the sub-model training task mapping. This ensures that the computing cards in the first cluster with the smallest sum of communication distances in the clustering results of each sub-model training task mapping are minimized, reducing the sum of communication distances between any two computing cards in the sub-model training task mapping. Since a larger communication distance between computing cards increases the probability of hash collisions, reducing the sum of communication distances between any two computing cards in the sub-model training task mapping reduces the probability of hash collisions.

[0194] As described above regarding "S801", the first device takes the sub-model training task with the largest predicted communication volume among at least one sub-model training tasks as the starting sub-model training task, and traverses each sub-model training task among at least one sub-model training tasks. Further, optionally, the implementation process of S801 may also include: the first device sorting the at least one sub-model training tasks according to the order of predicted communication volume from largest to smallest, obtaining a sub-model training task sequence, and traversing each sub-model training task among at least one sub-model training tasks based on the sub-model training task sequence.

[0195] For example, taking a sub-model training task set that includes sub-model training task 1, sub-model training task 2, and sub-model training task 3 as an example: assuming that the predicted communication volume of sub-model training task 1 is greater than that of sub-model training task 2, and the predicted communication volume of sub-model training task 2 is greater than that of sub-model training task 3, then the first device can be a mapping computing card for sub-model training task 1, sub-model training task 2, and sub-model training task 3 in sequence.

[0196] Understandably, the first device maps computation cards to sub-model training tasks in descending order of predicted communication volume. This allows the first device to prioritize mapping computation cards to sub-model training tasks with higher predicted communication volume, and to prioritize mapping computation cards with shorter communication distances to sub-model training tasks with higher predicted communication volume. This ensures that the distance between the computation cards mapped to sub-model training tasks with higher predicted communication volume is as small as possible, thereby making the network affinity of the determined group information stronger, so as to minimize the probability of hash collisions in the future.

[0197] Understandably, if in Figure 8 and Figure 9Based on the model training method shown, and by superimposing the technical solution of "the first device sorts at least one sub-model training task in descending order of predicted communication volume to obtain a sub-model training task sequence, and traverses each sub-model training task in at least one sub-model training task based on the sub-model training task sequence", the following mapping process can be obtained: The first device takes the sub-model training task with the largest predicted communication volume in at least one sub-model training task as the starting sub-model training task, sorts at least one sub-model training task in descending order of predicted communication volume to obtain a sub-model training task sequence, and traverses each sub-model training task in at least one sub-model training task based on the sub-model training task sequence to perform the following mapping process: The first device performs clustering processing on multiple computing cards based on the parallelism value of the sub-model training task and the communication distance between every two computing cards in multiple computing cards, and obtains the clustering result corresponding to the sub-model training task. Taking a clustering result as an example, the implementation process of determining the computing card for sub-model training task mapping is explained: The first device selects the first cluster with the smallest sum of communication distance from at least one cluster, and uses the computing cards included in the first cluster as the computing cards for sub-model training task mapping.

[0198] In other words, the first device can perform multiple clustering processes in descending order of predicted communication volume, and map at least one computing card (i.e., the computing card included in the first cluster) with the smallest sum of communication distances to the sub-model training task during each clustering process, until each sub-model training task is mapped to a computing card. This ensures that during each clustering, at least one computing card (i.e., the computing card included in the first cluster) with the smallest sum of communication distances is mapped to the sub-model training task with the largest current predicted communication volume.

[0199] The greater the predicted communication volume of the sub-model training task, the longer the switch's processing time. Consequently, if the computing card mapped to this sub-model training task experiences a hash collision with other computing cards, network congestion becomes more severe, resulting in longer network congestion times. Conversely, the smaller the distance between two computing cards, the lower the probability of hash collisions between the computing card mapped to the sub-model training task and other computing cards. Therefore, mapping the sub-model training task with the largest predicted communication volume to multiple computing cards with the smallest current communication distance allows the card with the greatest loss due to hash collisions to be mapped to at least one computing card with a lower probability of hash collisions. This minimizes the losses caused by hash collisions and improves communication efficiency.

[0200] As described above regarding the "predicted communication volume of each sub-model training task," generally, if at least one sub-model training task can include at least one DP sub-model training task, at least one TP sub-model training task, and at least one PP sub-model training task, then the predicted communication volume of the TP sub-model training task is greater than that of the DP sub-model training task, and the predicted communication volume of the DP sub-model training task is greater than that of the PP sub-model training task. Furthermore, the communication distance between the computing cards mapped to the TP sub-model training task is less than that between the computing cards mapped to the DP sub-model training task, and the communication distance between the computing cards mapped to the DP sub-model training task is less than that between the computing cards mapped to the PP sub-model training task.

[0201] In other words, the first device can map directly connected computing cards for the TP sub-model training task in the computing card set. If there are not enough computing cards that are directly connected, then computing cards with shorter communication distances can be mapped for the DP sub-model training task in the computing card set that are connected through a switch.

[0202] After mapping the computing card for the TP sub-model training task, the first device can map the DP sub-model training task to the computing card using a direct connection among the remaining computing cards. If there are not enough computing cards using a direct connection, the device can map the DP sub-model training task to the computing card with the shortest possible communication distance among the computing cards connected via a switch.

[0203] After mapping the computing card for the DP sub-model training task, the first device can map the PP sub-model training task to the computing card using a direct connection among the remaining computing cards. If there are not enough computing cards using a direct connection, the device can map the PP sub-model training task to the computing card with the shortest possible communication distance among the computing cards connected via a switch.

[0204] Furthermore, as described above regarding the "communication distance between computing cards," a smaller communication distance between computing cards can be understood as a stronger network affinity between them. Therefore, it can also be understood that the network affinity between computing cards mapped to the TP sub-model training task is greater than that between computing cards mapped to the DP sub-model training task, and the network affinity between computing cards mapped to the DP sub-model training task is greater than that between computing cards mapped to the PP sub-model training task.

[0205] Understandably, the greater the communication distance between computing cards, the greater the probability of hash collisions between those two computing cards and other computing cards. For example, if the computing cards are directly connected, meaning the communication distance between them is 0, the probability of hash collisions between those two computing cards and other computing cards is minimal. If the computing cards are connected through a switch, meaning the communication distance between them is 1, the probability of hash collisions between those two computing cards and other computing cards is greater than the probability of hash collisions between two directly connected computing cards and other computing cards. If the computing cards are connected through two switches, meaning the communication distance between them is 2, the probability of hash collisions between those two computing cards and other computing cards is greater than the probability of hash collisions between two computing cards connected through a single switch and other computing cards.

[0206] Understandably, just as Figure 8 , Figure 9 As can be seen from the description of the optional implementation process of S801, the above... Figure 8 , Figure 9 The model training methods described in the optional implementation process of S801 are all applied to the case where the computation card mapped for each sub-model training task is determined based on the predicted communication volume of each sub-model training task and the communication distance between every two computation cards in the multiple computation cards.

[0207] Furthermore, the computation card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task or the communication distance between any two computation cards in a plurality of computation cards.

[0208] Furthermore, when the predicted traffic for each sub-model training task cannot be determined, the compute card mapped to each sub-model training task is determined based on the predicted traffic for each sub-model training task and the communication distance between any two compute cards. When the communication distance between any two compute cards cannot be determined, the compute card mapped to each sub-model training task is determined based on the predicted traffic for each sub-model training task and whether the compute cards are located within the same training server.

[0209] As described above regarding the "computation cards mapped for each sub-training task," the computation cards mapped for each sub-model training task are determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computation cards. Therefore, after mapping computation cards for each sub-model training task, the first device can send the computation cards mapped for each sub-model training task so that the subsequent training server can control at least one of the computation cards mapped for each sub-model training task to perform model training of the large model. Therefore, as... Figure 10As shown, the model training method described in the embodiments of this application may further include the following S1001.

[0210] S1001, The first device sends packet information.

[0211] The grouping information is used to indicate the computation card obtained by mapping the training task for each sub-model.

[0212] Optionally, the term "instruction" used in the embodiments of this application can be replaced with "reflection". Of course, "reflection" is an exemplary description of an alternative to "instruction", and the "instruction" used in the embodiments of this application can also be replaced with other descriptions, such as mapping information, and the embodiments of this application do not impose any limitations on this.

[0213] For example, the grouping information described above may include the identification information of each sub-model training task and the identification information of the computing card mapped to each sub-model training task. Of course, the above is only an exemplary description of the grouping information, and the grouping information described in this application embodiment may also include other information, such as the identification information of the corresponding training task. This application embodiment does not impose any limitations on this.

[0214] Furthermore, as described above regarding the "first device," the first device can be a controller or a training server. Therefore, there are two possible scenarios: Scenario 1, where the first device is a controller; and Scenario 2, where the first device is a training server. However, the implementation process of S1001 differs depending on the scenario. The implementation process of S1001 in these two scenarios will be explained in detail below.

[0215] Case 1 is where the first device is the controller.

[0216] In case 1, such as Figure 11 As shown in (a), the above S1001 can be replaced by the following S1001A.

[0217] S1001A: The controller sends packet information to each training server in the training cluster. Correspondingly, the training servers receive packet information from the controller.

[0218] The training cluster includes a controller for controlling the training cluster, at least one switch, and at least one training server. Multiple compute cards are deployed in the at least one training server. Compute cards within the same training server are directly connected, or compute cards in different training servers are connected via a switch.

[0219] It is understood that when the training cluster includes a controller for controlling the training cluster, the controller can execute the relevant steps of the model training method described in the embodiments of this application, and then the controller can send group information to each training server in the training cluster so that the subsequent training server can control its multiple computing cards to perform model training tasks based on the group information.

[0220] For a more detailed description of the training cluster, please refer to the descriptions in the corresponding sections above; they will not be repeated here.

[0221] Furthermore, optionally, after the training server receives the grouping information, the training server can determine the sub-model training tasks mapped by each sub-model training task indicated by the grouping information, and then control the computing cards included in the training server to execute the sub-model training tasks mapped by each other in parallel.

[0222] Case 2 is where the first device is the training server.

[0223] In case 2, such as Figure 11 As shown in (b) above, the above S1001 can be replaced by the following S1001B.

[0224] S1001B: The first training server sends packet information so that the second training server can obtain the packet information.

[0225] The training cluster includes at least one switch and at least one training server. Multiple compute cards are deployed on each training server. Compute cards within the same training server are directly connected, or compute cards in different training servers are connected via a switch. The second training server is any training server other than the first training server. The first training server is used to determine grouping information.

[0226] It is understood that, in the absence of a controller for controlling the training cluster in the above-mentioned training cluster, the training server in the training cluster (referred to as the first training server) can execute the relevant steps of the model training method described in the embodiments of this application, and then the first training server can send group information to other training servers in the training cluster (referred to as the second training server), so that each training server can control its multiple computing cards to perform model training tasks based on the group information.

[0227] Furthermore, optionally, after the second training server receives the grouping information, the second training server can determine the sub-model training tasks mapped by each sub-model training task indicated by the grouping information, and then control the computing cards included in itself to execute the sub-model training tasks mapped by each other in parallel.

[0228] Furthermore, as described above regarding the "first training server," the first training server is used to determine grouping information. However, if the training cluster includes a single training server for determining grouping information, then the first training server can be that training server. If the training cluster includes multiple training servers for determining grouping information, then the first training server can be any one of the multiple training servers, or it can be a specific training server among the multiple training servers. This application embodiment does not impose any restrictions in this regard.

[0229] Furthermore, if the first training server can be any one of the above-mentioned training servers, the first training server can be determined through the following implementation method 1:

[0230] Implementation method 1 involves determining the first training server based on the user-input training request. Specifically, the load balancer can receive the user-input training request via an interface and send it to any one of the at least one training server mentioned above. In this case, the training server that receives the user-input training request is the first training server, and the first training server determines the grouping information.

[0231] Furthermore, the first training server can be determined in the following three ways:

[0232] Implementation method 2 involves determining the first training server through a distributed coordination service. Specifically, this can be achieved by using a distributed coordination service (e.g., Apache ZooKeeper) to select a master training server from at least one training server, and then designating this master training server as the first training server.

[0233] Optionally, in implementation method 2, the implementation process of S1001B above can be as follows: the first training server can send packet information to the distributed coordination server, and the distributed coordination server stores the packet information. The second training server can obtain the packet information by reading the information stored in the distributed coordination server.

[0234] Implementation method 3 involves determining the first training server using a distributed lock mechanism of the database (e.g., a remote dictionary server, Redis). Specifically, the database can provide a distributed lock, allowing at least one training server to preempt the lock. In this case, the training server that successfully acquires the distributed lock becomes the first training server.

[0235] Optionally, in implementation method 3, the implementation process of S1001B above can be as follows: the first training server can send packet information to the distributed database or distributed cache server, and the distributed database or distributed cache server stores the packet information. The second training server can obtain the packet information by reading the information stored in the distributed database or distributed cache server.

[0236] Implementation method 4 involves determining the first training server through a distributed messaging mechanism (e.g., Apache Kafka). Specifically, this process involves sending a training server indication message to the designated training server. The designated training server then receives this message. This instruction message indicates that the specified training server is the first training server.

[0237] Optionally, in implementation method 4, the above-mentioned S1001B implementation process can be as follows: the first training server sends packet information to the second training server in the training cluster. Correspondingly, the second training server receives the packet information from the first training server.

[0238] Of course, the above is merely an exemplary description of the implementation method for determining the first training server. Other implementation methods may also exist for determining the first training server, and this application embodiment does not impose any limitations on them. Furthermore, the above is also merely an exemplary description of the implementation method for the second training server to obtain the grouping information determined by the first training server. The second training server may also obtain the grouping information determined by the first training server through other methods, and this application embodiment does not impose any limitations on them.

[0239] For example, the model training method described in the embodiments of this application can be applied to at least one scenario of cloud services, large communication models, and large question-answering models, and the embodiments of this application do not impose any limitations on this.

[0240] Further, optionally, taking the model training method described in the embodiments of this application applied to a large-scale communication model scenario as an example: the model training method described in the embodiments of this application is used to perform a model training task for a large-scale communication model. The model training task of the large-scale communication model is divided into at least one sub-model training task. The at least one sub-model training task is mapped to multiple computing cards. The computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computing cards. The predicted communication volume of the sub-model training task is used to indicate the sum of the predicted data volume transmitted between the computing cards mapped to the sub-model training task. The communication distance between two computing cards is used to characterize the number of switches through which the data transmitted between the two computing cards passes.

[0241] In this case, optionally, the implementation process of the model training method described in the embodiments of this application may include:

[0242] The first device obtains the parallelism value of each sub-model training task and determines the prediction communication volume of each sub-model training task based on the parallelism value corresponding to each sub-model training task.

[0243] The first device acquires the network topology information of the training cluster and, based on the network topology information of the training cluster, calculates the communication distance between every two computing cards in the multiple computing cards.

[0244] The first device can sort at least one sub-model training task in descending order of predicted communication volume to obtain a sub-model training task sequence. Based on the sub-model training task sequence, it can traverse each sub-model training task in at least one sub-model training task and perform the following mapping process: based on the parallelism value of the sub-model training task and the communication distance between every two computing cards, it can perform clustering processing on multiple computing cards to obtain the clustering results corresponding to the sub-model training task, and select a cluster from the clustering results corresponding to the sub-model training task as the computing card for sub-model training task mapping.

[0245] The first device sends group information so that at least one training server can control the computing card mapped to at least one sub-model training task to perform model training of a large model.

[0246] The first device acquires the training results of each sub-model training task. Based on the training results of each sub-model training task, it determines the training result of the model training task.

[0247] For a detailed description of the implementation process of the model training method, please refer to the descriptions in the corresponding sections above; they will not be repeated here.

[0248] The above mainly describes the solutions provided by the embodiments of this application from the perspective of interaction between various network elements. Correspondingly, the embodiments of this application also provide a communication device for implementing the various methods described above. This communication device can be the first device in the above method embodiments, or a device including the first device, or a component usable in the first device. It is understood that, in order to achieve the above functions, the communication device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0249] This application embodiment can divide the communication device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be understood that the module division in this application embodiment is illustrative and represents a logical functional division; in actual implementation, there may be other division methods.

[0250] Figure 12 A schematic diagram of a communication device 120 is shown. The communication device 120 includes a processing module 1201 and a transceiver module 1202, which can also be called a transceiver unit to implement the transceiver function. For example, it can be a transceiver circuit, a transceiver, a transceiver device, or a communication interface.

[0251] In one possible implementation: the communication device 120 is used to execute a model training task, which is divided into at least one sub-model training task. The at least one sub-model training task is mapped to multiple computing cards. The computing card mapped to each sub-model training task is determined based on the predicted communication volume of each sub-model training task and / or the communication distance between every two computing cards. The predicted communication volume of the sub-model training task is used to indicate the sum of the predicted data volume transmitted between the computing cards to which the sub-model training task is mapped. The communication distance between two computing cards is used to characterize the number of switches through which the data transmitted between the two computing cards passes. The transceiver module 1202 is used to acquire the training result of each sub-model training task. The processing module 1201 is used to determine the training result of the model training task based on the training result of each sub-model training task. The training result of the sub-model training task is used to indicate the execution result obtained by the computing cards mapped to the sub-model training task executing the sub-model training task in parallel.

[0252] In one possible implementation, the greater the prediction communication volume of the sub-model training task, the smaller the sum of the communication distances between any two computing cards in the computing cards mapped to the sub-model training task.

[0253] In one possible implementation, the processing module 1201 is further configured to perform the following process: taking the sub-model training task with the largest predicted communication volume in at least one sub-model training task as the starting sub-model training task, and traversing each sub-model training task in at least one sub-model training task to perform the following mapping process: based on the parallelism value of the sub-model training task and the communication distance between every two computing cards in the multiple computing cards, perform clustering processing on the multiple computing cards to obtain the clustering results corresponding to the sub-model training task, and select a cluster from the clustering results corresponding to the sub-model training task as the computing card mapped by the sub-model training task, wherein the clustering result includes at least one cluster, and one cluster includes M computing cards; the parallelism value is used to indicate the number M of computing cards mapped by the sub-model training task.

[0254] In one possible implementation, the processing module 1201 is further configured to select a first cluster with the smallest sum of communication distances from at least one cluster, and use the computing cards included in the first cluster as computing cards for the sub-model training task mapping, wherein the sum of communication distances is the sum of the communication distances between every two computing cards included in the cluster.

[0255] In one possible implementation, the processing module 1201 is further configured to sort at least one sub-model training task in descending order of predicted communication volume to obtain a sub-model training task sequence, and traverse each sub-model training task in at least one sub-model training task based on the sub-model training task sequence.

[0256] In one possible implementation, the transceiver module 1202 is further configured to obtain the parallelism value of each sub-model training task, and the processing module 1201 is further configured to determine the prediction communication volume of each sub-model training task based on the parallelism value corresponding to each sub-model training task, wherein the parallelism value is used to indicate the number of computing cards mapped to the sub-model training task.

[0257] In one possible implementation, the processing module 1201 is further configured to determine the predicted traffic and predicted communication count of the second sub-model training task based on the parallelism value of the second sub-model training task, and to determine the predicted communication volume of the second sub-model training task based on the predicted traffic and predicted communication count of the second sub-model training task, wherein the second sub-model training task is any one of the sub-model training tasks in the set of sub-model training tasks, and the predicted traffic is the sum of the data volume transmitted in a single transmission between every two computing cards in the computing card mapped to the second sub-model training task.

[0258] In one possible implementation, the transceiver module 1202 is further configured to acquire network topology information of the training cluster, and the processing module 1201 is further configured to calculate the communication distance between every two computing cards in the multiple computing cards based on the network topology information. The training cluster includes at least one switch and at least one training server. Multiple computing cards are deployed in the at least one training server. The computing cards in the same training server are directly connected, or the computing cards in different training servers are connected through a switch.

[0259] In one possible implementation, the transceiver module 1202 is also used to send group information, which is used to indicate the computation card mapped to each sub-model training task.

[0260] In one possible implementation, the transceiver module 1202 is further configured to send packet information to each training server in the training cluster. The training cluster includes a controller for controlling the training cluster, at least one switch, and at least one training server. Multiple computing cards are deployed in the at least one training server, wherein computing cards within the same training server are directly connected, or computing cards within different training servers are connected through a switch.

[0261] In one possible implementation, the transceiver module 1202 is further configured to send packet information so that the second training server can obtain the packet information. The training cluster includes at least one switch and at least one training server. Multiple computing cards are deployed in the at least one training server. The computing cards in the same training server are directly connected, or the computing cards in different training servers are connected through a switch. The second training server is a training server other than the first training server in the at least one training server. The first training server is used to determine the packet information.

[0262] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0263] In this embodiment, the first device is presented as an integrated unit divided into functional modules. Here, "module" can refer to a specific ASIC, circuitry, a processor and memory executing one or more software or firmware programs, integrated logic circuitry, and / or other devices that can provide the aforementioned functions. In a simplified embodiment, those skilled in the art will recognize that the first device can employ... Figure 4 The communication device 410 shown is in the form of [example device].

[0264] for example, Figure 4 The processor 411 in the communication device 410 shown can call the computer execution instructions stored in the memory 412 to cause the communication device 410 to execute the model training method in the above method embodiment.

[0265] Specifically, Figure 12 The functions / implementation process of the transceiver module 1202 and the processing module 1201 can be obtained through... Figure 4 The processor 411 in the communication device 410 shown calls computer execution instructions stored in the memory 412 to implement the communication. Alternatively, Figure 12 The function / implementation process of the processing module 1201 can be achieved through... Figure 4 The processor 411 in the communication device 410 shown calls computer execution instructions stored in the memory 412 to implement the communication. Figure 12 The function / implementation process of the transceiver module 1202 can be obtained through Figure 4 This is achieved through the transceiver 415 in the communication device 410 shown.

[0266] Since the communication device 120 provided in this application embodiment can execute the above-described model training method, the technical effects it can achieve can be referred to the above-described method embodiment, and will not be repeated here.

[0267] In one possible implementation, this application embodiment also provides a communication device (e.g., the communication device may be a chip or a chip system), which includes a processor for implementing the methods in any of the above method embodiments. In one possible design, the communication device further includes a memory. The memory is used to store necessary program instructions and data, and the processor can call the program code stored in the memory to instruct the communication device to execute the methods in any of the above method embodiments. Of course, the memory may not be included in the communication device. When the communication device is a chip system, it may be composed of chips or may include chips and other discrete devices; this application embodiment does not specifically limit this.

[0268] In one possible implementation, this application also provides a computer-readable storage medium storing a computer program or instructions that, when run on a communication device, enable the communication device to execute the methods of any of the above-described method embodiments or any implementation thereof.

[0269] In one possible implementation, this application embodiment also provides a model training method, which includes the method of any of the above-described method embodiments or any implementation thereof.

[0270] In one possible implementation, this application embodiment also provides a communication system, which includes the first device described in the above method embodiment.

[0271] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0272] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0273] Although this application has been described in conjunction with specific features and embodiments, it is apparent that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are exemplary illustrations of this application as defined by the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A model training method, characterized in that, The model training method is used to perform a model training task, the model training task is divided into at least one sub-model training task, and the at least one sub-model training task maps a plurality of computing cards, and the computing cards mapped by each sub-model training task are determined based on predicted communication of the each sub-model training task and / or communication distance between each two computing cards in the plurality of computing cards; the predicted communication of the sub-model training task is used to indicate the sum of the amount of data transmitted between the computing cards mapped by the sub-model training task; and the communication distance between two computing cards is used to represent the number of switches through which data transmitted between the two computing cards passes; and the method comprises: obtaining a training result of each sub-model training task, the training result of the sub-model training task being used to indicate an execution result obtained by the computing cards mapped by the sub-model training task performing the sub-model training task in parallel; determining a training result of the model training task according to the training result of each sub-model training task.

2. The method of claim 1, wherein, The greater the predicted communication of the sub-model training task is, the smaller the sum of the communication distance between each two computing cards in the computing cards mapped by the sub-model training task is.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: taking a sub-model training task with the greatest predicted communication in the at least one sub-model training task as a starting sub-model training task, and traversing each sub-model training task in the at least one sub-model training task to perform the following mapping process: performing clustering processing on the plurality of computing cards based on a parallelism value of the sub-model training task and the communication distance between each two computing cards in the plurality of computing cards, to obtain a clustering result corresponding to the sub-model training task, the clustering result comprising at least one clustering cluster, and one clustering cluster comprising M computing cards; and the parallelism value being used to indicate the number M of computing cards mapped by the sub-model training task; selecting one clustering cluster from the clustering result corresponding to the sub-model training task as the computing cards mapped by the sub-model training task.

4. The method of claim 3, wherein, The selecting one clustering cluster from the clustering result corresponding to the sub-model training task as the computing cards mapped by the sub-model training task comprises: from the at least one clustering cluster, selecting a first clustering cluster with the smallest sum of communication distances, the sum of communication distances being the sum of the communication distances between each two computing cards included in the clustering cluster; taking the computing cards included in the first clustering cluster as the computing cards mapped by the sub-model training task.

5. The method according to claim 3 or 4, characterized in that, The taking the sub-model training task with the greatest predicted communication in the at least one sub-model training task as the starting sub-model training task and traversing each sub-model training task in the at least one sub-model training task comprises: sequencing the at least one sub-model training task in descending order of the predicted communication, to obtain a sub-model training task sequence; traversing each sub-model training task in the at least one sub-model training task based on the sub-model training task sequence.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: obtaining parallelism values of each of the sub-model training tasks, the parallelism values being used to indicate the number of computing cards mapped by the sub-model training tasks; determining predicted traffic of each of the sub-model training tasks based on the parallelism value corresponding to the sub-model training task.

7. The method of claim 6, wherein, The determining predicted traffic of each of the sub-model training tasks based on the parallelism value corresponding to the sub-model training task comprises: determining predicted traffic of a second sub-model training task and predicted communication times of the second sub-model training task based on the parallelism value of the second sub-model training task, the second sub-model training task being any one of the sub-model training tasks, the predicted traffic being a sum of data amounts transmitted between each two of the computing cards mapped by the second sub-model training task in a single time; determining predicted traffic of the second sub-model training task based on the predicted traffic of the second sub-model training task and the predicted communication times of the second sub-model training task.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: obtaining network topology information of a training cluster, the training cluster comprising at least one switch and at least one training server, the at least one training server being configured to deploy the plurality of computing cards, wherein the computing cards in a same training server are directly connected, or the computing cards in different training servers are connected through the switch; based on the network topology information, counting communication distances between each two of the plurality of computing cards.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: sending grouping information, the grouping information being used to indicate the computing cards mapped by each of the sub-model training tasks.

10. The method of claim 9, wherein, The sending the grouping information comprises: sending the grouping information to each of the training servers in a training cluster, the training cluster comprising a controller used to control the training cluster, at least one switch, and at least one training server, the at least one training server being configured to deploy the plurality of computing cards, wherein the computing cards in a same training server are directly connected, or the computing cards in different training servers are connected through the switch.

11. The method of claim 9, wherein, The sending the grouping information comprises: sending the grouping information, so that a second training server obtains the grouping information, the training cluster comprising at least one switch and at least one training server, the at least one training server being configured to deploy the plurality of computing cards, wherein the computing cards in a same training server are directly connected, or the computing cards in different training servers are connected through the switch, wherein the second training server is a training server other than a first training server in the at least one training server, and the first training server is used to determine the grouping information.

12. A training cluster, characterized in that, The training cluster is used to indicate a model training method, and the training cluster includes at least one switch and at least one training server, or the training cluster includes a controller for controlling the training cluster, at least one switch, and at least one training server, and a plurality of computing cards are deployed in the at least one training server; the plurality of computing cards are mapped with at least one sub-model training task, the at least one sub-model training task is obtained by dividing a model training task, and a computing card mapped by each sub-model training task is determined based on predicted traffic of each sub-model training task and a communication distance between each two computing cards in the plurality of computing cards; the predicted traffic of the sub-model training task is used to indicate a sum of data transmitted between computing cards to which the sub-model training task is mapped, and the communication distance of two computing cards is used to represent a number of switches through which data transmitted between the two computing cards passes; The computing card mapped by each sub-model training task is used to execute the corresponding sub-model training task in parallel, and a training result is obtained; The controller or any training server is used to obtain the training result of each sub-model training task, and determine a training result of the model training task according to the training result of each sub-model training task.

13. A communications device, characterized by Comprise: A functional unit for executing the method according to any one of claims 1-11; wherein the actions performed by the functional unit are implemented by hardware or corresponding software executed by hardware.

14. A communications device, characterized by The communication device comprises a processor; the processor is used to run computer programs or instructions, or is used to pass through a logic circuit, so that the communication device executes the method according to any one of claims 1-11.

15. A computer readable storage medium characterized by: The computer readable storage medium stores computer instructions or programs, when the computer instructions or programs run on the computer, so as to make the communication device execute the method according to any one of claims 1-11.

16. A computer program product comprising instructions, characterized in that, When it runs on the communication device, it makes the communication device implement the method according to any one of claims 1-11.

Images