Data processing method, device, equipment and readable storage medium

Abstract

The present application discloses a data processing method, device, equipment and readable storage medium, the method comprising: obtaining M initial chipsets; based on the learning parameter amount corresponding to a single network layer in the initial model, grouping the chips in each initial chipset to obtain P pipeline parallel groups; based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficient corresponding to the P pipeline parallel groups, grouping and combining the P pipeline parallel groups to obtain D data parallel groups and the sample input amount corresponding to the D data parallel groups; the input samples indicated by the D sample input amount are used to perform model training on the initial model configured on the corresponding data parallel group in parallel. The present application can improve the utilization rate of computing resources of a hybrid deployed chip cluster and improve the training efficiency of the model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment and readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, the parameter scale of large language models is growing, ranging from millions to trillions. In existing model training, due to the large scale of model parameters, huge computing resources and computing time requirements, chip clusters are usually required for model training. Existing technologies generally use homogeneous chips for data parallelism or pipeline parallelism for distributed training. Data parallelism refers to configuring the same initial model to multiple data parallel groups, and then each data parallel group independently processes its own input samples, and finally aggregates and integrates all model parameters. Pipeline parallelism refers to distributing the model's network layer to multiple pipeline parallel groups for completion, and each pipeline parallel group is responsible for processing a part of the model's network layer.

[0003] Since it is difficult to obtain a sufficient number of homogeneous chips, using a single type of chip to build a large-scale Wanka cluster faces huge challenges. When using mixed deployment of different types of chips, the same parallel strategy as homogeneous chips is adopted. Due to the large differences in the computing power (comprehensive computing power such as computing power, storage resources, and communication resources) of different chips, the chip load will be uneven, the chips with stronger computing power will not be effectively utilized, and the chips with weaker computing power will seriously delay the training performance of the chip cluster, resulting in low computing power utilization of the chip cluster and low training efficiency. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and readable storage medium, which can improve the utilization rate of computing resources of a hybrid deployed chip cluster and improve the training efficiency of the model.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Obtain M initial chipsets, where M is a positive integer, the chip models in an initial chipset are all the same, and the chip models between each initial chipset are different;

[0007] Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, the chips in each initial chipset are grouped to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount;

[0008] Based on the load balancing condition, the average computing power coefficient of P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training in parallel on the initial model configured on the corresponding data parallel groups.

[0009] Among them, P pipeline parallel groups include pipeline parallel groups , pipeline parallel group It includes S chips, where S is a positive integer; the pipeline parallel group Each chip in is used to train the network layer of the initial model through tensor parallelism, and the number of parameters of the network layer allocated to the S chips is equal.

[0010] Among them, based on the learning parameter quantity corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, the chips in each initial chipset are grouped to obtain P pipeline parallel groups, including:

[0011] Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, initial grouping parameters corresponding to the M initial chipsets are generated; the M initial grouping parameters are all greater than or equal to 2;

[0012] Based on the M initial grouping parameters, the chips in each initial chipset are grouped separately to obtain N initial parallel groups. Based on the average computing power coefficients of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped separately to obtain P pipeline parallel groups; N is a positive integer, N is greater than M, and N is less than or equal to P.

[0013] The M initial chipsets include the initial chipset , initial chipset Includes S target chips, initial chipset The corresponding grouping parameter is G, where G, S and i are all positive integers. Based on the M initial grouping parameters, the chips in each initial chipset are grouped to obtain N initial parallel groups, including:

[0014] Based on initial chipset The grouping parameter G for the initial chipset The chips in the group are grouped to obtain the initial chipset Corresponding T target parallel groups; initial chipset Each corresponding target parallel group includes G target model chips, and the product of T and G is S;

[0015] When the target parallel groups corresponding to the M initial chip groups are obtained, the target parallel groups corresponding to the M initial chip groups are determined as N initial parallel groups; the N initial parallel groups include T target parallel groups.

[0016] Among them, based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped respectively to obtain P pipeline parallel groups, including:

[0017] Obtain the average computing power coefficient of the N initial parallel groups, determine the initial parallel groups whose group computing power coefficient is greater than the average computing power coefficient of the N initial parallel groups as Q groups to be split, and split the chips in the Q groups to be split based on the learning parameter amount to obtain a set of split groups corresponding to each group to be split; Q is a positive integer, and the computing memory corresponding to the split groups in each set of split groups is greater than or equal to the learning parameter amount;

[0018] A pending parallel group sets are generated based on the split group set and NQ initial parallel groups excluding the Q groups to be split, the communication evaluation scores corresponding to the A pending parallel group sets are obtained, and the pending parallel group set with the maximum communication evaluation score is determined as P pipeline parallel groups; A is a positive integer.

[0019] Among them, A pending parallel group sets include the pending parallel group sets , pending parallel group set Including L pending parallel groups consisting of the split groups and the initial parallel groups, i and L are positive integers; obtaining the communication evaluation scores corresponding to the A pending parallel group sets respectively, including:

[0020] Get pending parallel group collection The communication parameters corresponding to the chips in the L pending parallel groups and the group computing power coefficients corresponding to the L pending parallel groups;

[0021] The chips whose communication parameters are greater than the average value of the communication parameters in the L pending parallel groups are determined as the first chip, and the chips whose communication parameters are less than or equal to the average value of the communication parameters in the L pending parallel groups are determined as the second chip; the average value of the communication parameters is determined based on the communication parameters of the chip models of the M initial chip groups;

[0022] Obtain a first evaluation parameter and a second evaluation parameter, generate a pipeline parallel evaluation score based on the first evaluation parameter and the group computing power coefficient of the pending parallel group including the first chip, generate a data parallel evaluation score based on the second evaluation parameter and the group computing power coefficient of the pending parallel group including the second chip, and determine the sum of the pipeline parallel evaluation score and the data parallel evaluation score as the pending parallel group set Communication assessment score.

[0023] Among them, the P pipeline parallel groups include pipeline parallel groups , pipeline parallel group The method includes S target chips, where S is a positive integer. The method further includes:

[0024] Obtain the business computing power parameters, storage parameters, bandwidth parameters, and communication parameters of the target chip model;

[0025] Pipeline-based parallel group The number of cards S and communication parameters generate the intra-group communication coefficient, based on the business computing power parameters, storage parameters, bandwidth parameters and pipeline parallel group of the target model chip The number of cards S generates the business computing power coefficient, and the sum of the intra-group communication coefficient and the business computing power coefficient is determined as the pipeline parallel group The group computing power coefficient.

[0026] Among them, based on the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups, including:

[0027] Obtain an initial micro-batch value of the initial model, and generate a target task duration for an average computing power coefficient of P pipeline parallel groups based on the initial micro-batch value; the initial micro-batch value is determined based on the task domain of the initial model;

[0028] Based on the initial micro-batch value and the total number of samples, generate data parallel grouping parameters, and based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, generate D selection coefficient lists and the initial sample input amounts corresponding to the D allocation parallel groups; each selection coefficient list includes the selection coefficients corresponding to the P pipeline parallel groups, and the selection coefficients of the pipeline parallel groups are used to indicate the selection mapping relationship between the pipeline parallel groups and the allocation parallel groups; D is the data parallel grouping parameter;

[0029] According to the selection coefficients in the selection coefficient list, the P pipeline parallel groups are assigned to corresponding allocation parallel groups to obtain D transition parallel groups including the pipeline parallel groups;

[0030] Based on the load balancing conditions, the target task duration, D transition parallel groups, D initial sample input amounts, and the group computing power coefficients corresponding to the P pipeline parallel groups, the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amounts corresponding to the D transition parallel groups are generated, the D transition parallel groups containing the layer number parameters are determined as D data parallel groups, and the transition sample input amounts corresponding to the D transition parallel groups are determined as the sample input amounts corresponding to the D data parallel groups.

[0031] Among them, D allocation parallel groups include allocation parallel groups , i is a positive integer; based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, a selection coefficient list and the initial sample input corresponding to the D allocation parallel groups are generated, including:

[0032] Get allocation parallel group corresponding P pending selection parameters, determining the product of the P pending selection parameters and the group computing power coefficients corresponding to the P pending selection parameters as P first load parameters, and generating second load parameters based on the P first load parameters; the P pending selection parameters correspond one-to-one to the P pipeline parallel groups respectively;

[0033] Get allocation parallel group Corresponding to the pending sample parameters, the ratio between the second load parameter and the pending sample parameters is determined as the pending task duration, and the P pending selection parameters and the pending sample parameters are adjusted to obtain H first parameter sets; H is a positive integer, and each first parameter set includes parameter values ​​of the P pending selection parameters and parameter values ​​of the pending sample parameters;

[0034] One or more first parameter sets satisfying the maximum ratio condition are determined as the second parameter set, the second parameter set containing the smallest pending sample parameter is determined as the target allocation parameter set, and the parameter values ​​of the pending sample parameters in the target allocation parameter set are determined as the allocation parallel group. The corresponding initial sample input volume is used to determine the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group The corresponding selection coefficient list; the maximum ratio condition refers to the condition that the ratio between the target task duration and the pending task duration reaches the maximum value.

[0035] Among them, D transition parallel groups include transition parallel groups , Transition Parallel Group It includes C pipeline parallel groups, i and C are both positive integers; based on the load balancing condition, the target task duration, the D transition parallel groups, the D initial sample input amounts and the group computing power coefficients corresponding to the P pipeline parallel groups, the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amounts corresponding to the D transition parallel groups are generated, including:

[0036] Generate a transition parallel group based on the group computing power coefficients corresponding to the C pipeline parallel groups The C undetermined layer number parameters are determined, and the product of the undetermined layer number parameters corresponding to each pipeline parallel group and the corresponding group computing power coefficient is determined as the third load parameter;

[0037] Transition-based parallel groups The corresponding initial sample input amount determines the target sample parameter, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter;

[0038] Based on the load balancing condition, adjust C undetermined layer number parameters and target sample parameters to obtain R third parameter sets; R is a positive integer, and each third parameter set includes parameter values ​​of C undetermined layer number parameters and parameter values ​​of target sample parameters;

[0039] One or more third parameter sets satisfying the maximum product condition are determined as the fourth parameter set, the fourth parameter set containing the minimum target sample parameter is determined as the target transition parameter set, and the parameter values ​​of the target sample parameters in the target transition parameter set are determined as the transition parallel set. The corresponding transition sample input amount is used to determine the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group The number of layers parameter corresponding to each pipeline parallel group; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches the maximum value.

[0040] Among them, it also includes:

[0041] Based on the sample input amounts corresponding to the D data parallel groups, the training sample set is divided to obtain training sample groups corresponding to each data parallel group;

[0042] Based on the training sample group, the initial model configured for each data parallel group is trained in parallel to obtain the sub-model parameters corresponding to each data parallel group, and based on the sub-model parameters corresponding to each data parallel group, a target model is generated.

[0043] An embodiment of the present application provides a data processing device, including:

[0044] The chip acquisition module is used to acquire M initial chip groups; M is a positive integer, the chip models in an initial chip group are the same, and the chip models of each initial chip group are different from each other;

[0045] The pipeline parallel grouping module is used to group the chips in each initial chipset based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount;

[0046] The data parallel grouping module is used to group and combine the P pipeline parallel groups based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, so as to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training in parallel on the initial model configured on the corresponding data parallel groups.

[0047] In one possible implementation, the P pipeline parallel groups include the pipeline parallel groups , pipeline parallel group It includes S chips, where S is a positive integer; the pipeline parallel group Each chip in is used to train the network layer of the initial model through tensor parallelism, and the number of parameters of the network layer allocated to the S chips is equal.

[0048] In a possible implementation, the pipeline parallel grouping module is used to group the chips in each initial chipset based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, and to obtain P pipeline parallel groups, and specifically to perform the following operations:

[0049] Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, initial grouping parameters corresponding to the M initial chipsets are generated; the M initial grouping parameters are all greater than or equal to 2;

[0050] Based on the M initial grouping parameters, the chips in each initial chipset are grouped separately to obtain N initial parallel groups. Based on the average computing power coefficients of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped separately to obtain P pipeline parallel groups; N is a positive integer, N is greater than M, and N is less than or equal to P.

[0051] In a possible implementation, the M initial chipsets include an initial chipset , initial chipset Includes S target chips, initial chipset The corresponding grouping parameter is G, where G, S and i are all positive integers; the pipeline parallel grouping is used to group the chips in each initial chipset based on the M initial grouping parameters to obtain N initial parallel groups, and is specifically used to perform the following operations:

[0052] Based on initial chipset The grouping parameter G for the initial chipset The chips in the group are grouped to obtain the initial chipset Corresponding T target parallel groups; initial chipset Each corresponding target parallel group includes G target model chips, and the product of T and G is S;

[0053] When the target parallel groups corresponding to the M initial chip groups are obtained, the target parallel groups corresponding to the M initial chip groups are determined as N initial parallel groups; the N initial parallel groups include T target parallel groups.

[0054] In a possible implementation, the pipeline parallel grouping module is used to group the chips in each initial parallel group based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, and when P pipeline parallel groups are obtained, specifically to perform the following operations:

[0055] Obtain the average computing power coefficient of the N initial parallel groups, determine the initial parallel groups whose group computing power coefficient is greater than the average computing power coefficient of the N initial parallel groups as Q groups to be split, and split the chips in the Q groups to be split based on the learning parameter amount to obtain a set of split groups corresponding to each group to be split; Q is a positive integer, and the computing memory corresponding to the split groups in each set of split groups is greater than or equal to the learning parameter amount;

[0056] A pending parallel group sets are generated based on the split group set and NQ initial parallel groups excluding the Q groups to be split, the communication evaluation scores corresponding to the A pending parallel group sets are obtained, and the pending parallel group set with the maximum communication evaluation score is determined as P pipeline parallel groups; A is a positive integer.

[0057] In a possible implementation, the A pending parallel group sets include the pending parallel group set , pending parallel group set Including L pending parallel groups consisting of the split groups and the initial parallel groups, i and L are positive integers; when the pipeline parallel grouping module is used to obtain the communication evaluation scores corresponding to the A pending parallel group sets, it is specifically used to perform the following operations:

[0058] Get pending parallel group collection The communication parameters corresponding to the chips in the L pending parallel groups and the group computing power coefficients corresponding to the L pending parallel groups;

[0059] The chips whose communication parameters are greater than the average value of the communication parameters in the L pending parallel groups are determined as the first chip, and the chips whose communication parameters are less than or equal to the average value of the communication parameters in the L pending parallel groups are determined as the second chip; the average value of the communication parameters is determined based on the communication parameters of the chip models of the M initial chip groups;

[0060] Obtain a first evaluation parameter and a second evaluation parameter, generate a pipeline parallel evaluation score based on the first evaluation parameter and the group computing power coefficient of the pending parallel group including the first chip, generate a data parallel evaluation score based on the second evaluation parameter and the group computing power coefficient of the pending parallel group including the second chip, and determine the sum of the pipeline parallel evaluation score and the data parallel evaluation score as the pending parallel group set Communication assessment score.

[0061] In one possible implementation, the P pipeline parallel groups include the pipeline parallel groups , pipeline parallel group The method comprises S target model chips, where S is a positive integer; and the pipeline parallel grouping module is further used to perform the following operations:

[0062] Obtain the business computing power parameters, storage parameters, bandwidth parameters, and communication parameters of the target chip model;

[0063] Pipeline-based parallel group The number of cards S and communication parameters generate the intra-group communication coefficient, based on the business computing power parameters, storage parameters, bandwidth parameters and pipeline parallel group of the target model chip The number of cards S generates the business computing power coefficient, and the sum of the intra-group communication coefficient and the business computing power coefficient is determined as the pipeline parallel group The group computing power coefficient.

[0064] In a possible implementation, the data parallel grouping module is used to group and combine the P pipeline parallel groups based on the average computing power coefficients of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups, and specifically to perform the following operations:

[0065] Obtain an initial micro-batch value of the initial model, and generate a target task duration for an average computing power coefficient of P pipeline parallel groups based on the initial micro-batch value; the initial micro-batch value is determined based on the task domain of the initial model;

[0066] Based on the initial micro-batch value and the total number of samples, generate data parallel grouping parameters, and based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, generate D selection coefficient lists and the initial sample input amounts corresponding to the D allocation parallel groups; each selection coefficient list includes the selection coefficients corresponding to the P pipeline parallel groups, and the selection coefficients of the pipeline parallel groups are used to indicate the selection mapping relationship between the pipeline parallel groups and the allocation parallel groups; D is the data parallel grouping parameter;

[0067] According to the selection coefficients in the selection coefficient list, the P pipeline parallel groups are assigned to corresponding allocation parallel groups to obtain D transition parallel groups including the pipeline parallel groups;

[0068] Based on the target task duration, D transition parallel groups, D initial sample input amounts, and the group computing power coefficients corresponding to the P pipeline parallel groups, the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amounts corresponding to the D transition parallel groups are generated. The D transition parallel groups containing the layer number parameters are determined as D data parallel groups, and the transition sample input amounts corresponding to the D transition parallel groups are determined as the sample input amounts corresponding to the D data parallel groups.

[0069] In a possible implementation, the D allocation parallel groups include the allocation parallel group , i is a positive integer; the data parallel grouping module is used to generate a selection coefficient list and an initial sample input amount corresponding to each of the D allocation parallel groups based on the target task duration and the group computing power coefficients corresponding to each of the P pipeline parallel groups, and is specifically used to perform the following operations:

[0070] Get allocation parallel group corresponding P pending selection parameters, determining the product of the P pending selection parameters and the group computing power coefficients corresponding to the P pending selection parameters as P first load parameters, and generating second load parameters based on the P first load parameters; the P pending selection parameters correspond one-to-one to the P pipeline parallel groups respectively;

[0071] Get allocation parallel group Corresponding to the pending sample parameters, the ratio between the second load parameter and the pending sample parameters is determined as the pending task duration, and the P pending selection parameters and the pending sample parameters are adjusted to obtain H first parameter sets; H is a positive integer, and each first parameter set includes parameter values ​​of the P pending selection parameters and parameter values ​​of the pending sample parameters;

[0072] One or more first parameter sets satisfying the maximum ratio condition are determined as the second parameter set, the second parameter set containing the smallest pending sample parameter is determined as the target allocation parameter set, and the parameter values ​​of the pending sample parameters in the target allocation parameter set are determined as the allocation parallel group. The corresponding initial sample input volume is used to determine the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group The corresponding selection coefficient list; the maximum ratio condition refers to the condition that the ratio between the target task duration and the pending task duration reaches the maximum value.

[0073] In a possible implementation, the D transition parallel groups include the transition parallel group , Transition Parallel Group It includes C pipeline parallel groups, i and C are both positive integers; the data parallel grouping module is used to generate the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input corresponding to the D transition parallel groups based on the load balancing condition, the target task duration, the D transition parallel groups, the D initial sample inputs and the group computing power coefficients corresponding to the P pipeline parallel groups, and specifically to perform the following operations:

[0074] Generate a transition parallel group based on the group computing power coefficients corresponding to the C pipeline parallel groups The C undetermined layer number parameters are determined, and the product of the undetermined layer number parameters corresponding to each pipeline parallel group and the corresponding group computing power coefficient is determined as the third load parameter;

[0075] Transition-based parallel groups The corresponding initial sample input amount determines the target sample parameter, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter;

[0076] Based on the load balancing condition, adjust C undetermined layer number parameters and target sample parameters to obtain R third parameter sets; R is a positive integer, and each third parameter set includes parameter values ​​of C undetermined layer number parameters and parameter values ​​of target sample parameters;

[0077] One or more third parameter sets satisfying the maximum product condition are determined as the fourth parameter set, the fourth parameter set containing the minimum target sample parameter is determined as the target transition parameter set, and the target sample parameters in the target transition parameter set are determined as the transition parallel set. The corresponding transition sample input volume determines the C undetermined layer number parameters in the target transition parameter set as the transition parallel group The number of layers parameter corresponding to each pipeline parallel group; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches the maximum value.

[0078] In a possible implementation, the data parallel grouping module is further used to perform the following operations:

[0079] Based on the sample input amounts corresponding to the D data parallel groups, the training sample set is divided to obtain training sample groups corresponding to each data parallel group;

[0080] Based on the training sample group, the initial model configured for each data parallel group is trained in parallel to obtain the sub-model parameters corresponding to each data parallel group, and based on the sub-model parameters corresponding to each data parallel group, a target model is generated.

[0081] On the one hand, an embodiment of the present application provides a computer device, including: a processor, a memory, and a network interface;

[0082] The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, and the memory is used to store a computer program. When the computer program is executed by the processor, the computer device executes the method provided in the embodiment of the present application.

[0083] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0084] In one aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in the embodiment of the present application.

[0085] The embodiment of the present application groups all chips by chip model to obtain an initial chip group with the same chip model in the group. The chip models of each initial chip group are different from each other. Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips in each initial chip group are grouped to obtain P pipeline parallel groups. The chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount. The network layer of the initial model is trained in parallel by tensors using chips of the same chip model. Based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups. Among them, the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training on the initial model configured on the corresponding data parallel group in parallel, and the load balancing condition is satisfied between the D data parallel groups. Among them, data parallelism refers to configuring the same initial model to multiple data parallel groups, and then each data parallel group independently processes its own input samples, and finally aggregates and integrates all model parameters. The embodiment of the present application determines the average computing power coefficient and group computing power coefficient of each pipeline parallel group through a pipeline parallel group composed of chips of the same chip model. Based on the idea of ​​computing power equivalence, the pipeline parallel groups are grouped and combined to obtain data parallel groups, and each data parallel group is assigned a task load equivalent to computing power. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model, the task load of the initial model is split by the data parallel group in the dimension of the sample input size, and the sample input amount corresponding to the data parallel group is determined, so that the load of each data parallel group is balanced, thereby improving the utilization rate of the computing resources of the hybrid deployed chips and improving the training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0087] Figure 1 This is a schematic diagram of a network architecture provided by an embodiment of the present application;

[0088] Figure 2 This is a data processing scenario provided by the embodiment of the present application. Figure 1 ;

[0089] Figure 3 This is a data processing scenario provided by the embodiment of the present application. Figure 2 ;

[0090] Figure 4 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 ;

[0091] Figure 5 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 ;

[0092] Figure 6 This is a data processing scenario provided by the embodiment of the present application. Figure 3 ;

[0093] Figure 7 This is a data processing scenario provided by the embodiment of the present application. Figure 4 ;

[0094] Figure 8 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0095] Fig. 9 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0097] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a business server 100 and N training nodes, and the N training nodes may include training nodes 10a, training nodes 10b, ..., training nodes 10n, wherein any training node may have a communication connection with the business server 100, for example, there is a communication connection between the training node 10a and the business server 100, and there is a communication connection between the training node 10b and the business server 100, wherein the above-mentioned communication connection does not limit the connection method, and may be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, or through other methods, and the present application does not make any restrictions here.

[0098] The training nodes may include: smart phones, tablet computers, laptops, desktop computers, intelligent voice interaction devices, smart home appliances (e.g., smart TVs), wearable devices, vehicle terminals, aircraft, and other smart terminals with data processing functions. The vehicle terminals may be devices in smart transportation scenarios and assisted driving scenarios. Figure 1 The training nodes shown in the figure can be installed with application clients with data processing functions. When the application clients run in each training node, they can be respectively connected to the above Figure 1 The business servers 100 shown perform data interaction.

[0099] The application client may specifically include: an in-vehicle client, a smart home client, an entertainment client (e.g., a game client), a multimedia client (e.g., a video client), a social client, and an information client (e.g., a news client). The application client in the embodiment of the present application may be integrated in a client (e.g., a social client) or may be an independent client (e.g., a news client). The embodiment of the present application does not limit the type of the application client.

[0100] like Figure 1 As shown, the business server 100 and N training nodes (which may include training node 10a, training node 10b, ..., training node 10n) can work together to perform distributed learning for the initial model. The business server 100 can be a server for aggregating model parameters, and the training node can be a server for model training. Each training node can be deployed with one or more chips of the same chip model (also referred to as graphics card chips, GPU (Graphics Processing Unit) chips or graphics processors), and the chip models between different training nodes can be the same or different. The chip model refers to the model of the chip, which is used to distinguish the product type and performance level of the chip. The machine model models between different training nodes can be different, and different machine models also make the chip models equipped on the training nodes different.

[0101] The business server 100 can determine a training node as a pipeline parallel group (PipelineParallel Group) for model training, that is, a pipeline parallel group can be used to train several network layers (Layers) of the initial model. For example, the pipeline parallel group 1 corresponding to the training node includes chip 1 and chip 2 of the same chip model, and is used to train two network layers of the initial model. Chip 1 and chip 2 can be used to calculate half of the learning parameter amount in the two network layers, that is, the tensor parallelism (Tensor Parallel Group Size) of the pipeline parallel group 1 is 2. Among them, tensor parallelism can also be called intra-layer parallelism, which refers to dividing the parameters in each network layer of the model, and distributing the training tasks of different parameters of the same network layer to multiple chips for completion. Pipeline parallelism can also be called inter-layer parallelism, which refers to distributing one or more network layers (usually continuous layers) of the model to multiple pipeline parallel groups for completion, and each pipeline parallel group is responsible for processing a part of the network layer of the model.

[0102] The business server 100 can group and combine the pipeline parallel groups based on the average computing power coefficient and the group computing power coefficient of the pipeline parallel groups to obtain D data parallel groups (Data Parallel Group) for model training. Its data parallelism (Data Parallel Group Size) is D. Among them, data parallelism refers to configuring the same initial model to multiple data parallel groups, and then each data parallel group uses different data batches for training. Each data parallel group independently calculates the loss function and gradient, and then aggregates the gradient through some form of communication to update the model parameters of all data parallel groups. Each data parallel group can be configured with an initial model, and the initial model deployed by each data parallel group can be the same. The group computing power coefficient is determined based on the business computing power parameters, storage parameters, bandwidth parameters and communication parameters of the chip. The average computing power coefficient is the average value of the group computing power coefficients of each pipeline parallel group.

[0103] The business server 100 can allocate different sample input amounts to each data parallel group. Each data parallel group can perform model training on the locally configured initial model based on the allocated input samples. After completing the forward propagation calculation and the backward propagation calculation, the D data parallel groups can return the calculated sub-model parameters to the business server 100. The learning parameters can include the total number of model parameters, activation value parameters, and gradient parameters. The business server 100 aggregates the sub-model parameters to obtain the aggregated model parameters, and then updates the initial models configured in each data parallel group based on the aggregated model parameters to perform the next batch of training. When the initial model in the business server meets the model convergence condition, the initial model that meets the model convergence condition is determined as the target model.

[0104] The embodiment of the present application determines the average computing power coefficient and group computing power coefficient of each pipeline parallel group through a pipeline parallel group composed of chips of the same chip model, and based on the idea of ​​computing power equivalence, groups and combines the pipeline parallel groups to obtain data parallel groups, and allocates a task load equivalent to computing power to each data parallel group. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model, the task load of the initial model is split in the dimension of the sample input size through the data parallel group, and the sample input amount corresponding to the data parallel group is determined, so that the load of each data parallel group is balanced, thereby improving the utilization rate of the computing resources of the mixed deployed chips and improving the training efficiency of the model. Each data parallel group can independently process its own input samples, perform model training in parallel on the configured initial model, and finally generate a target model based on the multiple sub-model parameters obtained by training, further improving the training efficiency of the model.

[0105] Please also see Figure 2 and Figure 3 , Figure 2 This is a data processing scenario provided by an embodiment of the present application. Figure 1 , Figure 3 This is a data processing scenario provided by an embodiment of the present application. Figure 2 .like Figure 2 As shown, the computer device may be the above Figure 1 In the business server 100 of the corresponding embodiment, the computer device can obtain M initial chipsets. Taking the M initial chipsets including initial chipset 1, initial chipset 2, initial chipset 3 and initial chipset 4 as an example, initial chipset 1 can include chips 11, 12, 13 and 14, all of which have chip models A. Initial chipset 2 can include chips 22 and 23, all of which have chip models B. Initial chipset 3 can include chips 31, 32, 33 and 34, all of which have chip models C. Initial chipset 4 can include chips 41, 42, 43, 44, 45, 46, 47 and 48, all of which have chip models D.

[0106] The computer device can obtain the learning parameter amount corresponding to a single network layer in the initial model, and the learning parameter amount can refer to the total number of model parameters, activation value parameters and gradient parameters calculated by a single network layer during training. The computer device can group the chips in each initial chipset based on the learning parameter amount and the chip models of the M initial chipsets to obtain P pipeline parallel groups, taking the P pipeline parallel groups including pipeline parallel group 1a, pipeline parallel group 1b, pipeline parallel group 2a, pipeline parallel group 3a, pipeline parallel group 3b, pipeline parallel group 3c, pipeline parallel group 3d, pipeline parallel group 4a and pipeline parallel group 4b as an example. The pipeline parallel group 1a may include chip 11 and chip 12, the pipeline parallel group 1b may include chip 13 and chip 14, the pipeline parallel group 2a may include chip 22 and chip 23, the pipeline parallel group 3a may include only chip 31, the pipeline parallel group 3b may include only chip 32, the pipeline parallel group 3c may include only chip 33, the pipeline parallel group 3d may include only chip 34, and the pipeline parallel group 4a may include chip 41, chip 42, chip 43 and chip 44. The pipeline parallel group 4b may include chip 45, chip 46, chip 47 and chip 48. Among them, the computing memory of the pipeline parallel group 1a, the pipeline parallel group 1b and the pipeline parallel group 2a is greater than or equal to four times the amount of learning parameters, that is, the pipeline parallel group 1a, the pipeline parallel group 1b and the pipeline parallel group 2a can be used for parallel training of at least four network layers in the initial model. The computing memory of pipeline parallel group 3a, pipeline parallel group 3b, pipeline parallel group 3c, pipeline parallel group 3d, pipeline parallel group 4a and pipeline parallel group 4b is greater than or equal to four times the amount of learning parameters, that is, pipeline parallel group 3a, pipeline parallel group 3b, pipeline parallel group 3c, pipeline parallel group 3d, pipeline parallel group 4a and pipeline parallel group 4b can all be used to parallel train at least four network layers in the initial model.

[0107] The computer device can obtain the average computing power coefficient of the pipeline parallel group and the group computing power coefficient corresponding to each pipeline parallel group. Based on the load balancing condition, the average computing power coefficient of the pipeline parallel group, the number of network layers of the initial model and the group computing power coefficient corresponding to each pipeline parallel group, the pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups. Taking the D data parallel groups including data parallel group 1, data parallel group 2 and data parallel group 3 as an example, the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model, that is, each data parallel group can train the initial model separately, and the sample input amount corresponding to each data parallel group can be the same or different. Grouping and combining based on the load balancing condition means that the pipeline parallel groups are grouped and combined for the purpose of making the training time of each data parallel group close to the same and matching the sample input amount corresponding to the data parallel group with the comprehensive performance of the chip in the group, so that the computing resources of the chip are effectively utilized.

[0108] Taking data parallel group 1 as an example, the computer device can divide the training sample set based on the sample input amount corresponding to data parallel group 1, the sample input amount corresponding to data parallel group 2, and the sample input amount corresponding to data parallel group 3, to obtain data shard 1 corresponding to data parallel group 1, data shard 2 corresponding to data parallel group 2, and data shard 3 corresponding to data parallel group 3. Taking data parallel group 1 as an example, data parallel group 1 can perform model training on the locally configured initial model based on the allocated data shard 1, wherein the initial model can have 8 network layers, pipeline parallel group 1a can be used to train 4 of the network layers, pipeline parallel group 3a can be used to train 2 of the network layers, and pipeline parallel group 4a can be used to train 2 of the network layers.

[0109] Understandably, Figure 3 The corresponding splitting strategies can be combined arbitrarily to meet the load balancing conditions. Specifically, corresponding to the distributed learning of the initial model, Figure 3The data parallelism in is 3, and the sample input amount and batch size (batch size) corresponding to each data parallel group can be different. Among them, the batch size refers to the amount of sample data input in each training batch. In data parallel group 1, the sample input amount of data parallel group 1 can be 1 / 4 of the training sample set, the batch size can be 2, the pipeline parallelism of data parallel group 1 is 3, the tensor parallelism of pipeline parallel group 1a is 2, the tensor parallelism of pipeline parallel group 3a is 1, and the tensor parallelism of pipeline parallel group 4a is 4. In data parallel group 2, the sample input amount of data parallel group 2 can be 1 / 4 of the training sample set, the batch size can be 2, the pipeline parallelism of data parallel group 2 is 3, the tensor parallelism of pipeline parallel group 1b is 2, the tensor parallelism of pipeline parallel group 3b is 1, and the tensor parallelism of pipeline parallel group 3d is 1. In data parallel group 3, the sample input amount of data parallel group 3 can be 1 / 2 of the training sample set, the batch size can be 4, the pipeline parallelism of data parallel group 2 is 3, the tensor parallelism of pipeline parallel group 3c is 1, the tensor parallelism of pipeline parallel group 2a is 2, and the tensor parallelism of pipeline parallel group 4b is 4.

[0110] After each data parallel group completes the forward propagation calculation and the backward propagation calculation, the data parallel group can return the calculated sub-model parameters to the computer device. The computer device aggregates the sub-model parameters to obtain the aggregated model parameters, and then updates the initial model configured in each data parallel group based on the aggregated model parameters to perform the next batch of training. When the initial model in the computer device meets the model convergence condition, the initial model that meets the model convergence condition can be determined as the target model.

[0111] The embodiment of the present application supports a flexible multi-dimensional model parallel strategy. The training platform can dynamically decompose the training task into multiple subtasks and assign them to different machine types according to the characteristics of the training task and the availability of resources. Support the configuration of different tensor parallelism in each pipeline parallel group. Based on the average computing power coefficient and group computing power coefficient of the pipeline parallel group, each pipeline parallel group can be assigned to a task load that matches its own group computing power coefficient. Different pipeline parallel groups can be assigned to different types of data parallel groups. The data parallel group can be a machine or device equipped with a chip in the pipeline parallel group. In this way, the number of model layers on the pipeline parallel group and the tensor parallelism of the chip in the pipeline parallel group can be configured according to the hardware characteristics of the machine itself on different types of machines. Based on the sample input amount of the data parallel group and the group computing power coefficient of the pipeline parallel group, different pipeline parallelism can be configured for each data parallel group. A flexible load task splitting mechanism is implemented, which provides a larger solution space for the search of parallel strategies, optimizes the allocation of training tasks, and improves the overall training performance.

[0112] See also Figure 4 , Figure 4 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The data processing method can be executed by a computer device, which can be Figure 1 The business server 100 shown in FIG. 1 is a diagram showing a data processing method executed by a computer device. The data processing method may include at least the following steps S101 to S103:

[0113] Step S101, obtaining M initial chipsets; M is a positive integer, the chip models in an initial chipset are all the same, and the chip models of each initial chipset are different from each other;

[0114] Specifically, the computer device can group all chips (also known as graphics chips, GPU chips or graphics processors) according to the chip model. Each initial chip group contains chips of the same chip model, and tensor parallelism is used within the group. For example, if there are M chip models, the computer device can group them into M initial chipsets. Among them, the chip models in an initial chipset are the same, and the chip models between each initial chipset are different. Among them, tensor parallelism can also be called intra-layer parallelism, which refers to the division of parameters in each network layer of the model, and the distribution of training tasks of different parameters of the same network layer to multiple chips for completion. The chip model refers to the model of the chip, which is used to distinguish the product type and performance level of the chip. Optionally, the chip can also be a computing chip containing a processing unit such as a CPU (Central Processing Unit) chip.

[0115] The computer device can sort the initial chipsets in ascending order by the group computing power coefficient of each initial chipset, that is, the initial chipsets with larger group computing power coefficients are sorted in front, and the M initial chipsets include initial chipset 1, initial chipset 2, ..., initial chipset M. The group computing power coefficient of initial chipset 1 is greater than the group computing power coefficient of initial chipset 2, and the group computing power coefficient of initial chipset M is the smallest. Among them, the group computing power coefficient is a parameter determined based on the business computing power parameters, storage parameters, bandwidth parameters and communication parameters of the chips in the parallel group. The larger the group computing power coefficient, the worse the overall performance of the chip, and the smaller the group computing power coefficient, the stronger the overall performance of the chip.

[0116] In the M initial chipsets, the number of chips of each chip model can be { , ,…, },in, This means that the number of chips in the initial chipset 1 is , This means that the number of chips in the initial chipset M is .

[0117] It can be understood that the above-mentioned chips or chipsets composed of chips, as well as the parallel groups mentioned below, can all be machines or devices equipped with chips, which can be used to train the initial model. Since the scale of machines or devices equipped with chips is usually 8 cards per machine, the number of chips in each initial chipset can be a multiple of 8.

[0118] Step S102, based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, the chips in each initial chipset are grouped to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount;

[0119] Specifically, the computer device may obtain the learning parameter amount corresponding to a single network layer in the initial model, and the learning parameter amount may refer to the total number of model parameters, activation value parameters, and gradient parameters calculated during training of a single network layer. The computer device may group the chips in each initial chipset based on the learning parameter amount and the chip models of the M initial chipsets to obtain P pipeline parallel groups.

[0120] For example, it may include an initial chipset 1 of chip model A, an initial chipset 2 of chip model B, and an initial chipset 3 of chip model C, and the number of chips in each initial chipset is 16. The computer device may divide the chips in the initial chipset 1 into a pipeline parallel group 1a containing 8 chips, a pipeline parallel group 1b containing 4 chips, and a pipeline parallel group 1c containing 4 chips; divide the chips in the initial chipset 2 into a pipeline parallel group 2a containing 12 chips and a pipeline parallel group 2b containing 4 chips; and divide the chips in the initial chipset 3 into a pipeline parallel group 3a containing 8 chips and a pipeline parallel group 3b containing 8 chips. Among them, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount. Pipeline parallelism refers to distributing one or more network layers (usually continuous layers) of a model to multiple pipeline parallel groups for completion respectively, and each pipeline parallel group is responsible for processing a part of the network layer of the model.

[0121] Step S103, based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training in parallel on the initial model configured on the corresponding data parallel groups.

[0122] Specifically, the computer device can obtain the average computing power coefficient of P pipeline parallel groups, which is the average value of the group computing power coefficients corresponding to the P pipeline parallel groups, and the group computing power coefficient is determined based on the business computing power parameters, storage parameters, bandwidth parameters and communication parameters of the chip. The computer device can group and combine the P pipeline parallel groups based on the average computing power coefficients of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, and obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups.

[0123] Among them, data parallelism refers to configuring the same initial model to multiple data parallel groups, and then each data parallel group uses different data batches for training. Each data parallel group independently calculates the loss function and gradient, and then aggregates the gradient through some form of communication to update the model parameters of all data parallel groups. Each data parallel group can include the assigned pipeline parallel group, and the computing memory corresponding to each data parallel group is the sum of the computing memories of the assigned pipeline parallel groups. The computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model, that is, each data parallel group can train the initial model separately, and the sample input amount corresponding to each data parallel group can be the same or different. Grouping and combining based on load balancing conditions means grouping and combining pipeline parallel groups with the purpose of making the training time of each data parallel group close to the same and matching the sample input amount corresponding to the data parallel group with the comprehensive performance of the chip in the group, so that the computing resources of the chip are effectively utilized.

[0124] It can be understood that the computer device can allocate a task load that matches its own group computing power coefficient to each pipeline parallel group based on the average computing power coefficient of the P pipeline parallel groups and the group computing power coefficients corresponding to the P pipeline parallel groups. The task load allocated to the pipeline parallel group with a larger group computing power coefficient (i.e., poorer comprehensive computing power) can meet its own comprehensive computing power, so that the training performance of the data parallel group will not be delayed due to excessive task load. The task load allocated to the pipeline parallel group with a smaller group computing power coefficient (i.e., stronger comprehensive computing power) can meet its own comprehensive computing power, thereby improving the utilization rate of chip computing resources in the pipeline parallel group with stronger comprehensive computing power.

[0125] The computer device can divide the training sample set based on the sample input amount corresponding to each data parallel group to obtain the training sample group corresponding to each pipeline parallel group, and perform model training in parallel on the initial model configured for each data parallel group based on the training sample group. After each data parallel group completes the forward propagation calculation and the backward propagation calculation, the data parallel group can return the calculated sub-model parameters to the computer device. The computer device aggregates the sub-model parameters to obtain the aggregated model parameters, and then updates the initial model configured in each data parallel group based on the aggregated model parameters to perform the next batch of training. When the initial model in the computer device meets the model convergence condition, the initial model that meets the model convergence condition can be determined as the target model.

[0126] The embodiment of the present application groups all chips by chip model to obtain an initial chip group with the same chip model in the group. The chip models of each initial chip group are different from each other. Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips in each initial chip group are grouped to obtain P pipeline parallel groups. The chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount. The network layer of the initial model is trained in parallel by tensors using chips of the same chip model. Based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups. Among them, the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training on the initial model configured on the corresponding data parallel group in parallel, and the load balancing condition is satisfied between the D data parallel groups. Among them, data parallelism refers to configuring the same initial model to multiple data parallel groups, and then each data parallel group independently processes its own input samples, and finally aggregates and integrates all model parameters. The embodiment of the present application determines the average computing power coefficient and group computing power coefficient of each pipeline parallel group through a pipeline parallel group composed of chips of the same chip model. Based on the idea of ​​computing power equivalence, the pipeline parallel groups are grouped and combined to obtain data parallel groups, and each data parallel group is assigned a task load equivalent to computing power. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model, the task load of the initial model is split by the data parallel group in the dimension of the sample input size, and the sample input amount corresponding to the data parallel group is determined, so that the load of each data parallel group is balanced, thereby improving the utilization rate of the computing resources of the hybrid deployed chips and improving the training efficiency of the model.

[0127] See also Figure 5 , Figure 5 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 The data processing method can be executed by a computer device, which can be Figure 1 The business server 100 shown. The following will take the data processing method executed by a computer device as an example for description. The data processing method may at least include the following steps S201-S207:

[0128] Step S201, obtaining M initial chipsets; M is a positive integer, the chip models in an initial chipset are the same, and the chip models of each initial chipset are different from each other;

[0129] Specifically, the computer device can group all chips according to chip models. Each initial chip group contains chips of the same chip model. Tensor parallelism is used within the group. For example, if there are M chip models in total, the computer device can group them to obtain M initial chip groups. Among them, the chip models in an initial chip group are the same, and the chip models between each initial chip group are different. The chip model refers to the model of the chip, which is used to distinguish the product type and performance level of the chip.

[0130] The computer device can sort the initial chipsets in ascending order by the group computing power coefficient of each initial chipset, that is, the initial chipsets with larger group computing power coefficients are sorted in front, and the M initial chipsets include initial chipset 1, initial chipset 2, ..., initial chipset M. The group computing power coefficient of initial chipset 1 is greater than the group computing power coefficient of initial chipset 2, and the group computing power coefficient of initial chipset M is the smallest. Among them, the group computing power coefficient is a parameter determined based on the business computing power parameters, storage parameters, bandwidth parameters and communication parameters of the chips in the parallel group. The larger the group computing power coefficient, the worse the overall performance of the chip, and the smaller the group computing power coefficient, the stronger the overall performance of the chip.

[0131] In the M initial chipsets, the number of chips of each chip model can be { , ,…, },in, This means that the number of chips in the initial chipset 1 is , This means that the number of chips in the initial chipset M is .

[0132] It can be understood that the above-mentioned chips or chipsets or parallel groups composed of chips can all be machines or devices equipped with chips, which can be used to train the initial model. Since the scale of machines or devices equipped with chips is usually 8 cards per machine, the number of chips in each initial chipset can be a multiple of 8.

[0133] Step S202, based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, generate initial grouping parameters corresponding to the M initial chipsets respectively; the M initial grouping parameters are all greater than or equal to 2;

[0134] Specifically, the computer device may obtain M preset initial grouping parameters, where the M initial grouping parameters are hyperparameters determined based on the chip model, the number of chips, and the amount of learning parameters. The M initial grouping parameters may be { , ,…, }, the initial grouping parameter is a positive integer.

[0135] in, The value of needs to meet the storage constraint, which can be The computing memory of a chip of a certain chip model is greater than or equal to the learning parameters of the initial model X network layers, that is, A chip of a certain chip model can accommodate at least the storage required for one network layer calculation, and X can be a positive integer greater than 2.

[0136] The M initial chipsets include initial chipset 1 of chip model A, initial chipset 2 of chip model B, and initial chipset 3 of chip model C. The number of chips in each initial chipset is 16. Assuming that one chip of chip model A meets the storage limit condition, two chips of chip model B meet the storage limit condition, and four chips of chip model C meet the storage limit condition, then The solution space for the initial chipset 1 is (1 to 16). The solution space for the initial chipset 2 is (2 to 16). The solution space for the possible values ​​of the initial chipset 3 is (4 to 16).

[0137] for The number of chips in the pipeline parallel group obtained after grouping needs to be a positive integer, that is, Can be The number of computer devices can be reduced by pruning and greedy search strategies. By searching the solution space, M initial grouping parameters corresponding to each of the M initial chipsets are obtained. Assuming that the number of chips in each initial chipset is 16, the M initial grouping parameters can be {2, 4, 8} in sequence.

[0138] Step S203, based on the M initial grouping parameters, the chips in each initial chip group are grouped respectively to obtain N initial parallel groups;

[0139] Specifically, the computer device can group the chips in each initial chipset based on the M initial grouping parameters to obtain N initial parallel groups. Each initial chipset can be divided into Initial parallel groups, that is, M initial grouping parameters can divide each chip model into groups with the number of { , ,…, The initial parallel group of}. The initial parallel group of M initial chips includes the initial chips , initial chipset Includes S target chips, initial chipset The corresponding grouping parameter is G, where G, S and i are all positive integers. The computer device performs initial chip grouping based on M initial grouping parameters. The process of grouping the chips in the initial parallel group can be as follows: based on the initial chip group The grouping parameter G for the initial chipset The chips in the group are grouped to obtain the initial chipset Corresponding T target parallel groups; initial chipset Each corresponding target parallel group includes G target model chips, and the product of T and G is S; when the target parallel groups corresponding to the M initial chipsets are obtained, the target parallel groups corresponding to the M initial chipsets are determined as N initial parallel groups; the N initial parallel groups include T target parallel groups.

[0140] Specifically, G can be 2, S can be 16, that is, the initial chipset Including 16 target chips, computer equipment can be used for the initial chipset The chips in the group are grouped to obtain the initial chipset The corresponding T target parallel groups, the product of T and G is S. When G is 2 and S is 16, the computer device can get the initial chipset There are 8 corresponding target parallel groups, and each of the 8 target parallel groups includes chips of 2 target models.

[0141] When the target parallel groups corresponding to the M initial chipsets are obtained, the target parallel groups corresponding to the M initial chipsets are determined as N initial parallel groups, and the N initial parallel groups include T target parallel groups. For example, the M initial chipsets may include initial chipset 1 of chip model A, initial chipset 2 of chip model B, and initial chipset 3 of chip model C. The number of chips in each initial chipset is 16, and the M initial grouping parameters may be {2, 4, 8} in sequence, that is, the computer device may divide the chips in initial chipset 1 into 8 initial parallel groups, and the 8 initial parallel groups each include 2 chips of chip model A; the chips in initial chipset 2 may be divided into 4 initial parallel groups, and the 4 initial parallel groups each include 4 chips of chip model B; the chips in initial chipset 3 may be divided into 2 initial parallel groups, and the 2 initial parallel groups each include 8 chips of chip model C.

[0142] Step S204, based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped to obtain P pipeline parallel groups;

[0143] Specifically, the computer device may obtain the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups. Taking the calculation of the group computing power coefficient of the initial parallel group 1 among the N initial parallel groups as an example, the initial parallel group 1 may include S target model chips, and the computer device may obtain the business computing power parameters, storage parameters, bandwidth parameters and communication parameters of the target model chips. Among them, the business computing power parameters may be parameters normalized based on the computing power performance of M chip models, the storage parameters may be parameters normalized based on the storage performance of M chip models, the bandwidth parameters may be parameters normalized based on the bandwidth performance of M chip models (e.g., h2d (host to device) bandwidth, d2h (device to host) bandwidth), and the communication parameters may be parameters normalized based on the communication performance of M chip models (e.g., communication performance between chips, communication performance between machines).

[0144] The computer device can generate the intra-group communication coefficient based on the number of cards S and communication parameters of the initial parallel group 1 , based on the business computing power parameters, storage parameters, bandwidth parameters of the target model chip and the number of cards S of the initial parallel group 1, the business computing power coefficient is generated, and the sum of the intra-group communication coefficient and the business computing power coefficient is determined as the group computing power coefficient of the initial parallel group 1.

[0145] When S is equal to 1, the group computing power coefficient The calculation process can be shown as formula (1):

[0146] Formula (1)

[0147] in, is a hyperparameter that indicates the linear relationship between the business computing power parameter and the group computing power coefficient. is a hyperparameter indicating that the storage parameter is linearly related to the group computing power coefficient, is a hyperparameter that indicates the linear relationship between bandwidth parameter and group computing power coefficient, is a hyperparameter that characterizes the intercept term in the linear relationship. For business computing parameters, To store parameters, is the bandwidth parameter. When S is equal to 1, the number of cards S in the initial parallel group 1 and the communication parameters generate the intra-group communication coefficient is 0.

[0148] When S is greater than 1, the group computing power coefficient The calculation process can be shown as formula (2):

[0149] Formula (2)

[0150] in, is the communication parameter, It has a function mapping relationship with the number of cards S, and the embodiment of the present application does not limit this.

[0151] Since chips with poor comprehensive computing power will delay the overall computing latency, computer equipment can calculate the computing power coefficients of all groups. The initial parallel group with a computing power coefficient greater than the average computing power coefficient continues to be split to reduce the group computing power coefficient, and the most suitable grouping strategy is traversed. The process of the computer device continuing to split the initial parallel group can be: obtaining the average computing power coefficient of the N initial parallel groups, determining the initial parallel group with a group computing power coefficient greater than the average computing power coefficient of the N initial parallel groups as Q groups to be split, based on the learning parameter amount, splitting the chips in the Q groups to be split respectively, and obtaining the split group set corresponding to each group to be split; Q is a positive integer, and the computing memory corresponding to the split groups in each split group set is greater than or equal to the learning parameter amount; based on the split group set and the NQ initial parallel groups other than the Q groups to be split, generate A pending parallel group sets, obtain the communication evaluation scores corresponding to the A pending parallel group sets, and determine the pending parallel group set with the maximum communication evaluation score as P pipeline parallel groups; A is a positive integer.

[0152] Specifically, the computer device may obtain an average computing power coefficient of the N initial parallel groups, and the average computing power coefficient may be an average value of the group computing power coefficients of the N initial parallel groups. The computer device may determine the initial parallel groups whose group computing power coefficients are greater than the average computing power coefficients of the N initial parallel groups as Q groups to be split, and split the Q groups to be split based on the learning parameter amount to obtain a set of split groups corresponding to each group to be split. The computing memory corresponding to the split groups in each set of split groups is greater than or equal to the learning parameter amount.

[0153] The computer device may generate A pending parallel group sets based on the split group set and NQ initial parallel groups other than the Q pending groups. Taking the average computing power coefficient equal to 1 as an example, the N initial parallel groups may include an initial parallel group 1 with a group computing power coefficient of 1.4, an initial parallel group 2 with a group computing power coefficient of 1.2, and an initial parallel group 1 with a group computing power coefficient of 0.8. The initial parallel group 1 may include 8 chips, and the initial parallel group 2 and the initial parallel group 3 may each include 4 chips. The computer device can split the initial parallel group 1 with a group computing power coefficient of 1.4 and the initial parallel group 2 with a group computing power coefficient of 1.2 to obtain a split group set corresponding to each initial parallel group. For example, the computer device can further split the initial parallel group 1 to reduce the impact of the comprehensive computing power brought by the communication parameters. If at least two chips in the initial parallel group 1 are required to accommodate the storage required for one network layer calculation, that is, the computing memory of the two chips in the initial parallel group 1 is greater than or equal to the learning parameter amount, the computer device can split the chips in the initial parallel group 1 into split group set 1 and split group set 2 respectively. Split group set 1 refers to splitting the chips in the initial parallel group 1 into split groups containing 2 chips and split groups containing 6 chips, and split group set 2 refers to splitting the chips in the initial parallel group 1 into split groups containing 4 chips and split groups containing 4 chips. Among them, since the split is symmetrical, there are only two splitting situations, and the computer device only needs to traverse and calculate two places. The computational memory corresponding to the split groups in the split group set 1 and the split group set 2 is greater than or equal to the learning parameter amount.

[0154] The computer device can further split the initial parallel group 2 to reduce the impact of the comprehensive computing power brought by the communication parameters. If at least two chips in the initial parallel group 2 are required to accommodate the storage required for one network layer calculation, that is, the computing memory of the two chips in the initial parallel group 2 is greater than or equal to the learning parameter amount, the computer device can split the initial parallel group 2 into a split group set 3. The split group set 3 refers to splitting the chips in the initial parallel group 2 into a split group containing two chips and a split group containing two chips. The computing memory corresponding to the split groups in the split group set 3 is greater than or equal to the learning parameter amount.

[0155] The computer device can generate A pending parallel group sets based on the split group set and NQ initial parallel groups other than Q to-be-split groups. In the above example, the computer device can arrange and combine the splitting conditions of initial parallel group 1, initial parallel group 2 and initial parallel group 3 to obtain A pending parallel group sets, which can include pending parallel group set 1 and pending parallel group set 2. Pending parallel group set 1 refers to split group set 1, split group set 3 and initial parallel group 3, and pending parallel group set 2 refers to split group set 2, split group set 3 and initial parallel group 3.

[0156] The computer device can obtain the communication evaluation scores corresponding to the A pending parallel group sets, determine the pending parallel group set with the maximum communication evaluation score as P pipeline parallel groups, and use the A pending parallel group sets including the pending parallel group set , pending parallel group set Includes L pending parallel groups consisting of the split groups and the initial parallel groups, where i and L are positive integers. Get the pending parallel group set The corresponding communication evaluation score process can be: Get the pending parallel group set The communication parameters corresponding to the chips in the L pending parallel groups and the group computing power coefficients corresponding to the L pending parallel groups are respectively determined; the chip whose communication parameters in the L pending parallel groups are greater than the communication parameter mean is determined as the first chip, and the chip whose communication parameters in the L pending parallel groups are less than or equal to the communication parameter mean is determined as the second chip; the communication parameter mean is determined based on the communication parameters of the chip models of the M initial chip groups; the first evaluation parameter and the second evaluation parameter are obtained, and a pipeline parallel evaluation score is generated based on the first evaluation parameter and the group computing power coefficient of the pending parallel group including the first chip, and a data parallel evaluation score is generated based on the second evaluation parameter and the group computing power coefficient of the pending parallel group including the second chip, and the sum of the pipeline parallel evaluation score and the data parallel evaluation score is determined as the pending parallel group set. Communication assessment score.

[0157] Specifically, the computer device can obtain the pending parallel group set The communication parameters corresponding to the chips in the L pending parallel groups are And the group computing power coefficients corresponding to the L pending parallel groups , the chip whose communication parameter is greater than the average value of the communication parameter in the L pending parallel groups is determined as the first chip, for example, the pending parallel group set The communication parameters of chips 1, 2, ..., and n in the L pending parallel groups are greater than the communication parameter mean. The computer device may determine the chip whose communication parameter is less than or equal to the communication parameter mean in the L pending parallel groups as the second chip. For example, the pending parallel group set Chip in To chip The communication parameter is less than the mean value of the communication parameter, and the pending parallel group set Chip in To chip All are determined as the first chip. Parallel group set to be determined The communication parameters of chips n+1, ..., and L in the chip are greater than the mean value of the communication parameters. Chip n+1, ..., chip L in are all determined as the second chip. Wherein, n is a positive integer, and the communication parameter mean is determined based on the communication parameters of the chip models of the M initial chip groups.

[0158] The computer device may obtain a first evaluation parameter and a second evaluation parameter, and generate a pipeline parallel evaluation score based on the first evaluation parameter and a group computing power coefficient of the pending parallel group including the first chip, wherein the pipeline parallel evaluation score The calculation method of can be shown as formula (3):

[0159] Formula (3)

[0160] in, is the first evaluation parameter, used to evaluate the communication score of the chips in the parallel group in pipeline parallelism, is a hyperparameter determined based on the maximum value of the communication parameter in the first chip.

[0161] The computer device may generate a data parallel evaluation score based on the second evaluation parameter and the group computing power coefficient of the pending parallel group including the second chip, wherein the data parallel evaluation score is The calculation method of can be shown as formula (4):

[0162] Formula (4)

[0163] in, is the second evaluation parameter, used to evaluate the communication score of the chips in the parallel group in data parallelism, is a hyperparameter determined based on the maximum value of the communication parameter in the second chip.

[0164] The computer device can determine the sum of the pipeline parallel evaluation score and the data parallel evaluation score as the pending parallel group set. Communication evaluation score. Communication evaluation score The calculation method of can be shown as formula (5):

[0165] Formula (5)

[0166] in, is the calibration item.

[0167] Step S205, obtaining the initial micro-batch value of the initial model, and generating the target task duration for the average computing power coefficient of the P pipeline parallel groups based on the initial micro-batch value; the initial micro-batch value is determined based on the task domain of the initial model; based on the initial micro-batch value and the total number of samples, generating data parallel grouping parameters, based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, generating D selection coefficient lists and initial sample input amounts corresponding to the D allocation parallel groups; each selection coefficient list includes selection coefficients corresponding to the P pipeline parallel groups, and the selection coefficients of the pipeline parallel groups are used to indicate the selection mapping relationship between the pipeline parallel group and the allocation parallel group; D is the data parallel grouping parameter;

[0168] Specifically, the computer device can obtain the initial micro-batch value of the initial model. The initial micro-batch value is a fixed value determined based on the task domain of the initial model. For example, the initial micro-batch value in the task domain of Vincent graph can be 1, and it can be other values ​​in Vincent graph. The embodiment of the present application does not limit this. The initial micro-batch value is the amount of sample data input by the parallel group in each training batch in the task domain of the initial model. The computer device can generate a target task duration for the average computing power coefficient of P pipeline parallel groups based on the initial micro-batch value. .

[0169] The computer device can generate a data parallel grouping parameter D based on the initial micro-batch value and the total number of samples. The data parallel grouping parameter D is used to determine the data parallelism of the distributed training for the initial model as D. Taking the data parallelism as D as an example, the computer device can group and combine the pipeline parallel groups based on the data parallel grouping parameter D, the target task duration, and the group computing power coefficients corresponding to the P pipeline parallel groups to obtain D allocation parallel groups. The computer device can generate a selection coefficient list and the initial sample input corresponding to the D allocation parallel groups based on the data parallel grouping parameter D, the target task duration, and the group computing power coefficients corresponding to the P pipeline parallel groups.

[0170] D allocation parallel groups include allocation parallel groups As an example, i is a positive integer; based on the data parallel grouping parameter D, the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, the process of generating the selection coefficient list and the initial sample input amount corresponding to the D allocation parallel groups can be: Get the allocation parallel group The corresponding P pending selection parameters are determined by multiplying the products of the P pending selection parameters and the group computing power coefficients corresponding to the P pending selection parameters as P first load parameters, and generating second load parameters based on the P first load parameters; the P pending selection parameters are respectively corresponding to the P pipeline parallel groups one by one; obtaining the allocation parallel group The corresponding pending sample parameters are determined, the ratio between the second load parameter and the pending sample parameters is determined as the pending task duration, the P pending selection parameters and the pending sample parameters are adjusted to obtain H first parameter sets; H is a positive integer, and each first parameter set includes parameter values ​​of the P pending selection parameters and parameter values ​​of the pending sample parameters; one or more first parameter sets that meet the maximum ratio condition are determined as the second parameter set, the second parameter set containing the smallest pending sample parameters is determined as the target allocation parameter set, and the parameter values ​​of the pending sample parameters in the target allocation parameter set are determined as the allocation parallel group The corresponding initial sample input volume is used to determine the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group The corresponding selection coefficient list; the maximum ratio condition refers to the condition that the ratio between the target task duration and the pending task duration reaches the maximum value.

[0171] Specifically, the computer device can generate and allocate parallel groups The corresponding P pending selection parameters , =1 means that the pipeline parallel group with sequence number k is assigned to the i-th allocation parallel group. = 0 means that the pipeline parallel group with the serial number K is not assigned to the i-th allocation parallel group, then the computer device can set the P pending selection parameters Group power coefficient corresponding to P undetermined selection parameters The product between is determined as P first load parameters, and the second load parameters are generated based on the P first load parameters. Among them, the P undetermined selection parameters correspond to the P pipeline parallel groups one by one. Get the allocation parallel group Corresponding undetermined sample parameters , determine the ratio between the second load parameter and the pending sample parameter as the pending task duration, adjust the P pending selection parameters and the pending sample parameters, and obtain H first parameter sets. Each first parameter set includes parameter values ​​of the P pending selection parameters and parameter values ​​of the pending sample parameters. The computer device may determine one or more first parameter sets that meet the maximum ratio condition as the second parameter set, determine the second parameter set containing the smallest pending sample parameter as the target allocation parameter set, and determine the parameter values ​​of the pending sample parameters in the target allocation parameter set as the allocation parallel group. The corresponding initial sample input volume is used to determine the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group The corresponding selection coefficient list. Among them, the maximum ratio condition refers to the condition that the ratio between the target task duration and the pending task duration reaches the maximum value. The above problem can be transformed into an integer linear programming problem, and the solution process is shown in formula (6):

[0172] Formula (6)

[0173] in, It refers to the parameter that makes the function reach the minimum value The value of max is The maximum value of represents the maximum ratio condition. The first parameter set that satisfies the maximum ratio condition may refer to The second parameter set containing the smallest undetermined sample parameter is determined as the target allocation parameter set, which may refer to the second parameter set that satisfies The solution set of . represents the pending sample parameters corresponding to the i-th allocation parallel group, is the target task duration, P is the number of pipeline parallel groups, represents the group computing power coefficient corresponding to the kth pipeline parallel group, D is the number of allocated parallel groups, B is gbs (global batch size), b is mbs (micro batch size), Refers to the set of natural numbers.

[0174] To assign parallel groups For example, the computer device can take P pending selection parameters Group power coefficient corresponding to P undetermined selection parameters The product between is determined as P first load parameters, that is, the first load parameter is The product of the first load parameters is determined as the accumulated sum of the P first load parameters as the second load parameter. The second load parameter is , by solving and partially enumerating integer linear programming problems, we can solve and , and obtain an optimal parallel strategy under the current colocation cluster resources.

[0175] Taking D as 2 and P as 10 as an example, P pipeline parallel groups can include 6 combinations of chip model A: pipeline parallel group A0 (1 card), pipeline parallel group A1 (1 card), pipeline parallel group A2 (2 cards), pipeline parallel group A3 (1 card), pipeline parallel group A4 (1 card), pipeline parallel group A5 (2 cards); 3 combinations of chip model B: pipeline parallel group B0 (2 cards), pipeline parallel group B1 (2 cards), pipeline parallel group B2 (4 cards); 1 combination of chip model C: pipeline parallel group C0 (8 cards). For example, the computer equipment can be solved by formula (6), and the undetermined sample parameters in the target allocation parameter set that meet the maximum ratio condition are respectively =16, =4, the selection coefficient lists corresponding to the target allocation parameter sets are =(1,1,1,1,0,0,1,1,0,0), = (0, 0, 0, 0, 1, 1, 0, 0, 1, 1), the computer device can be based on the initial sample input volume and the group computing power coefficient of each pipeline parallel group, determine the micro-batch value mbs in each pipeline parallel group in the allocation parallel group. For example, the computer device can solve and obtain 2 allocation parallel groups, including allocation parallel group FP0 and allocation parallel group FP1, wherein the allocation parallel group FP0 includes pipeline parallel group A0 (mbs=2), pipeline parallel group A1 (mbs=2), pipeline parallel group A2 (mbs=4), pipeline parallel group A3 (mbs=2), pipeline parallel group B0 (mbs=2), and pipeline parallel group B1 (mbs=4). The allocation parallel group FP1 includes pipeline parallel group A4 (mbs=1), pipeline parallel group A5 (mbs=1), pipeline parallel group B2 (mbs=2), and pipeline parallel group C0 (mbs=1).

[0176] Optionally, the embodiment of the present application can also use the above formula (6) to convert the task load problem of each pipeline parallel group into an integer linear programming problem based on the data parallel grouping parameter D, the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, thereby calculating the micro-batch value of each pipeline parallel group, and then grouping and combining each pipeline parallel group to obtain the allocated parallel group and the initial sample input amount corresponding to the allocated parallel group.

[0177] Step S206, according to the selection coefficients in the selection coefficient list, allocate P pipeline parallel groups to corresponding allocation parallel groups to obtain D transition parallel groups including pipeline parallel groups; based on the load balancing conditions, the target task duration, the D transition parallel groups, the D initial sample input amounts and the group computing power coefficients corresponding to the P pipeline parallel groups respectively, generate the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amounts corresponding to the D transition parallel groups respectively, determine the D transition parallel groups including the layer number parameters as D data parallel groups, and determine the transition sample input amounts corresponding to the D data parallel groups respectively.

[0178] Specifically, the computer device may allocate the P pipeline parallel groups to corresponding allocation parallel groups based on the selection coefficients in the selection coefficient list, thereby obtaining D transition parallel groups including the pipeline parallel groups. For example, transition parallel group The process may include C pipeline parallel groups, i and C are both positive integers; the computer device generates the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amount corresponding to the D transition parallel groups based on the load balancing condition, the target task duration, the D transition parallel groups, the D initial sample input amount and the group computing power coefficients corresponding to the P pipeline parallel groups respectively, and the process may be: generating the layer number parameters corresponding to the transition parallel groups based on the group computing power coefficients corresponding to the C pipeline parallel groups respectively The product of the C undetermined layer number parameters corresponding to each pipeline parallel group and the corresponding group computing power coefficient is determined as the third load parameter; based on the transition parallel group The target sample parameter is determined by the corresponding initial sample input, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter; based on the load balancing condition, C undetermined layer number parameters and the target sample parameter are adjusted to obtain R third parameter sets; R is a positive integer, and each third parameter set includes parameter values ​​of C undetermined layer number parameters and parameter values ​​of the target sample parameter; one or more third parameter sets that meet the maximum product condition are determined as the fourth parameter set, the fourth parameter set containing the smallest target sample parameter is determined as the target transition parameter set, and the parameter values ​​of the target sample parameters in the target transition parameter set are determined as the transition parallel group. The corresponding transition sample input quantity is used to determine the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group The number of layers parameter corresponding to each pipeline parallel group; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches the maximum value.

[0179] Specifically, in the case of fixed data parallel pipeline parallel groups, the problem of dividing the number of model layers can also be transformed into an integer linear programming problem, with D transition parallel groups including transition parallel groups For example, transition parallel group Including the C pipeline parallel groups obtained by solving the above formula (6), the computer device can generate a transition parallel group based on the group computing power coefficients corresponding to the C pipeline parallel groups respectively. The C undetermined number of layer parameters , the number of layers to be determined Indicates the number of layers assigned to the jth pipeline parallel group in the i-th transition parallel group, and the undetermined number of layers corresponding to each pipeline parallel group And the corresponding group computing power coefficient The product of is determined as the third load parameter. Among them, C undetermined layer number parameters It can mean that the computing memory of the chips in the pipeline parallel group is greater than or equal to the parameter of the number of layers to be determined. The indicated parameter quantity. For example, it can be the maximum value of the computation memory of the pipeline parallel group, that is, the parameter of the number of layers to be determined The indicated parameter amount is approximately equal to the computational memory of the chip within the pipeline parallel group.

[0180] Computer equipment can be based on transition parallel groups The target sample parameter is determined by the corresponding initial sample input amount, the product of the third load parameter and the target sample parameter is determined as the fourth load parameter, and based on the load balancing condition, the C undetermined layer number parameters and the target sample parameter are adjusted to obtain R third parameter sets. Among them, R is a positive integer, and each third parameter set includes parameter values ​​of C undetermined layer number parameters and parameter values ​​of the target sample parameter. The computer device can determine one or more third parameter sets that meet the maximum product condition as the fourth parameter set, determine the fourth parameter set containing the smallest target sample parameter as the target transition parameter set, and determine the parameter values ​​of the target sample parameters in the target transition parameter set as the transition parallel group. The corresponding transition sample input quantity is used to determine the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group The number of layers corresponding to each pipeline parallel group in the equation is ; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches the maximum value. The above problem can be transformed into an integer linear programming problem, and the solution process is shown in formula (7):

[0181] Formula (7)

[0182] in, It refers to the parameter that makes the function reach the minimum value The value of max is The maximum value of represents the maximum product condition. The third parameter set that satisfies the maximum product condition can refer to The solution set of determining the fourth parameter set containing the minimum target sample parameter as the target transition parameter set may refer to the fourth parameter set satisfying The solution set of . represents the target sample parameter corresponding to the i-th transition parallel group, Indicates the number of layers assigned to the jth pipeline parallel group in the ith transition parallel group. For example, =2 means that the number of model layers allocated to the first pipeline parallel group in the first transition parallel group is 2. represents the storage capacity of the jth pipeline parallel group in the i-th transition parallel group, It indicates that the amount of storage required by the pipeline parallel group is linearly related to the number of layers. Can refer to the maximum number of layers to be determined parameter , the maximum number of layers to be determined The corresponding parameter quantity can be equal to the maximum value of the computation memory of the pipeline parallel group.

[0183] To simplify the problem, the computer device can express the video memory constraint as a linear relationship with the number of model layers, that is, the demand for video memory of the assigned task cannot exceed the total storage capacity that the current machine can provide. Since the machine with the worst computing power determines the delay of the entire system, by adjusting the division of the number of model layers, the number of model layers assigned to each pipeline parallel group can be matched with the group computing power coefficient of the pipeline parallel group, and the load between the pipeline parallel groups can be balanced. By further adjusting the division of the transition sample input, task load balancing can be achieved between the transition parallel groups. The computer device can solve the above formula (7) to obtain multiple solution sets that meet the maximum product condition, that is, the target sample parameter set, and determine the largest target sample parameter in the target sample parameter set as the transition parallel group. The corresponding transition sample input amount, the C undetermined layer number parameters corresponding to the largest target sample parameter in the undetermined sample parameter set are determined as the transition parallel group The computer device may determine the D transition parallel groups including the layer number parameter as D data parallel groups, and determine the transition sample input amounts corresponding to the D transition parallel groups as the sample input amounts corresponding to the D data parallel groups.

[0184] It can be understood that the parallel strategy between chips is determined based on the co-location cluster resources. Due to the limitation of integer reduction, there will be multiple similar solutions. By solving and partially enumerating the integer linear programming problem, the computer device can determine the solution with the highest overall throughput as an optimal parallel strategy under the current co-location cluster resources. For example, some data parallel groups may be composed of pipeline parallel groups with larger storage. The PP parallelism can be reduced on these data parallel groups, or the TP parallelism of the pipeline parallel groups within the data parallel groups can be reduced to reduce the performance overhead caused by communication.

[0185] Task division (division of model layers) under fixed parallel strategy. Considering the memory constraint , and the number of model layers, number of chips, and sample input in the variable parameters are subject to integer constraints. Therefore, the linear programming problem of distributed training can be transformed into solving D+1 independent integer linear programming problems through branch and bound, cutting plane, and enumeration methods, thereby limiting the feasible domain of the solution set to integer points for step-by-step solution. Among them, the parameter solution set of an integer linear programming problem is used to solve the problem of layer division (i.e., the layer number parameter ), and D integer linear programming problems are used to solve how many training data samples need to be allocated to each data parallel group (i.e., the target sample parameter By solving the linear programming problem, the final task (number of layers parameter) can be determined. and target sample parameters )’s division.

[0186] Step S207: Based on the sample input amounts corresponding to the D data parallel groups, the training sample set is divided to obtain the training sample groups corresponding to each data parallel group; based on the training sample groups, the initial model configured for each data parallel group is trained in parallel to obtain the sub-model parameters corresponding to each data parallel group; based on the sub-model parameters corresponding to each data parallel group, a target model is generated.

[0187] Specifically, the computer device can divide the training sample set based on the sample input amount corresponding to each data parallel group to obtain the training sample group corresponding to each pipeline parallel group, and perform model training in parallel on the initial model configured for each data parallel group based on the training sample group. After each data parallel group completes the forward propagation calculation and the backward propagation calculation, the data parallel group can return the calculated sub-model parameters to the computer device. The computer device aggregates the sub-model parameters to obtain the aggregated model parameters, and then updates the initial model configured in each data parallel group based on the aggregated model parameters to perform the next batch of training. When the initial model in the computer device meets the model convergence condition, the initial model that meets the model convergence condition can be determined as the target model.

[0188] The embodiment of the present application first constructs an evaluation model for different chip models in terms of computing power, bandwidth, storage, etc., and splits the parallel strategy based on the model to achieve load balancing on heterogeneous machines and avoid idling waiting scenarios. Different parallel dimensions can be deployed on different types of machines. In order to better tap the strengths of each chip model, based on the mixed deployment of heterogeneous machines with equivalent computing power and a flexible task load splitting method, the formulation of each parallel group is realized, so that the allocated workload can match the comprehensive computing power of the corresponding chip. Ultimately, the completion time of the corresponding workload on each data parallel group is similar, load balancing is achieved, and the effectiveness of hybrid training is maximized.

[0189] Please join us Figure 6 , Figure 6 This is a data processing scenario provided by an embodiment of the present application. Figure 3 .like Figure 6 As shown, the computer device can be configured with two data parallel groups (data parallel group 1 and data parallel group 2) for distributed learning. The data parallel group 1 independently trains the initial model through data 1, and the data parallel group 2 independently trains the initial model through data 2. The data parallel group 1 includes the pipeline parallel group 1 and the pipeline parallel group 2, and the data parallel group 2 may include the pipeline parallel group 3 and the pipeline parallel group 4. Among them, the pipeline parallel group 1 may include chips 11, 12, 13 and 14, the pipeline parallel group 2 may include chips 22 and 23, the pipeline parallel group 3 may include chips 31 and 32, and the pipeline parallel group 4 may include chips 41, 42, 43 and 44. The pipeline parallel group 1 can be used to train the network layer 1 and the network layer 2 in the initial model, and the pipeline parallel group 2 can be used to train the network layer 3, the network layer 4, the network layer 6 and the network layer 6 in the initial model. Pipeline parallel group 3 can be used to train network layer 1 and network layer 2 in the initial model, and pipeline parallel group 4 can be used to train network layer 3, network layer 4, network layer 6, and network layer 6 in the initial model.

[0190] The tensor parallelism of the chip in pipeline parallel group 1 is 4, the tensor parallelism of the chip in pipeline parallel group 2 is 2, the tensor parallelism of the chip in pipeline parallel group 3 is 2, and the tensor parallelism of the chip in pipeline parallel group 4 is 4. Therefore, based on the existing method, this method further decomposes the configurable granularity of the task and supports the configuration of different tensor parallelisms in each pipeline parallel group. Different pipeline parallel groups can be assigned to different types of data parallel groups. The data parallel group can be a machine or device equipped with chips in the pipeline parallel group. In this way, the number of layers on the pipeline parallel group and different tensor parallelism can be configured on different types of machines according to the hardware characteristics of the machine itself. Computer equipment can also be split by the number of samples in the global batch value, and the task balance of chips with different capabilities can be achieved by passing in data of different sizes to avoid the barrel effect. At the same time, different micro-batch values ​​are passed in according to the storage capacity of the machine to achieve the balance of computing resources and storage resources in a single machine.

[0191] Please also refer to Figure 7 , Figure 7 This is a data processing scenario provided by an embodiment of the present application. Figure 4 .like Figure 7 As shown in FIG. 1 , during the training process of the data parallel group, the pipeline parallel group in the data parallel group can support different MBS heterogeneous configurations. Figure 7 The arrows in the figure indicate the direction of the computational data flow. The solid arrows indicate the forward propagation computation, and the dotted arrows indicate the reverse propagation computation. Taking the data parallel group including pipeline parallel group 1 and pipeline parallel group 2 as an example, the mbs of pipeline parallel group 1 and pipeline parallel group 2 can both be 1, such as Figure 7 As shown, arrow F1 can be used to represent the forward propagation calculation of pipeline parallel group 1 for network layer 1 and network layer 2. Arrow F2 can be used to represent the forward propagation calculation of pipeline parallel group 2 for network layer 3, network layer 4, network layer 5 and network layer 6. Arrow B1 can be used to represent the reverse propagation calculation of pipeline parallel group 1 for network layer 1 and network layer 2. Arrow B2 can be used to represent the reverse propagation calculation of pipeline parallel group 2 for network layer 3, network layer 4, network layer 5 and network layer 6. The computer device can adjust the chips in the pipeline parallel group to obtain a new pipeline parallel group 1 and a new pipeline parallel group 2. The mbs of the new pipeline parallel group 1 can be 2, and the mbs of the new pipeline parallel group 2 can be 1, that is, the new pipeline parallel group 1 performs one forward propagation calculation, and the new pipeline parallel group 2 can perform two forward propagation calculations. After the new pipeline parallel group 1 completes two reverse propagation calculations, it can be directly transferred back to the new pipeline parallel group 1 for one reverse propagation calculation.

[0192] It can be understood that the variables that need to be stored in distributed training mainly include the total number of parameters of model parameters, activation value parameters and gradient parameters. The allocation between model parameters and storage can be achieved by adjusting the PP parallelism and TP parallelism, and the storage requirements of activation values ​​and gradient parameters are often positively correlated with the sequence size of the input data. In the prior art, the micro-batch values ​​of all parallel groups are the same, and storage resource charges of some models will appear in the mixed cluster. This solution provides decoupling of the above dependencies, and different DP parallel groups, PP parallel groups and TP parallel groups can arbitrarily configure micro-batch values ​​under the restriction of satisfying the multiple relationship. The restriction of the multiple relationship is that when the mbs of the previous pipeline parallel group is 2, the next pipeline parallel group can perform two calculations with mbs of 1, that is, different mbs can be configured. By reasonably configuring the size of the micro-batch value, the storage resources of different machines can be fully utilized to achieve training of larger models on limited hardware resources. Finally, the completion time of the corresponding workload on each chip is similar, load balancing is achieved, and the hybrid training efficiency is maximized.

[0193] The embodiment of the present application groups all chips by chip model to obtain an initial chipset with the same chip model in the group. The chip models of each initial chipset are different from each other. Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, the chips in each initial chipset are grouped separately to obtain P pipeline parallel groups. The chip models in a pipeline parallel group are all the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount. The network layer of the initial model is trained in tensor parallel using chips of the same chip model, so that the computing time of multiple chips in the pipeline parallel group during tensor parallel training is exactly the same, thereby greatly reducing the training delay. Among them, tensor parallelism refers to the division of parameters in each network layer of the model, and the distribution of training tasks of different parameters of the same network layer to multiple chips for completion. Based on the load balancing condition, the average computing power coefficient of P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups. Among them, the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training in parallel on the initial model configured on the corresponding data parallel group, and the load balancing condition is met between the D data parallel groups. Among them, pipeline parallelism refers to distributing one or more network layers of the model to multiple pipeline parallel groups for completion, and each pipeline parallel group is responsible for processing a part of the network layer of the model. Data parallelism refers to configuring the same initial model to multiple data parallel groups, and then each data parallel group independently processes its own input samples, and finally aggregates and integrates all model parameters. Through the idea of ​​computing power equivalence, based on the average computing power coefficient and group computing power coefficient of the pipeline parallel group, each pipeline parallel group is assigned a task load that matches its own group computing power coefficient, so that the computing resources of the chips in the pipeline parallel group are effectively utilized, and multiple pipeline parallel groups are grouped into data parallel groups. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model, the sample input amount corresponding to the data parallel group is determined based on the task load assigned to each pipeline parallel group. The task load of the initial model can be split by the data parallel group in the dimension of sample input size, the task load of the initial model can be split by the dimension of slicing between model layers by the pipeline parallel group, and tensor parallel training can be performed by the chips in the pipeline parallel group. The task load of the initial model is split by the dimension divided within the model layer, so that the load of each parallel group is balanced, thereby improving the utilization of computing resources and improving the training efficiency of the model.

[0194] The embodiment of the present application supports a flexible multi-dimensional model parallel strategy. The training platform can dynamically decompose the training task into multiple subtasks according to the characteristics of the training task and the availability of resources, and assign them to different machine types. Support the configuration of different tensor parallelism in each pipeline parallel group. Based on the average computing power coefficient and group computing power coefficient of the pipeline parallel group, each pipeline parallel group can be assigned to a task load that matches its own group computing power coefficient. Different pipeline parallel groups can be assigned to different types of data parallel groups. The data parallel group can be a machine or device equipped with a chip in the pipeline parallel group. In this way, the number of model layers on the pipeline parallel group and the tensor parallelism of the chip in the pipeline parallel group can be configured according to the hardware characteristics of the machine itself on different types of machines. It is also possible to split the number of samples in the global batch value, and to achieve task balance of chips of different capabilities by passing in data of different sizes. At the same time, according to the storage capacity of the machine, different micro-batch values ​​are passed in to achieve the balance of computing resources and storage resources in a single machine. It can adapt to different hybrid cluster configurations and effectively improve the efficiency of model training. Based on the core idea of ​​computing power coefficient equivalence, the training tasks are allocated, and the allocation problem is simplified to an integer programming problem for solution, so that the computing load of each parallel group is basically balanced, thereby achieving efficient use of computing resources. The embodiment of the present application can be widely used in distributed training platforms based on hybrid deployment machines. This flexible parallel strategy can effectively optimize the utilization of computing resources, reduce communication overhead, has high flexibility and scalability, effectively improves the efficiency and effect of large model training, and reduces computing costs.

[0195] See also Figure 8 , Figure 8 Schematic diagram of a data processing device provided in an embodiment of the present application. Figure 8 As shown, the data processing device 1 includes a chip acquisition module 810 , a pipeline parallel grouping module 820 and a data parallel grouping module 830 .

[0196] The chip acquisition module 810 is used to acquire M initial chip groups; M is a positive integer, the chip models in an initial chip group are the same, and the chip models of each initial chip group are different from each other;

[0197] The pipeline parallel grouping module 820 is used to group the chips in each initial chipset based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount;

[0198] The data parallel grouping module 830 is used to group and combine the P pipeline parallel groups based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, so as to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training in parallel on the initial model configured on the corresponding data parallel groups.

[0199] In one possible implementation, the P pipeline parallel groups include the pipeline parallel groups , pipeline parallel group It includes S chips, where S is a positive integer; the pipeline parallel group Each chip in is used to train the network layer of the initial model through tensor parallelism, and the number of parameters of the network layer allocated to the S chips is equal.

[0200] In a possible implementation, the pipeline parallel grouping module 820 is used to group the chips in each initial chipset based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, and to obtain P pipeline parallel groups, and specifically to perform the following operations:

[0201] Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, initial grouping parameters corresponding to the M initial chipsets are generated; the M initial grouping parameters are all greater than or equal to 2;

[0202] Based on the M initial grouping parameters, the chips in each initial chipset are grouped separately to obtain N initial parallel groups. Based on the average computing power coefficients of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped separately to obtain P pipeline parallel groups; N is a positive integer, N is greater than M, and N is less than or equal to P.

[0203] In a possible implementation, the M initial chipsets include an initial chipset , initial chipset Includes S target chips, initial chipset The corresponding grouping parameter is G, where G, S and i are all positive integers; the pipeline parallel grouping is used to group the chips in each initial chipset based on the M initial grouping parameters to obtain N initial parallel groups, and is specifically used to perform the following operations:

[0204] Based on initial chipset The grouping parameter G for the initial chipset The chips in the group are grouped to obtain the initial chipset Corresponding T target parallel groups; initial chipset Each corresponding target parallel group includes G target model chips, and the product of T and G is S;

[0205] When the target parallel groups corresponding to the M initial chip groups are obtained, the target parallel groups corresponding to the M initial chip groups are determined as N initial parallel groups; the N initial parallel groups include T target parallel groups.

[0206] In a possible implementation, the pipeline parallel grouping module 820 is used to group the chips in each initial parallel group based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, and to obtain P pipeline parallel groups, and specifically to perform the following operations:

[0207] The average computing power coefficient of N initial parallel groups is obtained, and the initial parallel groups whose group computing power coefficient is greater than the average computing power coefficient of the N initial parallel groups are determined as Q groups to be split. Based on the learning parameter amount, the chips in the Q groups to be split are split respectively to obtain the split group sets corresponding to each group to be split; Q is a positive integer, and the computing memory corresponding to the split groups in each split group set is greater than or equal to the learning parameter amount; based on the split group set and NQ initial parallel groups other than the Q groups to be split, A pending parallel group sets are generated, and the communication evaluation scores corresponding to the A pending parallel group sets are obtained, and the pending parallel group set with the maximum communication evaluation score is determined as P pipeline parallel groups; A is a positive integer.

[0208] In a possible implementation, the A pending parallel group sets include the pending parallel group set , pending parallel group set Including L pending parallel groups consisting of the split groups and the initial parallel groups, i and L are positive integers; when the pipeline parallel grouping module 820 is used to obtain the communication evaluation scores corresponding to the A pending parallel group sets, it is specifically used to perform the following operations:

[0209] Get pending parallel group collection The communication parameters corresponding to the chips in the L pending parallel groups and the group computing power coefficients corresponding to the L pending parallel groups;

[0210] The chips whose communication parameters are greater than the average value of the communication parameters in the L pending parallel groups are determined as the first chip, and the chips whose communication parameters are less than or equal to the average value of the communication parameters in the L pending parallel groups are determined as the second chip; the average value of the communication parameters is determined based on the communication parameters of the chip models of the M initial chip groups;

[0211] Obtain a first evaluation parameter and a second evaluation parameter, generate a pipeline parallel evaluation score based on the first evaluation parameter and the group computing power coefficient of the pending parallel group including the first chip, generate a data parallel evaluation score based on the second evaluation parameter and the group computing power coefficient of the pending parallel group including the second chip, and determine the sum of the pipeline parallel evaluation score and the data parallel evaluation score as the pending parallel group set Communication assessment score.

[0212] In one possible implementation, the P pipeline parallel groups include the pipeline parallel groups , pipeline parallel group The target chip includes S chips, where S is a positive integer. The pipeline parallel grouping module 820 is further configured to perform the following operations:

[0213] Obtain the business computing power parameters, storage parameters, bandwidth parameters, and communication parameters of the target chip model;

[0214] Pipeline-based parallel groups The number of cards S and communication parameters generate the intra-group communication coefficient, based on the business computing power parameters, storage parameters, bandwidth parameters and pipeline parallel group of the target model chip The number of cards S generates the business computing power coefficient, and the sum of the intra-group communication coefficient and the business computing power coefficient is determined as the pipeline parallel group The group computing power coefficient.

[0215] In a possible implementation, the data parallel grouping module 830 is used to group and combine the P pipeline parallel groups based on the average computing power coefficients of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups, and specifically to perform the following operations:

[0216] Obtain an initial micro-batch value of the initial model, and generate a target task duration for an average computing power coefficient of P pipeline parallel groups based on the initial micro-batch value; the initial micro-batch value is determined based on the task domain of the initial model;

[0217] Based on the initial micro-batch value and the total number of samples, generate data parallel grouping parameters, and based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, generate D selection coefficient lists and the initial sample input amounts corresponding to the D allocation parallel groups; each selection coefficient list includes the selection coefficients corresponding to the P pipeline parallel groups, and the selection coefficients of the pipeline parallel groups are used to indicate the selection mapping relationship between the pipeline parallel groups and the allocation parallel groups; D is the data parallel grouping parameter;

[0218] According to the selection coefficients in the selection coefficient list, the P pipeline parallel groups are assigned to corresponding allocation parallel groups to obtain D transition parallel groups including the pipeline parallel groups;

[0219] Based on the load balancing conditions, the target task duration, D transition parallel groups, D initial sample input amounts, and the group computing power coefficients corresponding to the P pipeline parallel groups, the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amounts corresponding to the D transition parallel groups are generated, the D transition parallel groups containing the layer number parameters are determined as D data parallel groups, and the transition sample input amounts corresponding to the D transition parallel groups are determined as the sample input amounts corresponding to the D data parallel groups.

[0220] In a possible implementation, the D allocation parallel groups include the allocation parallel group , i is a positive integer; the data parallel grouping module 830 is used to generate a selection coefficient list and an initial sample input amount corresponding to each of the D allocation parallel groups based on the target task duration and the group computing power coefficients corresponding to each of the P pipeline parallel groups, and is specifically used to perform the following operations:

[0221] Get allocation parallel group corresponding P pending selection parameters, determining the product of the P pending selection parameters and the group computing power coefficients corresponding to the P pending selection parameters as P first load parameters, and generating second load parameters based on the P first load parameters; the P pending selection parameters correspond one-to-one to the P pipeline parallel groups respectively;

[0222] Get allocation parallel group Corresponding to the pending sample parameters, the ratio between the second load parameter and the pending sample parameters is determined as the pending task duration, and the P pending selection parameters and the pending sample parameters are adjusted to obtain H first parameter sets; H is a positive integer, and each first parameter set includes parameter values ​​of the P pending selection parameters and parameter values ​​of the pending sample parameters;

[0223] One or more first parameter sets satisfying the maximum ratio condition are determined as the second parameter set, the second parameter set containing the smallest pending sample parameter is determined as the target allocation parameter set, and the parameter values ​​of the pending sample parameters in the target allocation parameter set are determined as the allocation parallel group. The corresponding initial sample input volume is used to determine the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group The corresponding selection coefficient list; the maximum ratio condition refers to the condition that the ratio between the target task duration and the pending task duration reaches the maximum value.

[0224] In a possible implementation, the D transition parallel groups include the transition parallel group , Transition Parallel Group Including C pipeline parallel groups, i and C are both positive integers; the data parallel grouping module 830 is used to generate the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input corresponding to the D transition parallel groups based on the load balancing condition, the target task duration, the D transition parallel groups, the D initial sample inputs and the group computing power coefficients corresponding to the P pipeline parallel groups, and specifically to perform the following operations:

[0225] Generate a transition parallel group based on the group computing power coefficients corresponding to the C pipeline parallel groups The C undetermined layer number parameters are determined, and the product of the undetermined layer number parameters corresponding to each pipeline parallel group and the corresponding group computing power coefficient is determined as the third load parameter;

[0226] Transition-based parallel groups The corresponding initial sample input amount determines the target sample parameter, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter;

[0227] Based on the load balancing condition, adjust C undetermined layer number parameters and target sample parameters to obtain R third parameter sets; R is a positive integer, and each third parameter set includes parameter values ​​of C undetermined layer number parameters and parameter values ​​of target sample parameters;

[0228] One or more third parameter sets satisfying the maximum product condition are determined as the fourth parameter set, the fourth parameter set containing the minimum target sample parameter is determined as the target transition parameter set, and the parameter values ​​of the target sample parameters in the target transition parameter set are determined as the transition parallel set. The corresponding transition sample input quantity is used to determine the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group The number of layers parameter corresponding to each pipeline parallel group; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches the maximum value.

[0229] In a possible implementation, the data parallel grouping module 830 is further configured to perform the following operations:

[0230] Based on the sample input amounts corresponding to the D data parallel groups, the training sample set is divided to obtain training sample groups corresponding to each data parallel group;

[0231] Based on the training sample group, the initial model configured for each data parallel group is trained in parallel to obtain the sub-model parameters corresponding to each data parallel group, and based on the sub-model parameters corresponding to each data parallel group, a target model is generated.

[0232] The embodiment of the present application groups all chips by chip model to obtain an initial chip group with the same chip model in the group. The chip models of each initial chip group are different from each other. Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips in each initial chip group are grouped separately to obtain P pipeline parallel groups. The chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount. The network layer of the initial model is trained in tensor parallel by chips of the same chip model, so that the calculation time of multiple chips in the pipeline parallel group during tensor parallel training is exactly the same, thereby greatly reducing the training delay. Among them, tensor parallelism refers to the division of parameters in each network layer of the model, and the distribution of training tasks of different parameters of the same network layer to multiple chips to complete separately. Based on the average computing power coefficient of P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficient corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amount corresponding to the D data parallel groups. The input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training on the initial models configured on the corresponding data parallel groups in parallel, and the load balancing conditions are met between the D data parallel groups. Pipeline parallelism refers to distributing one or more network layers of the model to multiple pipeline parallel groups for completion, and each pipeline parallel group is responsible for processing a part of the network layer of the model. Data parallelism refers to configuring the same initial model to multiple data parallel groups, and then each data parallel group independently processes its own input samples, and finally aggregates and integrates all model parameters. Through the idea of ​​computing power equivalence, based on the average computing power coefficient and group computing power coefficient of the pipeline parallel group, each pipeline parallel group is assigned a task load that matches its own group computing power coefficient, so that the computing resources of the chips in the pipeline parallel group are effectively utilized, and multiple pipeline parallel groups are grouped into data parallel groups. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model, the sample input amount corresponding to the data parallel group is determined based on the task load assigned to each pipeline parallel group. The task load of the initial model can be split by the data parallel group in the dimension of sample input size, the task load of the initial model can be split by the dimension of slicing between model layers by the pipeline parallel group, and tensor parallel training can be performed by the chips in the pipeline parallel group. The task load of the initial model is split by the dimension divided within the model layer, so that the load of each parallel group is balanced, thereby improving the utilization of computing resources and improving the training efficiency of the model.

[0233] The embodiment of the present application supports a flexible multi-dimensional model parallel strategy. The training platform can dynamically decompose the training task into multiple subtasks according to the characteristics of the training task and the availability of resources, and assign them to different machine types. Support the configuration of different tensor parallelism in each pipeline parallel group. Based on the average computing power coefficient and group computing power coefficient of the pipeline parallel group, each pipeline parallel group can be assigned to a task load that matches its own group computing power coefficient. Different pipeline parallel groups can be assigned to different types of data parallel groups. The data parallel group can be a machine or device equipped with a chip in the pipeline parallel group. In this way, the number of model layers on the pipeline parallel group and the tensor parallelism of the chip in the pipeline parallel group can be configured according to the hardware characteristics of the machine itself on different types of machines. It is also possible to split the number of samples in the global batch value, and to achieve task balance of chips of different capabilities by passing in data of different sizes. At the same time, according to the storage capacity of the machine, different micro-batch values ​​are passed in to achieve the balance of computing resources and storage resources in a single machine. It can adapt to different hybrid cluster configurations and effectively improve the efficiency of model training. Based on the core idea of ​​computing power coefficient equivalence, the training tasks are allocated, and the allocation problem is simplified to an integer programming problem for solution, so that the computing load of each parallel group is basically balanced, thereby achieving efficient use of computing resources. The embodiment of the present application can be widely used in distributed training platforms based on hybrid deployment machines. This flexible parallel strategy can effectively optimize the utilization of computing resources, reduce communication overhead, has high flexibility and scalability, effectively improves the efficiency and effect of large model training, and reduces computing costs.

[0234] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0235] See also Fig. 9 , Fig. 9 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig. 9As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig. 9 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0236] In such Fig. 9 In the computer device 1000 shown, the network interface 1004 can provide a network communication network element; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0237] Obtain M initial chipsets, where M is a positive integer, the chip models in an initial chipset are all the same, and the chip models between each initial chipset are different;

[0238] Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, the chips in each initial chipset are grouped to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount;

[0239] Based on the load balancing condition, the average computing power coefficient of P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training in parallel on the initial model configured on the corresponding data parallel groups.

[0240] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 and Figure 4 The description of the data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0241] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the processor executes the computer program, it can execute the above Figure 3 and Figure 4 The description of the above data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0242] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been displayed or is to be displayed.

[0243] In addition, it should be noted that: the embodiment of the present application also provides a computer program product, the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above Figure 3 and Figure 4 Any method provided by the corresponding embodiment.

[0244] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0245] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of network elements in the above description. Whether these network elements are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described network elements for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0246] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0247] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0248] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0249] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Obtain M initial chipsets; M is a positive integer, the chip models in an initial chipset are all the same, and the chip models between each initial chipset are different; Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, the chips in each initial chipset are grouped to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount; Based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to parallelly train the initial model configured on the corresponding data parallel groups.

2. The method according to claim 1, characterized in that The P pipeline parallel groups include pipeline parallel groups , the pipeline parallel group The pipeline parallel group comprises S chips, i and S are both positive integers; Each chip in is used to perform model training on the network layer of the initial model through tensor parallelism, and the learning parameter quantities of the network layers allocated to the S chips are equal.

3. The method according to claim 1, characterized in that The chips in each initial chipset are grouped based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets to obtain P pipeline parallel groups, including: Based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets, generating initial grouping parameters corresponding to the M initial chipsets respectively; the M initial grouping parameters are all greater than or equal to 2; Based on the M initial grouping parameters, the chips in each initial chipset are grouped separately to obtain N initial parallel groups; based on the average computing power coefficients of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped separately to obtain P pipeline parallel groups; N is a positive integer, N is greater than M, and N is less than or equal to P.

4. The method according to claim 3, characterized in that The M initial chipsets include an initial chipset , the initial chipset The initial chipset includes S target chips. The corresponding grouping parameter is G, where G, S and i are all positive integers; based on the M initial grouping parameters, the chips in each initial chipset are grouped to obtain N initial parallel groups, including: Based on the initial chipset The grouping parameter G, for the initial chipset The chips in the group are grouped to obtain the initial chipset The corresponding T target parallel groups; the initial chipset Each corresponding target parallel group includes G target model chips, and the product of T and G is S; When the target parallel groups corresponding to the M initial chipsets are obtained, the target parallel groups corresponding to the M initial chipsets are determined as N initial parallel groups; the N initial parallel groups include the T target parallel groups.

5. The method according to claim 3, characterized in that: The chips in each initial parallel group are grouped based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups to obtain P pipeline parallel groups, including: Obtain the average computing power coefficient of the N initial parallel groups, determine the initial parallel groups whose group computing power coefficient is greater than the average computing power coefficient of the N initial parallel groups as Q groups to be split, and split the chips in the Q groups to be split based on the learning parameter amount to obtain a set of split groups corresponding to each group to be split; Q is a positive integer, and the computing memory corresponding to the split groups in each set of split groups is greater than or equal to the learning parameter amount; A pending parallel group sets are generated based on the split group set and NQ initial parallel groups other than the Q to-be-split groups, communication evaluation scores corresponding to the A pending parallel group sets are obtained, and the pending parallel group set with the maximum communication evaluation score is determined as the P pipeline parallel groups; A is a positive integer.

6. The method according to claim 5, characterized in that The A pending parallel group sets include a pending parallel group set , pending parallel group set Including L pending parallel groups consisting of the split groups and the initial parallel groups, i and L are positive integers; the obtaining of the communication evaluation scores corresponding to the A pending parallel group sets respectively includes: Get the pending parallel group set The communication parameters corresponding to the chips in the L pending parallel groups and the group computing power coefficients corresponding to the L pending parallel groups; Determine the chip whose communication parameter is greater than the communication parameter mean value in the L pending parallel groups as the first chip, and determine the chip whose communication parameter is less than or equal to the communication parameter mean value in the L pending parallel groups as the second chip; the communication parameter mean value is determined based on the communication parameters of the chip models of the M initial chip groups; Obtain a first evaluation parameter and a second evaluation parameter, generate a pipeline parallel evaluation score based on the first evaluation parameter and the group computing power coefficient of the pending parallel group including the first chip, generate a data parallel evaluation score based on the second evaluation parameter and the group computing power coefficient of the pending parallel group including the second chip, and determine the sum of the pipeline parallel evaluation score and the data parallel evaluation score as the pending parallel group set Communication assessment score.

7. The method according to claim 1, characterized in that The P pipeline parallel groups include pipeline parallel groups , the pipeline parallel group There are S target chips, i and S are both positive integers; the method further includes: Obtaining business computing power parameters, storage parameters, bandwidth parameters, and communication parameters of the target chip model; Based on the pipeline parallel group The number of chips S and the communication parameters generate the intra-group communication coefficient, based on the business computing power parameters, storage parameters, bandwidth parameters of the target model chip and the pipeline parallel group The number of chips S is used to generate a business computing power coefficient, and the sum of the intra-group communication coefficient and the business computing power coefficient is determined as the pipeline parallel group The group computing power coefficient.

8. The method according to claim 1, characterized in that The method comprises: grouping and combining the P pipeline parallel groups based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups, including: Obtaining an initial micro-batch value of the initial model, and generating a target task duration for an average computing power coefficient of the P pipeline parallel groups based on the initial micro-batch value; the initial micro-batch value is determined based on a task domain of the initial model; Based on the initial micro-batch value and the total number of samples, a data parallel grouping parameter is generated; based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, D selection coefficient lists and initial sample input amounts corresponding to the D allocation parallel groups are generated; each selection coefficient list includes selection coefficients corresponding to the P pipeline parallel groups, and the selection coefficient of the pipeline parallel group is used to indicate the selection mapping relationship between the pipeline parallel group and the allocation parallel group; D is the data parallel grouping parameter; the D allocation parallel groups are all empty; Allocate the P pipeline parallel groups to corresponding allocation parallel groups according to the selection coefficients in the selection coefficient list, and obtain D transition parallel groups including the pipeline parallel groups; Based on the load balancing condition, the target task duration, D transition parallel groups, D initial sample input amounts and the group computing power coefficients corresponding to the P pipeline parallel groups respectively, the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amounts corresponding to the D transition parallel groups respectively are generated, the D transition parallel groups containing the layer number parameters are determined as D data parallel groups, and the transition sample input amounts corresponding to the D transition parallel groups respectively are determined as the sample input amounts corresponding to the D data parallel groups respectively.

9. The method according to claim 8, characterized in that The D allocation parallel groups include allocation parallel groups , i is a positive integer; the generating of a selection coefficient list and initial sample input amounts corresponding to the D allocation parallel groups based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups respectively includes: Get allocation parallel group corresponding P pending selection parameters, determining the products of the P pending selection parameters and the group computing power coefficients corresponding to the P pending selection parameters as P first load parameters, and generating second load parameters based on the P first load parameters; the P pending selection parameters respectively correspond to the P pipeline parallel groups one by one; Get allocation parallel group corresponding to the pending sample parameters, determining the ratio between the second load parameter and the pending sample parameter as the pending task duration, adjusting the P pending selection parameters and the pending sample parameters, and obtaining H first parameter sets; H is a positive integer, and each first parameter set includes parameter values ​​of the P pending selection parameters and parameter values ​​of the pending sample parameters; One or more first parameter sets satisfying the maximum ratio condition are determined as the second parameter set, the second parameter set containing the smallest pending sample parameter is determined as the target allocation parameter set, and the parameter values ​​of the pending sample parameters in the target allocation parameter set are determined as the allocation parallel group. The corresponding initial sample input amount, the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set are determined as the allocation parallel group The corresponding selection coefficient list; the maximum ratio condition refers to the condition that the ratio between the target task duration and the pending task duration reaches the maximum value.

10. The method according to claim 8, characterized in that The D transition parallel groups include a transition parallel group , the transition parallel group Including C pipeline parallel groups, i and C are both positive integers; based on the load balancing condition, the target task duration, the D transition parallel groups, the D initial sample input amounts and the group computing power coefficients corresponding to the P pipeline parallel groups, the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input amounts corresponding to the D transition parallel groups are generated, including: Generate a computing power coefficient for the transition parallel group based on the group computing power coefficients corresponding to the C pipeline parallel groups respectively. The C undetermined layer number parameters are determined, and the product of the undetermined layer number parameters corresponding to each pipeline parallel group and the corresponding group computing power coefficient is determined as the third load parameter; Based on the transition parallel group The corresponding initial sample input amount determines the target sample parameter, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter; Based on the load balancing condition, the C undetermined layer number parameters and the target sample parameter are adjusted to obtain R third parameter sets; R is a positive integer, and each third parameter set includes parameter values ​​of the C undetermined layer number parameters and the target sample parameter; One or more third parameter sets satisfying the maximum product condition are determined as the fourth parameter set, the fourth parameter set containing the minimum target sample parameter is determined as the target transition parameter set, and the parameter values ​​of the target sample parameters in the target transition parameter set are determined as the transition parallel set. The corresponding transition sample input amount, the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set are determined as the transition parallel group The layer number parameter corresponding to each pipeline parallel group respectively; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches the maximum value.

11. The method according to claim 1, characterized in that: Also includes: Based on the sample input amounts corresponding to the D data parallel groups, the training sample set is divided to obtain training sample groups corresponding to each data parallel group; Based on the training sample group, model training is performed in parallel on the initial model configured for each data parallel group to obtain sub-model parameters corresponding to each data parallel group, and based on the sub-model parameters corresponding to each data parallel group, a target model is generated.

12. A data processing device, characterized in that: include: A chip acquisition module, used to acquire M initial chip sets; M is a positive integer, the chip models in an initial chipset are all the same, and the chip models between each initial chipset are different; A pipeline parallel grouping module, used to group the chips in each initial chipset based on the learning parameter amount corresponding to a single network layer in the initial model and the chip models of the M initial chipsets to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the learning parameter amount; A data parallel grouping module is used to group and combine the P pipeline parallel groups based on load balancing conditions, the average computing power coefficients of the P pipeline parallel groups, the number of network layers of the initial model and the group computing power coefficients corresponding to the P pipeline parallel groups, so as to obtain D data parallel groups and the sample input amounts corresponding to the D data parallel groups respectively; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total learning parameter amount of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to perform model training in parallel on the initial model configured on the corresponding data parallel groups.

13. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, which is stored in a computer-readable storage medium and is suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11.

Images