Model training method and device

Abstract

The present application discloses a model training method and device thereof, belonging to the field of artificial intelligence. The method comprises: determining a first video memory, which is the video memory required for training a first model; determining a first quantity according to the first video memory and a second video memory, wherein the second video memory is the video memory of a single graphics card that can be used for training the first model, and the first quantity is the number of graphics cards required for training the first model; according to the first video memory and the first quantity, splitting the first model into a first number of training stages, wherein each training stage is trained by a graphics card, and the video memory of each graphics card matches the video memory required for the corresponding training stage; and training the corresponding training stage of each graphics card respectively.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and specifically relates to a model training method and device thereof. Background Art

[0002] With the development of artificial intelligence technology, multimodal big models have emerged to process multiple different types of data at the same time. Multimodal big models can process two or more types of data to provide more comprehensive and accurate analysis and decision-making.

[0003] At present, the multimodal large model is obtained by parallel training of multiple computing units (such as graphics cards). Specifically, when training the multimodal large model, the largest sub-model (hereinafter referred to as the largest sub-model) in the initial multimodal large model can be evenly split into multiple training stages (Stage) according to the number of layers of the repeated structure (Layer), and then the multiple stages are loaded through different graphics cards to train the multimodal large model.

[0004] However, when splitting the sub-models, the Stage is evenly distributed according to the number of layers of the repeated structure of the largest sub-model, which will cause the size of the Stage to be uneven, resulting in too much or too little video memory used to load the Stage. This will cause a waste of video memory and computing power when training a large multi-modal model. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a model training method and device thereof, which can avoid the waste of video memory and computing power when training large multimodal models.

[0006] In a first aspect, an embodiment of the present application provides a model training method, the method comprising: determining a first video memory, the first video memory being the video memory required for training a first model; determining a first quantity based on the first video memory and a second video memory, the second video memory being the video memory of a single graphics card that can be used to train the first model, the first quantity being the number of graphics cards required for training the first model; splitting the first model into a first number of training stages based on the first video memory and the first quantity, each training stage being trained by a graphics card, the video memory of each graphics card matching the video memory required for the corresponding training stage; and training the corresponding training stages based on each graphics card respectively.

[0007] In a second aspect, an embodiment of the present application provides a model training device, which includes: a calculation module, a determination module, a splitting module and an execution module. The calculation module is used to calculate the first video memory, which is the video memory required for training the first model. The determination module is used to determine a first quantity based on the first video memory and the second video memory obtained by the calculation module. The second video memory is the video memory that a single graphics card can use to train the first model, and the first quantity is the number of graphics cards required for training the first model. The splitting module is used to split the first model into a first number of training stages based on the first video memory and the first quantity determined by the determination module, each training stage is trained by a graphics card, and the video memory of each graphics card matches the video memory required for the corresponding training stage. The execution module is used to train the corresponding training stage of each graphics card separately.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the electronic device can determine the first video memory required for training the first model, and then determine the first number of graphics cards required for training the first model based on the first video memory and the second video memory of a single graphics card that can be used to train the first model. Then, based on the first video memory and the first number, the first model can be split into a first number of training stages, each training stage is trained by a graphics card, the video memory of each graphics card matches the video memory required for the corresponding training stage, and the corresponding training stages are trained based on each graphics card. In this solution, since the electronic device can determine the total video memory required for training the first model, that is, the first video memory, and determine the appropriate number of graphics cards required for training the first model, that is, the first number, based on the total video memory and the video memory of a single graphics card that can be used to train the first model, it can reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model due to an inappropriate number of graphics cards required for training the first model. Furthermore, since the electronic device can split the first model into an appropriate number of training stages based on the total video memory and the appropriate number, and the video memory required for each graphics card to train each training stage matches, the problem of the graphics card occupying too much or too little video memory when training the first model can be further reduced, thereby avoiding the waste of video memory and computing power when training large multimodal models. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flow charts of the model training method provided in the embodiment of the present application;

[0014] Figure 2 is a schematic diagram of the model structure provided in the embodiment of the present application;

[0015] Figure 3 This is the second flowchart of the model training method provided in the embodiment of the present application;

[0016] Figure 4 This is the third flow chart of the model training method provided in the embodiment of the present application;

[0017] Figure 5 This is the fourth flowchart of the model training method provided in the embodiment of the present application;

[0018] Figure 6 is a schematic diagram of the training phase of the first model provided in an embodiment of the present application;

[0019] Figure 7 This is the fifth flow chart of the model training method provided in the embodiment of the present application;

[0020] Figure 8 is a schematic diagram of a training phase of loading using a graphics card provided in an embodiment of the present application;

[0021] Fig. 9is a schematic diagram of forward calculation provided by an embodiment of the present application;

[0022] Fig.10 is a schematic diagram of reverse calculation provided in an embodiment of the present application;

[0023] Fig.11 is a schematic diagram of a model training device provided in an embodiment of the present application;

[0024] Fig.12 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Fig.13 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The terms "at least one (item)", "at least one of" and the like in this application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0029] The model training method and device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0030] The embodiments of the present application can be applied to the scenario of parallel training of a multimodal large model through at least one graphics card. Specifically, when performing pipeline parallel training on a multimodal large model, due to the large scale of the multimodal large model, the computing power and video memory of a single graphics card are difficult to train the complete multimodal large model, and multiple graphics cards need to be used for joint training, that is, the multimodal large model needs to be split into at least one training stage, so that one training stage can be trained separately in each graphics card in at least one graphics card.

[0031] For example, the multimodal big model can be a medical image diagnosis multimodal big model, an intelligent traffic monitoring multimodal big model, a multi-language intelligent translation multimodal big model, etc. Among them:

[0032] (1) The multimodal large model of medical imaging diagnosis can process medical imaging data through image modality to identify information such as lesion areas and organ structures. It can also process patients' medical records through text modality, including text content such as symptom descriptions, medical history, and family medical history.

[0033] (2) The intelligent traffic monitoring multimodal large model can be used to analyze the video footage captured by traffic monitoring cameras through the image modality to identify visual elements such as vehicle type, license plate number, traffic signs, and road conditions. It can also collect and analyze sound information in traffic scenes through the audio modality, such as vehicle horns, engine sounds, and emergency brake sounds.

[0034] (3) The multi-language intelligent translation multi-modal large model can recognize handwritten text or text content in pictures through image modality, process the input source language text content through text modality, and process user input voice through audio modality.

[0035] The embodiment of the present application provides a model training method and device thereof. Since the electronic device can determine the total video memory required for training the first model, i.e., the first video memory, and determine the appropriate number of graphics cards required for training the first model, i.e., the first number, based on the total video memory and the video memory of a single graphics card that can be used to train the first model, it can reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model due to the inappropriate number of graphics cards required for training the first model. Furthermore, since the electronic device can split the first model into an appropriate number of training stages based on the total video memory and the appropriate number, and the video memory required for each graphics card to train each training stage matches, it can further reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model, thereby avoiding the waste of video memory and computing power when training a large multi-modal model.

[0036] The execution subject of the model training method provided in the embodiment of the present application may be a model training device, and the model training device may be an electronic device, or a functional module or functional entity in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0037] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown. Figure 1 As shown, the model training method provided in the embodiment of the present application may include the following steps 201 to 204.

[0038] Step 201: The electronic device determines a first video memory.

[0039] In the embodiment of the present application, the first video memory is the video memory required for training the first model.

[0040] In an embodiment of the present application, the first model is a multimodal large model. Modality is the type of input or output that the model can process. For example, text modality, image modality, audio modality, etc. The model is divided into a unimodal large model and a multimodal large model, wherein the input data and output data of the unimodal large model are both a single modality, such as the input and output of the large language model are both text data, that is, only text modality, while the input data and output data of the multimodal large model include two or more modalities, such as the visual understanding multimodal large model, whose input data is text data and image data, and output data is text data, including image modality and text modality.

[0041] It should be noted that the multimodal large model is composed of multiple sub-models, and each sub-model has its own model structure. The present application embodiment takes the model structure of the visual understanding multimodal large model as an example to illustrate the model training method provided in the present application embodiment.

[0042] In the embodiment of the present application, the above-mentioned video memory is the storage capacity for training the first model, usually in gigabytes (GB), 1GB = 1024*1024*1024 = 1073741824 bytes.

[0043] In the embodiment of the present application, the graphics card is a graphics processing unit (GPU).

[0044] Optionally, in an embodiment of the present application, in addition to a graphics card, the electronic device may also train the first model based on other computing units, such as a tensor processing unit (TPU), a field programmable gate array (FPGA), etc.

[0045] It should be noted that the scale of multimodal large models is usually large, and the video memory of a single graphics card often cannot accommodate a complete multimodal large model. In this case, pipeline parallel training is usually used for training. Pipeline parallel training is: split the large model into multiple parts according to the number of layers, and each part is distributed in multiple computing devices during training. The calculation results of a single computing device are transmitted between multiple computing devices through the network, so as to realize the training of the entire large model.

[0046] In the embodiment of the present application, in order to reasonably split the first model into multiple parts according to the number of layers so that each part can be trained in different graphics cards, the electronic device needs to calculate the first video memory. Specifically, the electronic device must first determine the model structure of the first model, such as the structure and connection relationship of each layer such as the input layer, the repeated layer, the connection layer, and the output layer, and then combine the model configuration of the first model, such as model parameters and other related information, to calculate the video memory required for training each model layer, so as to accurately calculate the first video memory, so as to better plan the distribution of the first model in different graphics cards to achieve efficient model training.

[0047] Optionally, in an embodiment of the present application, the electronic device can determine the model structure of the first model, that is, the structure and connection relationship of each layer of the first model, such as the input layer, repeated layer, connection layer, and output layer, and calculate the video memory required for training each model layer according to the model structure and the model configuration of the first model.

[0048] For example, Figure 2 As shown, it is assumed that the first model is a large multimodal model for visual understanding, which includes three sub-models. The forward calculation order of the three sub-models is model 1, model 2, and model 3 from left to right. Model 1 is the encoder of Transformer. The electronic device can define the input embedding layer of model 1 as the input layer 10, and define the layers of the repeated structure in model 1 as the 1st to the xth layers. Model 2 is composed of multiple linear layers. The electronic device can define the multiple linear layers as the connection layer 11. Model 3 is the decoder of Transformer. The electronic device can define the input embedding of model 3 as the input layer 12, and define the layers of the repeated structure of model 3 as the 1st to the yth layers, and the linear layer as the output layer 13. Then the electronic device can calculate the video memory required for the training of the six types of model layers respectively; x and y are both positive integers.

[0049] Optionally, in the embodiment of the present application, the first model includes M model parameters and T activation values, where M and T are both positive integers. Figure 1 ,like Figure 3 As shown, the above step 201 can be specifically implemented through the following steps 201a and 201b.

[0050] Step 201a: The electronic device calculates the sixth video memory and the seventh video memory.

[0051] In the embodiment of the present application, the sixth video memory is the video memory required for calculating the M model parameters, and the seventh video memory is the video memory required for calculating the T activation values.

[0052] Optionally, in the embodiment of the present application, the above model parameters can be various set values ​​related to the model structure, such as the number of layers, neuron dimensions, attention head related values, vocabulary size, etc. These values ​​determine the structure, scale and data processing capabilities of the model, and play a key role in calculating video memory usage, model training and operation.

[0053] Optionally, in an embodiment of the present application, the electronic device may first determine the parameter amounts of M model parameters, then define the number of bytes to be occupied by each parameter, and then calculate the sixth video memory based on the parameter amounts of the M model parameters and the number of bytes to be occupied by each parameter.

[0054] In the embodiment of the present application, the above activation value is the intermediate calculation result of the model that needs to be retained during the forward calculation of the model in order to complete the backward calculation.

[0055] Optionally, in an embodiment of the present application, the electronic device may first determine the number of T activation values, then define the number of bytes each activation value will occupy, and then calculate the seventh video memory based on the number of T activation values ​​and the number of bytes each activation value will occupy.

[0056] Optionally, in the embodiment of the present application, the above step 201a can be specifically implemented by the following steps 201a1 and 201a2.

[0057] Step 201a1: The electronic device calculates parameter quantities of M model parameters.

[0058] In the embodiment of the present application, the above-mentioned parameter amount is the number of model parameters during the first model training.

[0059] Optionally, in an embodiment of the present application, the electronic device can obtain the parameter quantity of the model by multiplying the dimensions in the tensor dimension. The tensor dimension is an attribute used to determine the shape of the tensor, which is used to represent the size of the tensor in each direction. A tensor is a multidimensional array whose dimension represents the size of each dimension of the multidimensional array.

[0060] For example, assume that the two dimensions of an Input Embedding are V and H, V represents the size of the vocabulary, that is, there are V different input symbols, such as words, and H represents the dimension size of the vector after each input symbol is mapped. Assuming that there are 10,000 words in the vocabulary, that is, V = 10,000, and each word is mapped to a 100-dimensional vector, that is, H = 100, then the tensor dimension of this Input Embedding is [10000, 100]. For Input Embedding, the parameter amount is calculated by multiplying the two dimensions. This is because from the perspective of the mathematical model, Input Embedding can be regarded as a weight matrix. The number of rows of this matrix is ​​V, corresponding to each symbol in the vocabulary, and the number of columns is H1, corresponding to the vector dimension after each symbol is mapped. Each element in the matrix is ​​a learnable parameter, so the total number of parameters is V×H. So, in this example, the number of parameters is 10,000×100=1,000,000 parameters.

[0061] Optionally, in an embodiment of the present application, the electronic device may calculate the parameter quantities in each model layer of the first model, and add these parameter quantities to obtain the parameter quantities of M first models.

[0062] Exemplarily, it is assumed that the first model is a large multimodal model of visual understanding, which includes three sub-models, and the forward calculation order of the three sub-models is model 1, model 2, and model 3 from left to right.

[0063] (1) Assume that the model configuration of Model 1 is: the number of repeated structure layers is L1, the hidden layer dimension is H1, the intermediate layer parameters are I1, the number of attention heads is A1, the attention head dimension is D1, the vocabulary size is V1, the training data sequence length is S1, and the training data batch size is B1.

[0064] (2) Assume that the model configuration of Model 3 is: the number of repeated structure layers is L3, the hidden layer dimension is H3, the intermediate layer parameters are I3, the number of attention heads is A3, the attention head dimension is D3, the vocabulary size is V3, the training data sequence length is S3, and the training data batch size is B3.

[0065] Then the electronic device can respectively calculate the parameter quantities of the input layer of model 1, the layer of model 1, the connection layer of model 2, the input layer of model 3, the layer of model 3, and the output layer of model 3. Among them:

[0066] 1) The input layer of model 1, that is, the tensor dimension of Input Embedding is [V1, H1], so the number of parameters in the input layer of model 1 is: V1·H1=V1H1.

[0067] 2) The layer of model 1 can be divided into two modules: Attention and Multi-Layer Perceptron (MLP). The Attention module has 4 tensor dimensions [H1, H1] and two scalars [H1]. A scalar is a single value. The number of parameters in the Attention module of model 1 is: 4·H1·H1+2·H1=4H1 2 +2H1.

[0068] There are 3 tensor dimensions [I1, H1] in the MLP module, so the number of parameters in the MLP module of model 1 is: 3·I1·H1=3H1I1.

[0069] Therefore, the total number of Layer parameters is: 4H1 2 +2H1+3H1I1.

[0070] 3) The connection layer consists of two Linear layers. The first Linear layer has a tensor [H 3， H1] and a scalar [H3], and the second Linear has a tensor [H 3， H3] and 1 scalar [H3], then the total number of parameters in the connection layer is: H3·H1+H3·H3+2·H3=H3 2 +H3H1+2H3.

[0071] 4) The input layer of model 3, that is, the tensor dimension of Input Embedding is [V3, H3], so the number of parameters in the input layer of model 3 is: V3·H3=V3H3.

[0072] 5) The Layer of model 3 can be divided into two modules: Attention and MLP. The Attention module has four tensor dimensions [H3, H3] and two scalars [H3]. The number of parameters in the Attention module of model 3 is: 4·H3·H3+2·H3=4H3 2 +2H3.

[0073] There are 3 tensor dimensions [I3, H3] in the MLP module, so the number of parameters in the MLP module of model 3 is: 3·I3·H3=3H3I3.

[0074] Therefore, the total number of Layer parameters is: 4H3 2 +2H3+3H3I3

[0075] 6) The tensor dimension of the Linear in the output layer of model 3 is [V3, H3], so the number of parameters in model 3 is: V3·H3=V3H3.

[0076] In summary, the number of parameters of the above multimodal model of visual understanding (i.e., the above M) is V1H1+(4H1 2 +2H1+3H1I1)·L1+H3 2 +H3H1+2H3+V3H3+(4H3 2 +2H3)·L3+V3H3.

[0077] Step 201a2: The electronic device determines the product of the parameter amount of the M model parameters and the first byte number as the sixth display memory, and calculates the seventh display memory.

[0078] In the embodiment of the present application, the above-mentioned first number of bytes is the number of bytes required to calculate each model parameter.

[0079] Optionally, in the embodiment of the present application, the first byte number may be a default number of the electronic device or a preset number of the user. For example, the first byte number may be 2 bytes, 4 bytes, or 8 bytes. Specifically, it may be determined according to actual use requirements such as the data type of the model parameter, the storage characteristics of the electronic device, and the requirements of the training algorithm for accuracy, and the embodiment of the present application does not limit this.

[0080] For example, assuming that the first byte number is F1, the number of parameters of the visual understanding multimodal large model is V1H1+(4H1 2 +2H1+3H1I1)·L1+H3 2 +H3H1+2H3+V3H3+(4H3 2 +2H3)·L3+V3H3, then the video memory required to calculate the parameters of the visual understanding multimodal large model is: [V1H1+(4H1 2 +2H1+3H1I1)·L1+H3 2 +H3H1+2H3+V3H3+(4H3 2 +2H3)·L3+V3H3]·F1 byte.

[0081] In this way, since the electronic device can calculate the parameter amounts of all model parameters included in the first model, that is, the parameter amounts of M model parameters, and the electronic device can know the first number of bytes required to calculate each model parameter, the electronic device can obtain the accurate sixth video memory based on the product of the parameter amounts of all model parameters and the first number of bytes, and thus the electronic device can accurately determine the first video memory based on the accurate sixth video memory and the seventh video memory. Furthermore, in subsequent steps, the electronic device can accurately determine the number of graphics cards required to train the first model and split the first model into the first number of training stages based on the accurate first video memory, so that the problem of the graphics card occupying too much or too little video memory when training the first model can be further reduced.

[0082] Optionally, in the embodiment of the present application, the above step 201a can be specifically implemented by the following steps 201a3 and 201a4.

[0083] Step 201a3: The electronic device calculates the sixth video memory and the number of T activation values.

[0084] It should be noted that, for the description of calculating the sixth video memory of the electronic device, reference may be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0085] Optionally, in an embodiment of the present application, the electronic device may calculate the number of activation values ​​in each layer of the first model, and add the numbers of these activation values ​​to obtain the number of activation values ​​of the first model.

[0086] Exemplarily, assume that the first model is a large multimodal model for visual understanding, which includes three sub-models. The forward calculation order of the three sub-models is model 1, model 2, and model 3 from left to right. The model configuration of each model is described in the above example and will not be repeated here. The activation values ​​mainly exist in the Layer of model 1 and the Layer of model 3. The electronic device can calculate the number of activation values ​​in the Layer of model 1 and the Layer of model 3 respectively. The layer can be divided into two modules: Attention and MLP, where:

[0087] The calculation formula of the Attention module is as follows: Formula (1) to Formula (5):

[0088] Q=x·W q Formula (1)

[0089] K=x·W k Formula (2)

[0090] V=x·W v Formula (3)

[0091] Where x represents the input vector, Q is the query vector, K is the key vector, and V is the value vector. q , W k , W v are respectively learnable weight matrices, where W q is the weight matrix used to transform the input vector x into the query vector Q, W k is the weight matrix that transforms the input vector x into the key vector K, W v is the weight matrix that transforms the input vector x into the value vector V.

[0092]

[0093] Where d is the dimension of the K vector. Dividing the transposed dot product of Q and K by d is a scaling operation that helps optimize the calculation of the attention score and avoid values ​​that are too large or too small. Softmax is an activation function that converts The calculation result of is converted into a probability distribution. Dropout is a regularization technique. The Dropout operation is to prevent overfitting and improve the generalization ability of the model. self It represents the output vector after the self-attention mechanism is processed, which is the result of a series of calculations on Q, K, and V.

[0094] The above formula (4) first calculates the dot product of Q and K transpose and scales it (divided by ), and then convert the result into attention weight (probability distribution) through the softmax function, and then perform Dropout operation to increase the generalization ability of the model, and finally use the obtained result to perform weighted summation of V vector to obtain x self .

[0095] x attn =LN[Dropout(x self ·w o )+x] Formula (5)

[0096] Among them, w o is a learnable weight matrix. During the model training process, the weight matrix will be continuously adjusted to self Perform a linear transformation to adapt to the overall optimization goal of the model. x is the original input vector. Dropout is a regularization operation that randomly sets the output of the neuron to 0 according to a certain probability. self ·w o Dropout operation can be performed on the result to prevent overfitting and increase the generalization ability of the model. LN stands for layer normalization, which is a normalization technique used to normalize the input of a certain layer in the neural network, making the input distribution of each layer more stable, which helps to accelerate the training process of the model and improve the performance of the model.

[0097] The process of calculating the number of activation values ​​of the Attention module of the Layer in model 1 based on the calculation formula of the Attention module is as follows:

[0098] Q=x·W q : The dimension is [B1,S1,H1], and the number of activation values ​​= B1S1H1.

[0099] K=x·W k: The dimension is [B1,S1,H1], and the number of activation values ​​= B1S1H1.

[0100] V=x·W v : The dimension is [B1,S1,H1], and the number of activation values ​​= B1S1H1.

[0101] Q.K T : The dimension is [B1,A1,S1,S1], the number of activation values ​​= B1 A1 S1 2 .

[0102] The dimension is [B1,A1,S1,S1], and the number of activation values ​​= B1A1 S1 2 .

[0103] The dimension is [B1,A1,S1,S1], and the number of activation values ​​= B1A1S1 2 .

[0104] The dimension is [B1,S1,H1], and the number of activation values ​​= B1S1H1.

[0105] x self ·w o : The dimension is [B1,S1,H1], and the number of activation values ​​= B1S1H1.

[0106] Dropout(x self ·w o ): dimension is [B1,S1,H1], number of activation values ​​= B1S1H1.

[0107] LN[Dropout(x self ·w o )+x]: dimension is [B1,S1,H1], number of activation values ​​= B1S1H1.

[0108] The calculation formula of the MLP module is as follows:

[0109] x fcn =GeLU(x attn ·w fc1 )·w fc2 Formula (6)

[0110] Among them, x fcn It is the feature vector after the Fully-Connected Network (FCN) related operations. fc1 and w fc2is the weight matrix in the fully connected layer. GeLU is the Gaussian Error Linear Unit, which is an activation function.

[0111] x o =LN[Dropout(x fcn )+x attn ] Formula (7)

[0112] Among them, x o It is an intermediate variable or output variable in a neural network, representing the final result after a series of operations or the intermediate result of further processing.

[0113] The process of calculating the number of activation values ​​of the MLP module of Layer in model 1 based on the calculation formula of the MLP module is as follows:

[0114] x attn ·w fc1 : Dimension is [B1,S1,4H1], number of activation values ​​= 4B1S1H1.

[0115] GeLU(x attn ·w fc1 ): dimension is [B1,S1,H1], number of activation values ​​= 4B1S1H1.

[0116] x fcn =GeLU(x attn ·w fc1 )·w fc2 : The dimension is [B1,S1,H1], and the number of activation values ​​= B1S1H1.

[0117] Dropout(x fcn ): dimension is [B1,S1,H1], number of activation values ​​= B1S1H1.

[0118] x o =LN[Dropout(x fcn )+x attn ]: Dimension is [B1,S1,H1], number of activation values ​​= B1S1H1.

[0119] In summary, the number of activation values ​​of Layer in Model 1 is: B1S1H1+B1S1H1+

[0120] B1S1H1+B1 A1 S1 2 +B1 A1 S1 2 +B1 A1 S1 2+B1S1H1+B1S1H1+B1S1H1+4B1S1H1+4B1S1H1+B1S1H1+B1S1H1+B1S1H1+B1S1H1=17B1S1H1+3B1 A1 S1 2 Similarly, the number of activation values ​​of Layer in Model 2 is: 17B2S2H2+3B2 A2 S2 2 , then the number of activation values ​​in the above multimodal model of visual understanding is 17B1S1H1+3B1 A1S1 2 +17B2S2H2+3B2A2 S2 2 .

[0121] Step 201a4: The electronic device determines the product of the number of T activation values ​​and the second byte number as the seventh video memory.

[0122] In the embodiment of the present application, the above-mentioned second number of bytes is the number of bytes required to calculate each activation value.

[0123] Optionally, in the embodiment of the present application, the second byte number may be a default number of the electronic device or a preset number of the user. For example, the second byte number may be 2 bytes, 4 bytes, or 8 bytes. Specifically, it may be determined according to actual use requirements such as the data type of the model parameter, the storage characteristics of the electronic device, and the requirements of the training algorithm for accuracy, and the embodiment of the present application does not limit it.

[0124] For example, assuming that the second byte number is F2, the number of activation values ​​of the visual understanding multimodal large model is 17B1S1H1+3B1A1S1 2 +17B2S2H2+3B2 A2 S2 2 , then the video memory required to calculate the activation value of the visual understanding multimodal large model is: [17B1S1H1+3B1 A1 S1 2 +17B2S2H2+3B2 A2 S2 2 ]·F2 bytes.

[0125] In this way, since the electronic device can calculate the number of all activation values ​​included in the first model, that is, the number of T activation values, and the electronic device can know the number of second bytes required to calculate each activation value, the electronic device can obtain the accurate seventh video memory based on the product of the number of all activation values ​​and the second byte number, and thus the electronic device can accurately determine the first video memory based on the accurate sixth video memory and the seventh video memory. Furthermore, in subsequent steps, the electronic device can accurately determine the number of graphics cards required to train the first model and split the first model into the first number of training stages based on the accurate first video memory, so that the problem of the graphics card occupying too much or too little video memory when training the first model can be further reduced.

[0126] Step 201b: The electronic device determines the sum of the eighth video memory and the seventh video memory as the first video memory.

[0127] In the embodiment of the present application, the eighth video memory and the sixth video memory are in a multiple relationship.

[0128] Optionally, in the embodiment of the present application, the multiple relationship between the eighth video memory and the sixth video memory may be 1.5 times, 2 times, 3 times, etc., which is specifically determined by factors such as the storage method of the model parameters and the influence of the parameter algorithm. Usually, the multiple relationship between the eighth video memory and the sixth video memory may be 2 times.

[0129] Optionally, in the embodiment of the present application, in the process of training the first model, in addition to the calculation of model parameters and activation values, the calculation of the gradients and optimizer states corresponding to the parameters also requires the use of video memory. Among them, the video memory occupied by the gradients is the same as the video memory occupied by the parameters, that is, it is also the sixth video memory, and the optimizer state will be evenly distributed to different graphics cards in actual training, so this item can be temporarily ignored. Therefore, the total video memory is twice the sum of the sixth video memory and the seventh video memory.

[0130] For example, assume that the memory required for the parameters of the large multimodal model for visual understanding is: [V1H1+(4H1 2 +2H1+3H1I1)·L1+H3 2 +H3H1+2H3+V3H3+(4H3 2 +2H3)·L3+V3H3]·F1 bytes, the video memory required to calculate the activation value of the visual understanding multimodal large model is: [17B1S1H1+3B1 A1 S1 2 +17B2S2H2+3B2 A2 S2 2 ]·F2 bytes, the total video memory required to train the visual understanding multimodal large model is 2[V1H1+(4H1 2 +2H1+3H1I1)·L1+H3 2 +H3H1+2H3+V3H3+(4H3 2 +2H3)·L3+V3H3]·F 1+ [17B1S1H1+3B1 A1 S1 2 +17B2S2H2+3B2 A2 S2 2 ]·F2 byte, i.e. the first video memory.

[0131] In this way, since the electronic device can respectively calculate the sixth video memory required for parameter calculation of the first model and the seventh video memory required for activation value calculation of the first model, and the electronic device can know that in addition to the model parameters and activation values ​​occupying the video memory during the model training process, the gradients corresponding to the parameters occupy the same amount of video memory as the parameters (i.e., equal to the sixth video memory), and the optimizer state will be evenly distributed to different GPU devices in actual training and this item can be temporarily ignored, the electronic device can obtain the accurate first video memory based on twice the sixth video memory plus the seventh video memory, so the electronic device can accurately determine the number of graphics cards required for training the first model and split the first model into a first number of training stages based on the accurate first video memory. In this way, the problem of the graphics card occupying too much or too little video memory when training the first model can be further reduced.

[0132] Step 202: The electronic device determines a first quantity according to the first video memory and the second video memory.

[0133] In an embodiment of the present application, the second video memory is the video memory of a single graphics card that can be used to train the first model, and the first number is the number of graphics cards required to train the first model.

[0134] Optionally, in an embodiment of the present application, the video memory of each graphics card that can be used for training the first model can be the same.

[0135] Optionally, in the embodiment of the present application, the electronic device may first determine a video memory that is a multiple of the second video memory based on the second video memory, and then determine the first quantity based on the size relationship between the video memory and the first video memory.

[0136] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 4 As shown, the above step 202 can be specifically implemented through the following steps 202a and 202b.

[0137] Step 202a: The electronic device determines the product of the second video memory and the second quantity as the third video memory.

[0138] In the embodiment of the present application, the second number mentioned above is the initial value of the number of graphics cards.

[0139] Optionally, in the embodiment of the present application, the second number may be a default number of the electronic device or a preset number of the user. For example, the second number may be 1 or 2. The specific number may be determined according to actual use requirements and is not limited in the embodiment of the present application.

[0140] Optionally, in an embodiment of the present application, the electronic device can calculate the memory size of the second number of graphics cards and determine it as the third video memory, so that the electronic device can determine whether the second number of graphics cards can train the first model by comparing the size relationship between the third video memory and the first video memory. In order to use as few graphics cards as possible for training while meeting the training requirements of the first model, the preset value of the second number needs to be set as small as possible, for example, the second number can be 1, which can reduce hardware costs and resource usage.

[0141] Step 202b: When the third video memory is greater than or equal to the first video memory, the electronic device determines the second quantity as the first quantity.

[0142] Optionally, in an embodiment of the present application, the electronic device may determine that a second number of graphics cards can train the first model when the third graphics memory is greater than or equal to the first graphics memory, thereby determining the second number as the first number, that is, using the second number of graphics cards to train the first model.

[0143] Exemplarily, assuming that the video memory size of each graphics card is 8GB, the second quantity is 1, and the video memory required to train the first model is 7GB, the electronic device can determine that the third video memory is 8GB, and judge that the third video memory is larger than the video memory required to train the first model, thereby determining that the first quantity is 1, that is, 1 graphics card can train the first model.

[0144] In this way, since the electronic device can determine the total video memory that can be provided by the second number of graphics cards, and the electronic device can obtain the video memory required for calculating the parameters of the first model by calculating the parameter amount and the first number of bytes of the model parameters of the first model, and can also calculate the video memory required for the activation value of the first model, thereby obtaining the total video memory required for training the first model. When the total available video memory can meet the total video memory required for training the first model, that is, when the second number of graphics cards meets the training requirements of the first model, the electronic device can use the second number of graphics cards to train the first model, thereby reducing the situation where the graphics card video memory is occupied too much or too little during the training of the first model.

[0145] Optionally, in an embodiment of the present application, the model training method provided in the embodiment of the present application also includes the following steps 301 and 302.

[0146] Step 301: When the third video memory is smaller than the first video memory, the electronic device determines the product of the second video memory and the third number as the fourth video memory.

[0147] In the embodiment of the present application, the third quantity mentioned above is a multiple of the second quantity.

[0148] Optionally, in an embodiment of the present application, when the third video memory is smaller than the first video memory, the electronic device may multiply the second quantity by a multiple to obtain a third quantity, and then multiply the second video memory by the third quantity to obtain the video memory of a third number of graphics cards, i.e., a fourth video memory, so that the electronic device can determine whether the third number of graphics cards can train the first model by comparing the size relationship between the fourth video memory and the first video memory.

[0149] Optionally, in an embodiment of the present application, the multiple relationship between the third quantity and the second quantity is variable, and the initial multiple relationship is n, that is, the third video memory is equal to n second video memories. n can be the default of the electronic device or pre-set by the user, for example, n=2. In the case where the electronic device determines that the third number of graphics cards cannot train the first model, the multiple relationship can be updated to 2n, that is, each time it is determined that the third number of graphics cards cannot train the first model, the multiple relationship is doubled, and then the new third number and the fourth video memory are calculated again, so that the electronic device can compare the size relationship between the new fourth video memory and the first video memory for the second time to determine whether the new third number of graphics cards can train the first model. If the new third number of graphics cards cannot train the first model, the multiple relationship will continue to be doubled until the fourth video memory is greater than or equal to the first video memory.

[0150] Exemplarily, assuming that the video memory of each graphics card is 8GB, the second number is 1, and the video memory required for training the first model is 28GB, the electronic device can determine that the third video memory is 8GB, and judge that the third video memory is less than the video memory required for training the first model. Assuming that the initial multiple relationship is 2, the electronic device can multiply the second number by 2 to obtain the third number 2, and calculate the fourth video memory as 2 times 8GB equal to 16GB. The electronic device can then determine that the fourth video memory is less than the first video memory, and the electronic device can update the multiple relationship to 4, and then calculate the third number as 4, and the fourth video memory as 4 times 8GB equal to 32GB. The electronic device can then determine that the fourth video memory is greater than the first video memory, thereby determining 4 as the first number, that is, 4 graphics cards can train the first model.

[0151] Optionally, in an embodiment of the present application, the multiple relationship between the third number and the second number may be 1.5 times, 2 times, 3 times, etc.

[0152] Optionally, in the embodiment of the present application, usually, the multiple relationship between the third number and the second number may be 2 times, for the following reasons:

[0153] (1) Because the video memory capacity in the graphics card is often allocated and managed in powers of 2 in hardware design and storage management. For example, common video memory capacities may be 2GB, 4GB, 8GB, etc. This video memory capacity characteristic based on multiples of 2 makes it easier to match the actual allocation and management of video memory when adjusting the number of graphics cards in multiples of 2, making it more convenient and efficient to calculate the total video memory of the graphics card (such as the fourth video memory), and more in line with the storage management logic at the hardware level.

[0154] (2) Increasing the number of graphics cards by a multiple of 2 (i.e., increasing the third number by a multiple) can gradually explore the amount of resources required to train the first model while avoiding wasting computing resources as much as possible. If the multiple is set too small, for example, only one graphics card is added each time, it may take too many iterations to find the appropriate number of graphics cards; and if the multiple is set too large, it may result in too many graphics cards being allocated at one time, resulting in a waste of training resources. A multiple of 2 can increase the number of graphics cards by a relatively moderate amount in each iteration, thereby finding the number of graphics cards that meet the training requirements of the first model more quickly while avoiding the waste of training resources due to excessive increase.

[0155] Step 302: When the fourth video memory is greater than or equal to the first video memory, the electronic device determines the third quantity as the first quantity.

[0156] In an embodiment of the present application, when the third video memory is smaller than the first video memory, the electronic device may multiply the second number by a multiple to obtain a third number, and then multiply the second video memory by the third number to obtain the video memory of the third number of graphics cards, i.e., the fourth video memory. Thus, the electronic device can determine that the third number of graphics cards can train the first model when the fourth video memory is greater than or equal to the first video memory, thereby determining the third number as the first number.

[0157] In this way, electronic devices can determine the required number of graphics cards by dynamically adjusting the multiple relationship, which can more accurately match the video memory required for model training, avoid the problem of over-allocation or under-allocation of video memory resources due to a fixed multiple relationship, and improve the flexibility of model training.

[0158] Step 203: The electronic device divides the first model into a first number of training stages according to the first video memory and the first number.

[0159] In the embodiment of the present application, each training stage is trained by a graphics card, and the video memory of each graphics card matches the video memory required by the corresponding training stage.

[0160] It should be noted that the above-mentioned "matching" can be understood as: the video memory of each graphics card is the same as the video memory required for the corresponding training stage, or the difference between the video memory of any graphics card and the video memory required for the corresponding training stage and the video memory of graphics cards other than the any graphics card and the video memory required for the corresponding training stage is less than or equal to the difference threshold.

[0161] In the embodiment of the present application, the matching of the video memory of each graphics card with the video memory required for the corresponding training stage can be understood as: the video memory of each graphics card is close to the video memory required for training in the corresponding training stage, that is, the video memory of each graphics card is approximately averaged with the video memory required for training in the corresponding training stage.

[0162] Optionally, in the embodiment of the present application, the first model includes N model layers, where N is a positive integer. Figure 1 ,like Figure 5 As shown, the above step 203 can be specifically implemented through the following steps 203a and 203b.

[0163] Step 203a: The electronic device determines the ratio of the first video memory to the first quantity as the fifth video memory.

[0164] Optionally, in an embodiment of the present application, the electronic device may calculate the size of the video memory required for each training stage when the first model is divided into a first number of training stages on average, that is, the average video memory required for each training stage, that is, the fifth video memory mentioned above.

[0165] Exemplarily, assuming that the video memory required for training the first model is 28 GB and the first quantity is 4, the electronic device can determine the fifth video memory as the ratio of 28 GB to 4, that is, 7 GB.

[0166] Step 203b: The electronic device determines at least one continuous model layer among the N model layers, whose video memory required for calculation matches the fifth video memory, as a training stage.

[0167] In the embodiment of the present application, the video memory required for the above calculation is matched with the fifth video memory by continuously accumulating the above-mentioned at least one continuous model layer in the order of model calculation until the video memory required for the calculation of the at least one model layer is closest to the fifth video memory.

[0168] Optionally, in the embodiment of the present application, the electronic device may accumulate the video memory size required for the calculation of each layer in the above-mentioned at least one continuous model layer from the front to the back according to the calculation order of the model, so that the accumulated total video memory (i.e., the video memory required for the above-mentioned calculation) is close to the fifth video memory. When the accumulated video memory is closest to the fifth video memory, the at least one model layer is determined as a training stage.

[0169] For example, in combination Figure 2 ,like Figure 6 As shown, the electronic device divides the model layer in the visual understanding multimodal large model into four training stages, including a first training stage 14, a second training stage 15, a third training stage 16, and a fourth training stage 17, wherein the video memory of each training stage is close to the fifth video memory.

[0170] In this way, the electronic device can make the video memory requirement of each training stage close to the average level during the process of splitting the first model into training stages, thereby improving the overall utilization of video memory resources and avoiding the problem of uneven video memory usage.

[0171] Optionally, in the embodiment of the present application, combined with Figure 5 ,like Figure 7 As shown, the above step 203b can be specifically implemented through the following steps 203b1, 203b2 and 203b3, or the above step 203b can be specifically implemented through the following steps 203b1, 203b2 and 203b4.

[0172] Step 203b1: The electronic device determines, from the N model layers, X consecutive model layers whose absolute value of the first difference between the video memory required for calculation and the fifth video memory is less than or equal to a preset difference.

[0173] In the embodiment of the present application, X is a positive integer less than N.

[0174] Optionally, in the embodiment of the present application, the preset difference value may be a default value of the electronic device or a preset value of the user. For example, the preset difference value may be 1 GB. The specific value may be determined according to actual use requirements and is not limited in the embodiment of the present application.

[0175] Optionally, in the embodiment of the present application, the electronic device may start from the jth model layer and determine X consecutive model layers from the jth to the j+X-1th. Specifically:

[0176] (1) In the case of the first determination, the j-th model layer is the first model layer, that is, j=1, and the electronic device can determine the 1st to X-th model layers.

[0177] (2) In the case of the second determination, the j-th model layer is the next model layer of the last model layer in the model layers determined for the first time, that is, the X+1-th model layer. Then the electronic device can continue to start from the X+1-th model layer, accumulate the video memory required for the calculation of each model layer in sequence according to the order of the model layers, and continue to determine the model layers.

[0178] By analogy, each time a model layer is determined, the electronic device may start determining from the next model layer of the last model layer in the model layer determined last time.

[0179] In the embodiment of the present application, the first absolute value of the difference refers to the absolute value of the difference between the video memory required for the calculation of the above X model layers and the fifth video memory. Assuming that the video memory required for the calculation is V1 and the fifth video memory is V2, then the first absolute value of the difference |ΔV|=|V1-V2|. When calculating specifically, first calculate the value of V1-V2. If this value is a positive number, then it is the first absolute value of the difference; if this value is a negative number, then take its opposite as the first absolute value of the difference.

[0180] Step 203b2: The electronic device determines the second absolute value of the difference between the video memory required for calculating X-1 model layers and the fifth video memory.

[0181] In the embodiment of the present application, the above-mentioned X-1 model layers are model layers except the last model layer among the X model layers.

[0182] In the embodiment of the present application, the second absolute value of the difference refers to the absolute value of the difference between the video memory required for calculating X-1 model layers and the fifth video memory. The calculation method is: first calculate the difference between the video memory required for calculating X-1 model layers and the fifth video memory. If this value is a positive number, then it is the second absolute value of the difference; if this value is a negative number, then take its opposite as the second absolute value of the difference.

[0183] Step 203b3: When the absolute value of the first difference is greater than the absolute value of the second difference, the electronic device determines X model layers as one training stage.

[0184] Step 203b4: When the absolute value of the first difference is smaller than the absolute value of the second difference, the electronic device determines X-1 model layers as one training stage.

[0185] Optionally, in an embodiment of the present application, there may be a situation in which the data cannot be accurately accumulated to the fifth video memory during the actual segmentation process, that is, when not adding one more layer (i.e., the above X-1 model layers), the total video memory required for calculating at least one continuous model layer is less than the fifth video memory, and after adding one more layer (i.e., the above X model layers), the total video memory required for calculating at least one continuous model layer is greater than the fifth video memory. In this case, the electronic device can calculate the absolute value of the difference between the size of the video memory required for the accumulated calculation when adding one more layer and when not adding one more layer and the size of the fifth video memory. If the absolute value of the difference after adding one more layer is less than the absolute value of the difference when not adding one more layer, one more layer can be added, otherwise it will not be added. And so on, split all N model layers.

[0186] Exemplarily, it is assumed that the video memory required for calculation of each layer of the first model is: 5GB for the first layer, 3GB for the second layer, 3GB for the third layer, 6GB for the fourth layer, 3GB for the fifth layer, 3GB for the sixth layer, etc., and the fifth video memory is 10GB. The electronic device can start to accumulate from the first layer. When there is only the first layer, the total video memory is 5GB, which is less than 10GB; after adding the second layer, the total video memory is 5+3=8GB, which is less than 10GB; when adding the third layer, the total video memory becomes 8+3=11GB, which is greater than 10GB. At this time, the absolute value of the difference between the video memory size required for calculation when adding the third layer and when not adding the third layer and the fifth video memory is calculated. The absolute value of the difference value when not adding the third layer is |10-8|=2GB, and the absolute value of the difference value when adding the third layer is |11-10|=1GB. Since the absolute value of the difference value when adding the third layer is smaller, the third layer can be added according to the rule, and the first layer, the second layer, and the third layer are used as the first training stage. Then, a new accumulation starts from the 4th layer. When there is only the 4th layer, the total video memory is 6GB, which is less than 10GB; after adding the 5th layer, the total video memory is 6+3=9GB, which is less than 10GB. At this time, the total video memory with the 6th layer is 9+3=12GB, which is greater than 10GB. Calculate the absolute value of the difference between the video memory size required for the calculation when the 6th layer is added and when the 6th layer is not added and the fifth video memory. The absolute value of the difference value when the 6th layer is not added is |10-9|=1GB, and the absolute value of the difference value when the 6th layer is added is |12-10|=2GB. Since the absolute value of the difference value is smaller when the 6th layer is not added, the electronic device can not add the 6th layer, that is, the 4th and 5th layers are used as the second training stage. And so on, until all model layers of the first model are split.

[0187] In this way, the electronic device can make the video memory requirement of each training stage close to the average level during the process of splitting the first model into training stages, thereby improving the overall utilization of video memory resources and avoiding the problem of uneven video memory usage.

[0188] Step 204: The electronic device performs training on each graphics card in its corresponding training phase.

[0189] Optionally, in an embodiment of the present application, the electronic device may load each training stage in a graphics card in a calculation order.

[0190] For example, Figure 8 As shown, assuming that the electronic device divides the first model into four training stages, namely Stage 1, Stage 2, Stage 3, and Stage 4, the electronic device can use four graphics cards, such as GPU1, GPU2, GPU3, and GPU4, to load these four training stages respectively.

[0191] Optionally, in the embodiment of the present application, the electronic device can use these graphics cards to perform forward calculations for each training phase. The purpose of the forward calculation is to calculate the error between the actual output of the model and the expected output.

[0192] For example, in combination Figure 8 ,like Fig. 9 As shown, the electronic device can use GPU1 where Stage1 is located to start calculation, and perform matrix operation on the question part of the training data (including questions and answers) and the model layer of Stage1 to obtain the actual output Q1. After the calculation is completed, it needs to communicate with GPU2 where Stage2 is located twice. The first communication tells GPU2 the dimension of the calculation result Q1 to be transmitted, and the second communication transmits the calculation result Q1. After that, GPU2 can perform matrix operation on the model layer included in Stage2 and Q1 to obtain the actual output Q2. After the calculation is completed, GPU2 can communicate with GPU3 where Stage3 is located twice. The first communication tells GPU3 the dimension of the calculation result Q2 to be transmitted, and the second communication transmits the calculation result Q2. And so on and so forth, and finally it is transmitted to the last graphics card and the last training stage to perform matrix operation to obtain the calculation result Q. Then, a cross entropy loss function is used to calculate the error between the answer part in the training data and Q. The formula of the cross entropy loss function is the following formula (8):

[0193] H(p,q)=-∑ x p(x)log q(x) Formula (8)

[0194] Among them, q is the probability distribution of the model calculation result Q, and p is the probability distribution of the answer part in the training data.

[0195] Optionally, in an embodiment of the present application, the electronic device can perform reverse calculations on each training stage through these graphics cards. The reverse calculation can be based on the above error, that is, the calculation result H(p,q) of the cross entropy loss function, to obtain the gradient of the model parameters by taking partial derivatives of the model parameters in each stage.

[0196] For example, in combination Fig. 9 ,like Fig.10 As shown, GPU 4 where Stage 4 is located starts matrix operation to obtain the error H4 of the current Stage, and calculates the gradient G4 of the current Stage 4. The mathematical formula is the following formula (9):

[0197]

[0198] Where W is the model parameter of the current Stage. After the calculation is completed, it needs to communicate with GPU 3 where Stage 3 is located twice. The first communication tells GPU3 the dimension of the error to be transmitted, and the second communication transmits the error. Then GPU3 starts matrix operation to obtain the error H3 of Staeg3, and calculates the gradient of the current Stage as G3. And so on, it is finally transmitted to GPU 1 to complete the entire reverse calculation.

[0199] In the embodiment of the present application, after the gradient corresponding to the model parameter is calculated, the electronic device can update the model parameter once based on the learning rate strategy set during training, and the formula is the following formula (10):

[0200] W′=W+ηGW formula (10)

[0201] Among them, W is the model parameter before updating, η is the learning rate, G is the gradient corresponding to the model parameter, and W′ is the model parameter after updating.

[0202] Optionally, in an embodiment of the present application, the electronic device may repeatedly perform the process of forward calculation, reverse calculation and updating model parameters until all training data are completed and the final model parameters are obtained, that is, a trained multimodal large model can be obtained.

[0203] In an embodiment of the present application, the electronic device can calculate the size of each model layer in the repeated structure and non-repeated structure of the model in detail, and then decide how to allocate it to multiple stages, and make the size of each stage more uniform. In addition, the electronic device can not only split the largest sub-model in the multimodal large model, but also calculate whether the remaining sub-models need to be split according to the size of the model, and comprehensively calculate the structure and layer size of each sub-model to finally allocate which model layers each stage contains. Therefore, the video memory consumed by each computing device during training is more uniform, thereby avoiding the waste of video memory and computing power to the greatest extent, and regardless of the size of each sub-model, the pipeline parallel training of the entire multimodal large model can be achieved.

[0204] The method provided in the embodiment of the present application can realize pipeline parallel training of a multi-modal model composed of sub-models of any modality and any size on multiple GPUs, solving the problem that the model is too large to be trained on a single GPU. In addition, the method can make the video memory of each GPU more uniform and the computing power more fully utilized during pipeline parallel training on multiple GPUs, thereby improving training efficiency.

[0205] The embodiment of the present application provides a model training method, in which the electronic device can determine the first video memory required for training the first model, and then determine the first number of graphics cards required for training the first model based on the first video memory and the second video memory of a single graphics card that can be used to train the first model, and then the first model can be divided into a first number of training stages based on the first video memory and the first number, each training stage is trained by a graphics card, the video memory of each graphics card matches the video memory required for the corresponding training stage, and the training is performed on each graphics card for its corresponding training stage. Since the electronic device can determine the total video memory required for training the first model, that is, the first video memory, and determine the appropriate number of graphics cards required for training the first model, that is, the first number, based on the total video memory and the video memory of a single graphics card that can be used to train the first model, it can reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model due to an inappropriate number of graphics cards required for training the first model. Furthermore, since the electronic device can split the first model into an appropriate number of training stages based on the total video memory and the appropriate number, and the video memory required for each graphics card to train each training stage matches, the problem of the graphics card occupying too much or too little video memory when training the first model can be further reduced, thereby avoiding the waste of video memory and computing power when training large multimodal models.

[0206] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0207] The model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.

[0208] Fig.11 A possible structural diagram of a model training device involved in some embodiments of the present application is shown. Fig.11 As shown, the model training device 70 may include: a determination module 71, a splitting module 72 and an execution module 73.

[0209] The determination module 71 is used to determine a first video memory, where the first video memory is the video memory required for training the first model.

[0210] The determination module 71 is used to determine a first quantity based on a first video memory and a second video memory, where the second video memory is the video memory of a single graphics card that can be used to train the first model, and the first quantity is the quantity of graphics cards required to train the first model.

[0211] The splitting module 72 is used to split the first model into a first number of training stages according to the first video memory and the first number determined by the determination module 71, each training stage is trained by a graphics card, and the video memory of each graphics card matches the video memory required for the corresponding training stage.

[0212] The execution module 73 is used to perform training on each graphics card in its corresponding training phase.

[0213] In a possible implementation, the determination module 71 is specifically configured to: determine the product of the second video memory and the second quantity as the third video memory; and, when the third video memory is greater than or equal to the first video memory, determine the second quantity as the first quantity.

[0214] In a possible implementation, the determination module 71 is further used to: when the third video memory is smaller than the first video memory, determine the product of the second video memory and the third quantity as the fourth video memory, and the third quantity is a multiple of the second quantity; and when the fourth video memory is greater than or equal to the first video memory, determine the third quantity as the first quantity.

[0215] In a possible implementation, the first model includes N model layers, where N is a positive integer. The determination module 71 is specifically configured to: determine the ratio of the first video memory to the first quantity as the fifth video memory; and determine at least one model layer in the N model layers, in which the video memory required for calculation matches the fifth video memory, as a training stage.

[0216] In a possible implementation, the determination module 71 is specifically used to: determine, from the N model layers, X consecutive model layers whose first absolute value of the difference between the video memory required for calculation and the fifth video memory is less than or equal to a preset difference, where X is a positive integer less than N; determine the second absolute value of the difference between the video memory required for calculation of X-1 model layers and the fifth video memory, where the X-1 model layers are model layers other than the last model layer among the X model layers; when the absolute value of the first difference is greater than the absolute value of the second difference, determine the X model layers as one training stage; or, when the absolute value of the first difference is less than the absolute value of the second difference, determine the X-1 model layers as one training stage.

[0217] In an embodiment of the present application, a model training device is provided. Since the model training device can determine the total video memory required for training the first model, i.e., the first video memory, and determine the appropriate number of graphics cards required for training the first model, i.e., the first number, based on the total video memory and the video memory of a single graphics card that can be used to train the first model, it can reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model due to the inappropriate number of graphics cards required for training the first model. In addition, since the model training device can split the first model into an appropriate number of training stages based on the total video memory and the appropriate number, and the video memory required for each graphics card to train each training stage matches, it can further reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model, thereby avoiding the waste of video memory and computing power when training a large multi-modal model.

[0218] The model training device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personalcomputer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0219] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0220] The model training device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be described here.

[0221] Alternatively, if Fig.12As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores programs or instructions that can be executed on the processor 1001, and when the program or instructions are executed by the processor 1001, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0222] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0223] Fig.13 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0224] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0225] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.13 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0226] The processor 110 is used to determine a first video memory, where the first video memory is the video memory required for training the first model.

[0227] The processor 110 is used to determine a first quantity based on the obtained first video memory and second video memory, where the second video memory is the video memory of a single graphics card that can be used to train the first model, and the first quantity is the number of graphics cards required to train the first model.

[0228] The processor 110 is used to split the first model into a first number of training stages according to the first video memory and the first number, each training stage is trained by a graphics card, and the video memory of each graphics card matches the video memory required for the corresponding training stage.

[0229] The processor 110 is used to perform training on each graphics card in its corresponding training phase.

[0230] In a possible implementation, the processor 110 is specifically configured to: determine the product of the second video memory and the second quantity as the third video memory; and, when the third video memory is greater than or equal to the first video memory, determine the second quantity as the first quantity.

[0231] In a possible implementation, the processor 110 is further used to: when the third video memory is smaller than the first video memory, determine the product of the second video memory and the third quantity as the fourth video memory, and the third quantity is a multiple of the second quantity; and when the fourth video memory is greater than or equal to the first video memory, determine the third quantity as the first quantity.

[0232] In a possible implementation, the first model includes N model layers, where N is a positive integer. The processor 110 is specifically configured to: determine the ratio of the first video memory to the first quantity as the fifth video memory; and determine at least one continuous model layer in the N model layers, in which the video memory required for calculation matches the fifth video memory, as a training stage.

[0233] In a possible implementation, the processor 110 is specifically used to: determine, from the N model layers, X consecutive model layers whose first absolute value of the difference between the video memory required for calculation and the fifth video memory is less than or equal to a preset difference, where X is a positive integer less than N; determine the second absolute value of the difference between the video memory required for calculation of X-1 model layers and the fifth video memory, where the X-1 model layers are model layers other than the last model layer among the X model layers; when the absolute value of the first difference is greater than the absolute value of the second difference, determine the X model layers as one training stage; or, when the absolute value of the first difference is less than the absolute value of the second difference, determine the X-1 model layers as one training stage.

[0234] In an embodiment of the present application, a model training electronic device is provided. Since the electronic device can determine the total video memory required for training the first model, i.e., the first video memory, and determine the appropriate number of graphics cards required for training the first model, i.e., the first number, based on the total video memory and the video memory of a single graphics card that can be used to train the first model, it can reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model due to an inappropriate number of graphics cards required for training the first model. Furthermore, since the electronic device can split the first model into an appropriate number of training stages based on the total video memory and the appropriate number, and the video memory required for each graphics card to train each training stage matches, it can further reduce the problem of excessive or insufficient video memory occupied by the graphics card when training the first model, thereby avoiding the waste of video memory and computing power when training a large multi-modal model.

[0235] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and can achieve the same technical effect. To avoid repetition, it is not repeated here. The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, it is not repeated here.

[0236] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0237] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0238] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0239] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0240] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0241] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0242] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0243] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0244] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0245] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0246] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model training method, characterized in that: include: Determine a first video memory, where the first video memory is the video memory required for training the first model; Determine a first quantity according to the first video memory and the second video memory, where the second video memory is the video memory of a single graphics card that can be used to train the first model, and the first quantity is the quantity of graphics cards required to train the first model; According to the first video memory and the first number, splitting the first model into the first number of training stages, each training stage is trained by a graphics card, and the video memory of each graphics card matches the video memory required by the corresponding training stage; Based on each of the graphics cards, training is performed on the corresponding training phase.

2. The method according to claim 1, characterized in that: The determining a first quantity according to the first video memory and the second video memory includes: Determine the third video memory by multiplying the second video memory by a second quantity, wherein the second quantity is an initial value of the quantity of the video card; When the third video memory is greater than or equal to the first video memory, the second number is determined as the first number.

3. The method according to claim 2, characterized in that The method further comprises: In the case where the third video memory is smaller than the first video memory, the product of the second video memory and a third quantity is determined as the fourth video memory, and the third quantity is a multiple of the second quantity; When the fourth video memory is greater than or equal to the first video memory, the third number is determined as the first number.

4. The method according to claim 1, characterized in that The first model includes N model layers, where N is a positive integer; The step of dividing the first model into the first number of training stages according to the first video memory and the first number includes: Determine a ratio of the first video memory to the first quantity as a fifth video memory; At least one continuous model layer among the N model layers, for which the video memory required for calculation matches the fifth video memory, is determined as a training stage.

5. The method according to claim 4, characterized in that The step of determining at least one model layer in the N model layers whose required video memory for calculation matches the fifth video memory as a training stage comprises: From the N model layers, determine X consecutive model layers whose absolute value of the first difference between the video memory required for calculation and the fifth video memory is less than or equal to a preset difference, where X is a positive integer less than N; Determine a second absolute value of a difference between a video memory required for calculating X-1 model layers and the fifth video memory, wherein the X-1 model layers are model layers other than the last model layer in the X model layers; When the absolute value of the first difference is greater than the absolute value of the second difference, the X model layers are determined as one training stage; or, When the absolute value of the first difference is smaller than the absolute value of the second difference, the X-1 model layers are determined as one training stage.

6. A model training device, characterized in that: include: Identify modules, split modules, and execute modules; The determination module is used to calculate a first video memory, where the first video memory is the video memory required for training the first model; The determination module is further used to determine a first quantity according to the first video memory and the second video memory, where the second video memory is the video memory of a single graphics card that can be used to train the first model, and the first quantity is the quantity of graphics cards required to train the first model; The splitting module is used to split the first model into the first number of training stages according to the first graphics memory and the first number determined by the determination module, each of the training stages is trained by a graphics card, and the graphics memory of each graphics card matches the graphics memory required by the corresponding training stage; The execution module is used to perform training on the corresponding training phase of each graphics card respectively.

7. The device according to claim 6, characterized in that The determination module is specifically used for: The product of the second video memory and a second quantity is determined as the third video memory, where the second quantity is the initial value of the quantity of the graphics card; and when the third video memory is greater than or equal to the first video memory, the second quantity is determined as the first quantity.

8. The device according to claim 7, characterized in that The determining module is further used for: In the case where the third video memory is smaller than the first video memory, the product of the second video memory and a third quantity is determined as the fourth video memory, and the third quantity is a multiple of the second quantity; And, when the fourth video memory is greater than or equal to the first video memory, the third number is determined as the first number.

9. The device according to claim 6, characterized in that The first model includes N model layers, where N is a positive integer; The determination module is specifically used for: The ratio of the first video memory to the first number is determined as the fifth video memory; and at least one continuous model layer in the N model layers, in which the video memory required for calculation matches the fifth video memory, is determined as a training stage.

10. The device according to claim 9, characterized in that The determination module is specifically used for: From the N model layers, determine X consecutive model layers whose absolute value of the first difference between the video memory required for calculation and the fifth video memory is less than or equal to a preset difference, where X is a positive integer less than N; Determine a second absolute value of a difference between a video memory required for calculating X-1 model layers and the fifth video memory, wherein the X-1 model layers are model layers other than the last model layer in the X model layers; When the absolute value of the first difference is greater than the absolute value of the second difference, determining the X model layers as one training stage; or, When the absolute value of the first difference is smaller than the absolute value of the second difference, the X-1 model layers are determined as one training stage.

Images