Model training method and device, equipment, storage medium and program product

Abstract

The application discloses a model training method and device, equipment, a storage medium and a program product. The method comprises the following steps: in a forward process, performing type conversion on an input tensor of a second data type based on a scaling parameter corresponding to a linear layer, to obtain an input tensor of a first data type; after the linear layer completes calculation, converting an output tensor of the first data type into an output tensor of the second data type; in a reverse process, performing type conversion on a model parameter of the first data type based on the scaling parameter corresponding to the linear layer, to obtain a model parameter of the second data type; determining a gradient of the second data type based on the model parameter of the second data type and a loss value of the second data type; and updating the model parameter of the model by using the gradient of the second data type, to obtain a trained model. In this way, the model training efficiency can be improved, and the requirements for computing power and storage during model training are reduced while ensuring the model accuracy.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a model training method, apparatus, device, storage medium, and program product. Background Technology

[0002] Current deep learning models primarily utilize FP32 and FP16 data types for training. However, with the increasing number of model parameters, their computational resource consumption is also rising. While the latest graphics processing units (GPUs) support lower data types, reducing computational resource requirements and achieving higher training efficiency, their representation range and accuracy remain relatively low, typically preventing direct model training. Therefore, training larger-scale models with limited computational resources has become a key research direction in the industry. Summary of the Invention

[0003] In view of the above, embodiments of this application provide at least one model training method, apparatus, device, storage medium, and program product.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] On one hand, embodiments of this application provide a model training method. The model includes a linear layer, and the linear layer includes model parameters of a first data type. The method includes: in the forward pass, performing a type conversion on the input tensor of a second data type based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type; after the linear layer completes the calculation, converting the output tensor of the first data type into the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; in the backward pass, performing a type conversion on the model parameters of the first data type based on the scaling parameter corresponding to the linear layer to obtain the model parameters of the second data type; determining the gradient of the second data type based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained after completing the forward pass; and updating the model parameters of the model using the gradient of the second data type to obtain the trained model.

[0006] In some embodiments, the method further includes: determining a maximum input value based on an input tensor of a second data type; and determining a scaling parameter based on the maximum input value and the maximum value of the type corresponding to the first data type.

[0007] In some embodiments, the input tensor of the second data type is converted based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type, including: determining the target input tensor based on the scaling parameter and the input tensor of the second data type; and converting the target input tensor to obtain the input tensor of the first data type.

[0008] In some embodiments, the model parameters of the first data type are converted based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type, including: converting the model parameters of the first data type to obtain the target model parameters; and determining the model parameters of the second data type based on the target model parameters and the scaling parameters.

[0009] In some embodiments, updating the model parameters of the model using the gradient of the second data type to obtain the trained model includes: determining a target gradient based on the gradient of the second data type and scaling parameters; converting the target gradient to obtain a gradient of the first data type; and updating the model parameters of the model based on the gradient of the first data type to obtain the trained model.

[0010] In some embodiments, determining the gradient of the second data type based on the model parameters and loss value of the second data type includes: determining the current gradient based on the model parameters and loss value of the second data type; determining the current first-order momentum and the current second-order momentum based on the current gradient, first-order statistics, and second-order statistics; adjusting the current first-order momentum based on the first-order statistics to obtain an adjusted first-order momentum, and adjusting the current second-order momentum based on the second-order statistics to obtain an adjusted second-order momentum; and determining the gradient of the second data type based on the adjusted first-order momentum and the adjusted second-order momentum.

[0011] In some embodiments, the first data type is a floating-point type with a first precision, and the second data type is a floating-point type with a second precision, wherein the first precision is less than the second precision.

[0012] In some embodiments, the first data type is an 8-bit floating-point number FP8, and the second data type is a 16-bit floating-point number FP16 or a 32-bit floating-point number FP32.

[0013] In some embodiments, the initialized model parameters are of type FP32, and the method further includes: converting the initialized model parameters to type FP8 to obtain model parameters of type FP8.

[0014] On the other hand, embodiments of this application provide a model training apparatus. The model includes a linear layer, and the linear layer includes model parameters of a first data type. The apparatus includes: a first conversion module, used to perform type conversion on the input tensor of a second data type based on the scaling parameters corresponding to the linear layer during the forward pass, to obtain the input tensor of the first data type; after the linear layer completes the calculation, converting the output tensor of the first data type into the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; a second conversion module, used to perform type conversion on the model parameters of the first data type based on the scaling parameters corresponding to the linear layer during the backward pass, to obtain the model parameters of the second data type; determining the gradient of the second data type based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained after completing the forward pass; and an update module, used to update the model parameters of the model using the gradient of the second data type, to obtain the trained model.

[0015] In another aspect, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0016] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0017] In another aspect, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above-described method.

[0018] In this embodiment, during the forward pass, the input tensor of the second data type is converted based on the scaling parameter to obtain the input tensor of the first data type. This unifies the data types of the input tensors and model parameters within the linear layer, resulting in an accurate output tensor. After the linear layer completes its computation, the output tensor of the first data type is converted to the output tensor of the second data type for easier computation in subsequent model layers. During the backward pass, the model parameters of the first data type are converted based on the scaling parameter to obtain the model parameters of the second data type. This unifies the data types of the model parameters and the loss value, resulting in an accurate gradient. The model parameters are updated using the gradient of the second data type, ultimately yielding the trained model. Compared to related technologies, this embodiment allows for model training based on model parameters and input tensors of the first data type. The accuracy of the first data type is lower than that of the second data type, thus improving model training efficiency while maintaining model accuracy and reducing the computational and storage requirements during model training.

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0021] Figure 1 A schematic diagram of the implementation process of a model training method provided in this application embodiment. Figure 1 ;

[0022] Figure 2 A schematic diagram of the implementation process of a model training method provided in this application embodiment. Figure 2 ;

[0023] Figure 3 A schematic diagram of the implementation process of a model training method provided in this application embodiment. Figure 3 ;

[0024] Figure 4 A schematic diagram of the implementation process of a model training method provided in this application embodiment. Figure 4 ;

[0025] Figure 5 A schematic diagram of the implementation process of a model training method provided in this application embodiment. Figure 5 ;

[0026] Figure 6 A schematic diagram of the implementation process of a model training method provided in this application embodiment. Figure 6 ;

[0027] Figure 7 This is a schematic diagram of model parameter conversion provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of tensor transformation during forward training provided in an embodiment of this application;

[0029] Figure 9 A schematic diagram of tensor transformation during reverse training provided in an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0035] This application provides a model training method that can be executed by a processor of a computer device. The computer device can refer to a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other device with data processing capabilities. The processor can be, but is not limited to, a graphics processing unit (GPU), a tensor processing unit (TPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0036] Figure 1 A schematic diagram of the implementation process of a model training method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, the method includes the following steps S101 to S103:

[0037] Step S101: During the forward pass, the input tensor of the second data type is converted based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type; after the linear layer completes the calculation, the output tensor of the first data type is converted into the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type.

[0038] The model can be any neural network model that needs to be trained. The model includes at least one linear layer, and each linear layer includes model parameters of a first data type. The number of model parameters in each linear layer can be arbitrary, and this embodiment of the application does not limit it. A linear layer is a layer that performs linear operations. For example, a linear layer can be a convolutional layer or a layer normalization layer, and the first data type can be FP8 data type. Model parameters can include, but are not limited to, weights and biases.

[0039] Different data types have different numerical ranges and precisions. The scaling parameter is an adjustment factor corresponding to the model layer, used to adjust model parameters or other tensors when converting between different data types, so that the range distribution of the adjusted values ​​is more reasonable and the numerical precision is close to that before adjustment.

[0040] In some embodiments, different data types have different numerical ranges and precisions. Directly converting data types may result in missing values ​​or unreasonable value distribution in the converted data type. For example, directly converting an FP16 data type to an FP8 data type reduces the numerical range and may lead to missing values; directly converting an FP8 data type to an FP16 data type increases the numerical range and may lead to unreasonable value distribution. Therefore, scaling parameters need to be considered when performing data type conversions.

[0041] The input tensor is a tensor composed of the input data of the model layer. The output tensor is a tensor composed of the output data of the model layer.

[0042] In some embodiments, the latest GPU hardware architectures have supported lower data types (e.g., FP8 data type). Compared to data types such as FP32 and FP16, using FP8 in deep learning large model training scenarios can reduce the demand for computing resources and achieve higher training efficiency. However, because the representation range and precision of the FP8 data type are still relatively low, it is usually not possible to directly use the FP8 data type for model training.

[0043] In some embodiments, the linear layer includes model parameters of a first data type, while the input tensor of the linear layer is of a second data type. To enable efficient computation on the input tensor, the data types of the model parameters and the input tensor need to be unified. Therefore, during the forward training of the model, the input tensor of the second data type needs to be type-converted to obtain the input tensor of the first data type. Thus, within the linear layer, computation can be performed based on the model parameters and input tensors of the first data type to obtain the output tensor of the first data type. After the computation is completed in the linear layer, the output tensor of the first data type needs to be converted to the output tensor of the second data type for computation in the next model layer.

[0044] In this data type, the precision of the first data type is less than that of the second data type; the data range and precision of the first data type are also less than those of the second data type. For example, the first data type is an FP8 data type, and the second data type is an FP16 / FP32 data type. Accordingly, the storage space occupied by the data of the first data type is less than that occupied by the data of the second data type.

[0045] In some embodiments, the input tensor of the second data type is multiplied by a scaling parameter to obtain an intermediate input tensor of the second data type. This intermediate input tensor of the second data type is then converted to an input tensor of the first data type using a hardware-provided cast method. The cast method is a commonly used data type conversion function in the art.

[0046] In some embodiments, after the computation is completed in the linear layer, the output tensor of the first data type is converted into the output tensor of the second data type using the cast method.

[0047] For example, the input tensor 1 of the second data type is of FP16 data type, with a scaling parameter of scale. Multiplying input tensor 1 by scale yields the target input tensor 2. The input tensor 2 is then converted to input tensor 3 of the first data type using the cast method. Input tensor 3 is of FP8 data type. Based on input tensor 3 and the model parameters weight and bias of FP8 data type, computation is performed within the linear layer, outputting an output tensor of FP8 data type, input tensor 4. After computation in the linear layer, input tensor 4 of FP8 data type is converted to input tensor 5 of FP16 data type using the cast method.

[0048] In some embodiments, the model further includes an activation layer. During the forward pass, when passing through the activation layer, the output tensor of the first data type output by the previous layer needs to be converted into an input tensor of the second data type using the cast method before being input into the activation layer to ensure computational accuracy.

[0049] The aforementioned model is a model used for image processing, i.e., an image processing model. During training, the initial image processing model can be trained using an image sample set, which includes multiple image samples and the corresponding label for each image sample.

[0050] In some embodiments, the input tensor of the second data type is an image sample (such as RGB pixel values) directly loaded from the image sample set, which is then preprocessed (normalized, cropped, flipped, etc.) and stored in a high-precision format (FP32). For example, the format of the input tensor of this second data type can be [Batch_size, Channels, Height, Width], such as [32, 3, 224, 224], where the numerical range can be normalized pixel values, such as [0, 1] or [-1, 1], and the data type is torch.float32. Here, because high-precision data is used, image details and complete numerical information after preprocessing can be preserved, avoiding information loss in the early stages. Correspondingly, the input tensor of the first data type is a low-precision image sample obtained by downscaling the high-precision format image sample.

[0051] In some embodiments, the output tensor of the first data type can be the direct output obtained by performing matrix operations (e.g., y=Wx+b) on the linear layer using the input tensor (FP16) of the first data type and the model parameters (FP16) of the first data type. For example, the linear layer can be a convolutional layer, a fully connected layer, etc. Taking a fully connected layer in an image classification model as an example, the output tensor of the first data type can be feature data carrying the classification result. Correspondingly, the output tensor of the second data type is the high-precision format feature data carrying the classification result obtained by amplifying the low-precision format feature data carrying the classification result.

[0052] Step S102: In the reverse process, the model parameters of the first data type are converted based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type; the gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained after completing the forward process.

[0053] In some embodiments, the input tensor is computed through multiple model layers to ultimately output a predicted value. The difference between the predicted value and the true label value is calculated using a loss function to obtain the loss value. Since both the predicted value and the true label value are of secondary data type, the resulting loss value is also of secondary data type.

[0054] In some embodiments, after obtaining the loss value, a backpropagation process is required. The backpropagation process starts from the output layer and propagates the gradient of the loss function (i.e., the partial derivative of the loss value with respect to the output layer parameters) layer by layer to each layer of the network. During this process, the chain rule is used to calculate the gradient of the parameters in each layer. During the backpropagation process, to ensure training accuracy, a second data type is used to calculate and store the gradient when calculating the gradient of the activation layer.

[0055] In some embodiments, during the inverse process, the model parameters of the linear layer are of a first data type, while the loss value is of a second data type. Therefore, it is necessary to unify the data types of the model parameters and the loss value. Thus, when calculating the gradient of the linear layer, the model parameters of the first data type are first converted to obtain the model parameters of the second data type. The gradient of the second data type is then determined based on the model parameters of the second data type and the loss value of the second data type.

[0056] In some embodiments, the model parameters of the first data type are converted into intermediate model parameters of the second data type using the cast method. Then, the intermediate model parameters of the second data type are divided by the scaling parameter corresponding to the linear layer to obtain the model parameters of the second data type. The gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type.

[0057] For example, the model parameters weight1 and bias1 of the first data type are of FP8 data type. Using the cast method, weight1 and bias1 are converted into model parameters weight2 and bias2 of the second data type, which are of FP16 data type. Dividing weight2, bias2, and the scaling parameter scale yields the model parameters weight3 and bias3 of the second data type, which are also of FP16 data type. The loss value of the second data type is also of FP16 data type. Based on weight3, bias3, and the loss value of the second data type, the gradient of the second data type is determined.

[0058] In some embodiments, the loss value is obtained by calculating the difference between the output of the forward process corresponding to the image sample and the label corresponding to the image sample using a preset loss function.

[0059] Step S103: Update the model parameters of the model using the gradient of the second data type to obtain the trained model.

[0060] In some embodiments, after calculating the gradient in the backward propagation process, the model parameters of the linear layer need to be updated using the gradient. Then, the gradient is restored, and the next forward and backward training process is performed. The gradient calculated and stored in the backward propagation process is of a second data type, while the model parameters are of a first data type. Therefore, when updating the parameters, the data types of the gradient and model parameters need to be unified.

[0061] In some embodiments, the gradient of the second data type is multiplied by the scaling parameter to obtain the intermediate gradient of the second data type. The intermediate gradient of the second data type is then converted into the gradient of the first data type using the cast method. Finally, the model parameters of the first data type are updated based on the gradient of the first data type to obtain the trained model.

[0062] In some embodiments, during parameter updates, after obtaining the gradient of the first data type, the gradient of the first data type is multiplied by the learning rate to obtain an intermediate result. The updated model parameters are obtained by subtracting the intermediate result from the original model parameters.

[0063] For example, the gradient of the second data type is grad1, the model parameter of the first data type is w0, the scaling parameter is scale, and the learning rate is lr. The gradient of the first data type is cast (grad1*scale) by casting the gradient of grad1 through the cast method. Based on cast (grad1*scale) and lr, the updated model parameter w1 is determined as w0-lr*cast (grad1*scale).

[0064] In some embodiments, after training the model using the aforementioned image sample set to obtain the trained model, the trained model can be used for corresponding image processing tasks. The embodiments of this application can accelerate the model training process for image processing tasks. Furthermore, by employing input and output tensors of a second data type (high precision), the parameters of the image processing model can be updated with the precision of the second data type during training, thereby making the trained model more suitable for the task requirements of image processing and improving model training efficiency.

[0065] During the model training process described above, both the model parameters of the first data type and the gradients of the second data type are stored in the storage components of the computer device. These storage components can be temporary storage or volatile memory; for example, the storage component can be the video memory in a graphics processor or system memory. In achieving high-precision data training (i.e., using the gradients of the second data type to ensure training accuracy), to reduce the occupation of storage resources in the storage components, compared to directly using the model parameters of the second data type to complete model training in the prior art, the above embodiment uses the model parameters of the first data type to complete model training, thus reducing the occupation of storage resources in the storage components.

[0066] In some embodiments, the method further includes: in response to a training instruction, determining a target strategy in a first training strategy and a second training strategy; and performing a training process based on the target strategy to achieve the accuracy of a second data type, thereby obtaining a trained model.

[0067] The first training strategy involves training a model whose linear layers include model parameters of a first data type, and allocating a first cache space in a storage component during training. The first cache space stores the model parameters of the first data type and the gradients of the second data type. In some embodiments, the first cache space is further used to store momentum, such as first-order momentum and second-order momentum, generated during the process of determining the gradients of the second data type.

[0068] The second training strategy involves training a model whose linear layers include model parameters of the second data type, and allocating a second cache space in the storage component during training. The second cache space stores the model parameters of the second data type and the gradients of the second data type. In some embodiments, the second cache space also stores momentum, such as first-order momentum and second-order momentum, generated during the process of determining the gradients of the second data type.

[0069] Here, the first cache space is smaller than the second cache space.

[0070] In this embodiment, during the forward pass, the input tensor of the second data type is converted based on the scaling parameter to obtain the input tensor of the first data type. This unifies the data types of the input tensors and model parameters within the linear layer, resulting in an accurate output tensor. After the linear layer completes its computation, the output tensor of the first data type is converted to the output tensor of the second data type for easier computation in subsequent model layers. During the backward pass, the model parameters of the first data type are converted based on the scaling parameter to obtain the model parameters of the second data type. This unifies the data types of the model parameters and the loss value, resulting in an accurate gradient. The model parameters are updated using the gradient of the second data type, ultimately yielding the trained model. Compared to related technologies, this embodiment allows for model training based on model parameters and input tensors of the first data type. The accuracy of the first data type is lower than that of the second data type, thus improving model training efficiency while maintaining model accuracy and reducing the computational and storage requirements during model training.

[0071] Figure 2 This is a schematic diagram of the implementation process of a model training method provided in an embodiment of this application. Figure 2 This method can be executed by the processor of a computer device. Based on Figure 1 The method further includes steps S201 and S202.

[0072] Step S201: Determine the maximum input value based on the input tensor of the second data type.

[0073] In some embodiments, the scaling parameter of the linear layer is related to the input tensor of the linear layer. The scaling parameter of the linear layer is determined based on the maximum value of the input tensor of the linear layer. Therefore, it is necessary to determine the maximum input value based on the input tensor of the second data type.

[0074] The input tensor includes all the data input to the linear layer.

[0075] For example, the input tensor T is a one-dimensional array. First, the absolute value of the input tensor T, abs(T), is taken. Then, the maximum value in the input tensor after taking the absolute value, max(abs(T)), is taken as the maximum input value. The values ​​in the input tensor T can be positive, negative, or 0.

[0076] Step S202: Determine the scaling parameter based on the maximum input value and the maximum value corresponding to the first data type.

[0077] The first data type has a corresponding data range, and the maximum value of the type can be directly obtained based on the first data type. For example, if the first data type is FP8, and the FP8 data type is E5M2, the maximum value of the type is 57334.

[0078] In some embodiments, the scaling parameter is obtained by dividing the maximum value of the first data type by the maximum value of the input. It is understood that the scaling parameter is less than 1 when the maximum value of the type is less than the maximum value of the input, and greater than 1 when the maximum value of the type is greater than the maximum value of the input.

[0079] In some embodiments, the model includes at least two linear layers, each with a corresponding scaling parameter, and the different scaling parameters are independent of each other. The scaling parameter is a 32-bit floating-point number, initialized to 1.0.

[0080] In this embodiment, the maximum input value is determined based on the input tensor of the second data type; the scaling parameter is determined based on the maximum input value and the maximum value corresponding to the first data type. In this way, accurate scaling parameters can be obtained based on the input tensor of the linear layer.

[0081] Figure 3 This is a schematic diagram of the implementation process of a model training method provided in an embodiment of this application. Figure 3 This method can be executed by the processor of a computer device. Based on Figure 1 The step S101, "converting the input tensor of the second data type based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type," can be updated to S301 to S302, combining... Figure 3 The steps shown are explained.

[0082] Step S301: Determine the target input tensor based on the scaling parameter and the input tensor of the second data type.

[0083] In some embodiments, before performing type conversion on the input tensor of the second data type, it is necessary to adjust the input tensor of the second data type using scaling parameters to obtain the target input tensor.

[0084] Specifically, the scaling parameter is multiplied by the input tensor of the second data type to obtain the target input tensor. This can be understood as the scaling parameter being multiplied by each input data point in the input tensor.

[0085] Step S302: Perform type conversion on the target input tensor to obtain an input tensor of the first data type.

[0086] Specifically, the target input tensor is type-converted using the cast method provided by the hardware to obtain an input tensor of the first data type.

[0087] For example, the input tensor T is of FP16 data type and the scaling parameter is scale. First, T is multiplied by scale to obtain the target input tensor T*scale. The target input tensor is then converted to a first data type using the cast method to obtain the input tensor cast(T*scale), which is of FP8 data type.

[0088] In this embodiment, a target input tensor is determined based on a scaling parameter and an input tensor of a second data type; the target input tensor is then type-converted to obtain an input tensor of a first data type. This allows the input tensor to be adjusted based on the scaling parameter, thereby obtaining an accurate type-converted input tensor.

[0089] Figure 4 This is a schematic diagram of the implementation process of a model training method provided in an embodiment of this application. Figure 4 This method can be executed by the processor of a computer device. Based on Figure 1 The step S102, "converting the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type," can be updated to S401 to S402, combining... Figure 4 The steps shown are explained.

[0090] Step S401: Convert the model parameters of the first data type to obtain the target model parameters.

[0091] In some embodiments, when converting model parameters of a first data type to model parameters of a second data type, it is necessary to first perform a type conversion on the model parameters of the first data type using the cast method to obtain the target model parameters of the second data type. Then, the target model parameters are adjusted to obtain the model parameters of the second data type.

[0092] Step S402: Determine the model parameters of the second data type based on the target model parameters and scaling parameters.

[0093] After obtaining the target model parameters for the second data type, these parameters need to be adjusted to better match the second data type. Here, the target model parameters are divided by the scaling parameters to obtain the model parameters for the second data type.

[0094] For example, the model parameter weight is of FP8 data type. The model parameter weight is converted into the target model parameter cast(weight) using the cast method. cast(weight) is of FP16 data type. The scaling parameter is scale. The model parameter cast(weight) / scale is of the second data type. cast(weight) / scale is of FP16 data type.

[0095] In this embodiment, the model parameters of the first data type are converted to obtain the target model parameters; based on the target model parameters and scaling parameters, the model parameters of the second data type are determined. Thus, by first converting the model parameters to their types and then adjusting them based on the scaling parameters, accurate model parameters for the second data type can be obtained.

[0096] Figure 5 This is a schematic diagram of the implementation process of a model training method provided in an embodiment of this application. Figure 5 This method can be executed by the processor of a computer device. Based on Figure 1 , Figure 1 S103 in the middle can be updated to S501 to S503, which will combine Figure 5 The steps shown are explained.

[0097] Step S501: Determine the target gradient based on the gradient of the second data type and the scaling parameter.

[0098] In some embodiments, the calculated gradient is of a second data type, while the model parameters are of a first data type. Therefore, when updating the model parameters based on the gradient, the gradient needs to be converted from the second data type to the first data type. When converting the gradient type, the second data type gradient needs to be adjusted based on a scaling parameter to obtain the target gradient.

[0099] The target gradient is obtained by multiplying the gradient of the second data type by a scaling parameter. The target gradient is of the second data type.

[0100] Step S502: Convert the target gradient to obtain a gradient of the first data type.

[0101] In this process, after obtaining the target gradient, the target gradient of the second data type is directly converted using the cast method to obtain the gradient of the first data type.

[0102] Step S503: Update the model parameters of the model based on the gradient of the first data type to obtain the trained model.

[0103] In this case, after obtaining the gradient of the first data type, the model parameters are also of the first data type. Therefore, the model parameters can be updated based on the gradient of the first data type to obtain the trained model.

[0104] In some embodiments, the gradient of the first data type is multiplied by the learning rate to obtain an intermediate result, and then the intermediate result is subtracted from the current model parameters to obtain the updated model parameters. For each model parameter, the model parameters are updated based on the gradient of the corresponding first data type, and finally the trained model is obtained.

[0105] For example, the current model parameters are weights W0, learning rate lr, scaling parameter scale, and gradient of the second data type grad. The target gradient is grad*scale, the gradient of the first data type is cast(grad*scale), and the updated weights W1 are W0-lr*cast(grad*scale).

[0106] In this embodiment, a target gradient is determined based on the gradient of a second data type and scaling parameters; the target gradient is then converted to a first data type gradient; and the model parameters are updated based on the first data type gradient to obtain the trained model. This unifies the data types of the model gradient and model parameters, enabling accurate updates to the model parameters based on the converted model gradient.

[0107] Figure 6 This is a schematic diagram of the implementation process of a model training method provided in an embodiment of this application. Figure 6 This method can be executed by the processor of a computer device. Based on Figure 1 , Figure 1 The phrase "determine the gradient of the second data type based on the model parameters and loss value of the second data type" can be updated to S601 to S604, combining... Figure 6 The steps shown are explained.

[0108] Step S601: Determine the current gradient based on the model parameters of the second data type and the loss value of the second data type.

[0109] Here, the current gradient is the gradient at the current time step (current iteration number) during model training. After obtaining the model parameters and loss values ​​of the second data type, the current gradient corresponding to the model parameters can be determined based on the backpropagation algorithm and the chain rule. The current gradient is of the second data type, and each model parameter corresponds to one current gradient.

[0110] Step S602: Based on the current gradient, first-order statistics and second-order statistics, determine the current first-order momentum and the current second-order momentum.

[0111] During model training, an optimizer is used to update model parameters, optimizing the model's weights and biases to improve the accuracy of predictions. The optimizer determines the method and rate of parameter updates, thus affecting the model's convergence speed and final performance.

[0112] In some embodiments, during model training, the optimizer needs to store first-order and second-order statistics, which are used to determine first-order and second-order momentum. First-order momentum introduces a momentum term during iteration, accelerating the gradient descent process by accumulating past gradient information; second-order momentum typically refers to optimization algorithms that consider the squared gradients of the past, which can adaptively adjust the learning rate of each parameter.

[0113] In some embodiments, the current first-order momentum is determined based on the current gradient, the first-order momentum of the previous time step, and the first-order statistic; the current first-order momentum is determined by formula (1):

[0114] (1);

[0115] in, Given the current first-order momentum, The first-order momentum of the previous time step. These are first-order statistical values. This is the current gradient.

[0116] In some embodiments, the current second-order momentum is determined based on the current gradient, the second-order momentum of the previous time step, and the second-order statistics; the current second-order momentum is determined by formula (2):

[0117] (2);

[0118] in, Given the current second-order momentum, The second momentum of the previous time step. These are second-order statistical values. This is the current gradient.

[0119] Step S603: Adjust the current first-order momentum based on the first-order statistical value to obtain the adjusted first-order momentum, and adjust the current second-order momentum based on the second-order statistical value to obtain the adjusted second-order momentum.

[0120] In this process, the initial first-order momentum and the initial second-order momentum are initialized to 0. and It tends to be close to 0. To correct this bias, a bias correction term is introduced to adjust the current first-order momentum and the current second-order momentum.

[0121] In some embodiments, the adjusted first-order momentum is determined based on the first-order statistics and the current first-order momentum. The adjusted first-order momentum is determined by formula (3):

[0122] (3);

[0123] in, This is the adjusted first-order momentum.

[0124] In some embodiments, the adjusted second-order momentum is determined based on the second-order statistics and the current second-order momentum. The adjusted second-order momentum is determined by formula (4):

[0125] (4);

[0126] in, This is the adjusted second-order momentum.

[0127] Step S604: Determine the gradient of the second data type based on the adjusted first-order momentum and the adjusted second-order momentum.

[0128] In this process, after obtaining the adjusted first-order momentum and the adjusted second-order momentum, the gradient of the second data type can be determined based on the adjusted first-order momentum and the adjusted second-order momentum.

[0129] In some embodiments, the gradient of the second data type is determined by formula (5):

[0130] (5);

[0131] in, The adjusted first-order momentum, Let be the adjusted second-order momentum, grad be the adjusted gradient, and ϵ be a very small number.

[0132] In this embodiment, the current gradient is first determined. Based on the current gradient, first-order statistics, and second-order statistics, the current first-order momentum and second-order momentum are determined, and the first-order momentum and second-order momentum are adjusted to determine the gradient of the second data type. In this way, an accurate gradient can be determined based on the momentum statistics, optimizing the model parameters and making the model prediction results more accurate.

[0133] In some embodiments, the first data type is a floating-point type with a first precision, the second data type is a floating-point type with a second precision, and the first precision is less than the second precision.

[0134] The model parameters are of a first data type, which is a first-precision floating-point type; for example, the first data type is FP8. The model's input tensors are of a second data type, which is a second-precision floating-point type; for example, the second data type is FP16.

[0135] In floating-point data types, the mantissa bits represent the precision; more mantissa bits result in higher numerical precision. The first precision is less than the second precision. For example, if the first data type is FP8, the E5M2 format has 2 mantissa bits, and the E4M3 format has 3 mantissa bits; if the second data type is FP16, the FP16 data type has 10 mantissa bits. Therefore, the first precision is less than the second precision.

[0136] In this case, both the first and second data types are floating-point numbers. Thus, the design circuits for the first and second data types can be reused to some extent. The conversion circuits between the first and second data types can be designed to be simpler and more direct, without the overhead of multiplication and addition required for conversion from integer to floating-point.

[0137] In this model, the first precision is less than the second precision. Thus, the bit width of the first data type is less than that of the second data type. During model training, training can be performed based on the first data type with a smaller bit width. Compared to the second data type, using the first data type during model training can reduce the demand for computing resources and achieve higher training efficiency.

[0138] In this embodiment, the first data type is a floating-point type with a first precision, and the second data type is a floating-point type with a second precision, where the first precision is less than the second precision. This simplifies the conversion circuit between different data types, reduces the computational resource requirements during model training, and achieves higher training efficiency.

[0139] In some embodiments, the first data type is an 8-bit floating-point number FP8, and the second data type is a 16-bit floating-point number FP16 or a 32-bit floating-point number FP32.

[0140] In related technologies, deep learning models primarily utilize FP32 or FP16 data types for training. However, with the increasing number of model parameters, their computational resource consumption is also rising. Currently, many of the latest GPU hardware architectures support the FP8 data type. Under the same hardware bandwidth, it can transmit more data compared to FP16 and 16-bit floating-point BF16 data types, and offers higher computational precision compared to 8-bit integer INT8 data types. Therefore, using FP8 as the first data type and FP16 as the second can improve training efficiency and reduce resource consumption while meeting the current training data requirements.

[0141] The FP8 data type has two formats: E5M2 and E4M3 (where E5M2 represents the number of bits used for the exponent and mantissa in an 8-bit data set; E5M3 has 5 bits for the exponent, 2 bits for the mantissa, and 1 sign bit, for a total of 8 bits). These two formats represent different data ranges and have different precisions. Generally, more exponent bits provide a wider numerical range, while more mantissa bits provide higher numerical precision. For large language models, the FP8 format is set to E5M2, while for visual models, the FP8 format is set to E4M3.

[0142] The FP16 data type includes 1 sign bit, 5 exponent bits, and 10 mantissa bits. FP16 is a half-precision floating-point data type that uses 16 binary bits to represent floating-point numbers. It has lower precision and numerical range, but it has significant advantages in memory usage and computation time in deep learning and high-performance computing.

[0143] The FP32 uses 32 bits to represent floating-point numbers, including a sign bit, 8 exponent bits, and 23 mantissa bits. This representation allows the FP32 to provide high precision and dynamic range, making it suitable for most scientific and general-purpose computing tasks.

[0144] In this embodiment, the first data type is FP8, and the second data type is FP16 or FP32. This allows for conversion between the first and second data types, meeting the requirements for input training data, improving training efficiency, and reducing resource consumption.

[0145] In some embodiments, the initialized model parameters are of type FP32, and the method further includes: performing type conversion on the initialized model parameters to obtain model parameters of type FP8.

[0146] For the initial model, its model parameters are usually of FP32 data type. In order to train based on FP8 data type, the model parameters need to be converted to FP8 data type first. This conversion can be performed directly using the cast method without involving actual hardware computation. The converted FP8 data type is the first data type.

[0147] For example, the initialized model parameters include weights0 and biases0. The initialized weights0 and biases are of FP32 data type. The cast method is used to convert weights0 and biases0 into weights1 and biases1, which are of FP8 data type. Then, training is performed based on the FP8 data type weights1 and biases1.

[0148] In this embodiment, the initialized model parameters are of type FP32. The initialized model parameters are then converted to type FP8. This conversion of model parameters during initialization, followed by training based on the converted parameters, improves training efficiency and reduces resource consumption.

[0149] The following describes the application of the model training method provided in the embodiments of this application in a real-world scenario.

[0150] Currently, deep learning models primarily utilize FP32 and FP16 data types for training. With the increasing number of model parameters, their consumption of computing resources is also rising. Therefore, how to train larger-scale models with limited computing resources has become a research direction in the industry.

[0151] Many of the latest GPU hardware architectures now support the FP8 data type (8-bit floating-point), which can transmit more data per unit time with the same hardware bandwidth and offers higher computational precision compared to the INT8 (8-bit integer) data type. However, compared to commonly used data types such as FP32 and FP16, the representation range and precision of FP8 are still insufficient for direct model training. Therefore, this application proposes a model training method for the FP8 data type (corresponding to the first data type in the above embodiments), through:

[0152] 1. The range of the activation value is dynamically scaled to better suit the FP8 data type;

[0153] 2. A new optimizer was designed for training FP8 data types to minimize GPU memory usage while ensuring training accuracy.

[0154] This allows the model to fully utilize the FP8 data type during training, achieving higher training efficiency and reducing resource consumption during the training process.

[0155] Solutions for training low-precision models typically employ quantization-aware training based on the INT8 data type. This involves inserting observation points into the model to record changes in activation values ​​during training, and then quantizing the weights using INT8 based on statistical values. However, this method does not significantly reduce computational resource consumption, and it requires dynamic conversion of the data types of tensors such as weights and activation values, resulting in low training efficiency.

[0156] The latest GPU hardware architecture supports the FP8 data type. Compared with data types such as FP32 and FP16, using FP8 in deep learning large model training scenarios can reduce the demand for computing resources and achieve higher training efficiency. However, because the representation range and precision of the FP8 data type are still lower than those of FP16 and BF16, the FP8 data type cannot usually be used directly for model training.

[0157] The model training method based on the FP8 data type in this application has higher training efficiency, and the FP8 data type has better dynamic range and accuracy than the INT8 format used in the former, resulting in better training effect of the final model.

[0158] The FP8 data type offers nearly twice the performance of 16-bit data types like FP16. As an 8-bit data type, FP8 also has some unique advantages over INT8:

[0159] 1. FP8 is a floating-point type, and the design circuit of FP8 MAC can be reused with FP16 to some extent;

[0160] 2. The conversion circuit between FP8 and FP16 / FP32 / BF16 can be designed to be simpler and more direct, without the overhead of multiplication and addition required for the conversion from INT8 / UINT8 to FP.

[0161] The hardware architecture supports two FP8 formats: E5M2 and E4M3. The FP8 data type model training method designed in this application sets the FP8 format to E5M2 for large language models and to E4M3 for visual models.

[0162] In some embodiments, the weights and biases of the initialized model are typically of FP32 data type. Therefore, it is necessary to convert the model's weights and biases to FP8 data type first. This conversion can be performed directly using the hardware-provided cast method, without involving actual hardware computation, and is not required here. After the model conversion, a scaling parameter, scale, needs to be added to all model layers that need to be trained (such as convolutional layers, linear layers, batch normalization layers, layer normalization layers, etc., detailed requirements are not specified here). This parameter is a 32-bit floating-point number, initialized to 1.0. It is important to note that each model layer has its own scaling parameter, and these parameters are independent of each other. The scaling parameter represents the scaling factor when converting between different data types. The scale parameter can make the data distribution corresponding to the converted data type more reasonable.

[0163] Figure 7 This is a schematic diagram illustrating a model parameter transformation provided in an embodiment of this application. For example... Figure 7 As shown, in the initialized model, the weights 701 and biases 702 in the convolutional layer (Conv2d layer) and linear layer (Linear layer) are of FP32 data type. The weights 701 and biases 702 in the convolutional layer and linear layer are converted to FP8 data type weights 703 and biases 704 by the cast method, and a scaling parameter 705 is added to each of the convolutional layer and linear layer.

[0164] In some embodiments, during model training, the optimizer needs to simultaneously store model parameters, model gradients, and momentum statistics. Model parameters refer to the size of the model parameters; for example, a large language model of 7 bytes has seven billion model parameters. If the FP32 data type is used, approximately 28GB of storage space is required (each parameter is stored in FP32, and one FP32 data occupies 4 bytes). The model gradient size is the same as the model parameters, while the momentum statistics vary depending on the optimizer: for the most commonly used Adam optimizer, it needs to store first-order and second-order momentum, requiring twice the storage of the model parameters. Therefore, for training any size large language model, if Adam and FP32 data types are used, the memory usage is calculated using formula (6):

[0165] (4+4+4+4)*model_size = 16*model_size(6);

[0166] Where model_size is the size of the model parameters.

[0167] The FP8 training method in this application converts the model parameters to FP8 data type in advance, reducing the model parameter storage overhead by 75%. For model gradients, the gradient usually needs to be kept in FP32 data type to ensure training accuracy. This application uses FP8 model parameters, and using FP16 as the gradient is sufficient to represent the parameter information (FP32 can represent numerical precision far exceeding FP8, so FP16 can be used to store the gradient). Therefore, this application uses FP16 data type to store the gradient, reducing the gradient storage requirement by half. The first-order statistical value (α) in Adam is stored in FP16, while the second-order statistical value (β) still maintains the FP32 data format because it requires higher precision. In summary, the GPU memory required for the new optimizer to train a large language model of arbitrary size is calculated by formula (7):

[0168] (1+2+2+4)*model_size = 9*model_size(7);

[0169] Among them, the video memory consumption was only 56.25% of the previous level.

[0170] In some embodiments, during the forward pass of FP8 model training, the training data is typically preprocessed into FP32 / FP16 data type (corresponding to the second data type input tensor in the above embodiments). In this embodiment, the training data type is maintained and is not required. After training starts, before inputting to each parameterized model layer (a scale parameter is added to each parameterized model layer), it is determined whether the input is of FP8 type. If so, the following operations are performed:

[0171] 1. First, the scaling parameter of the model layer needs to be calculated and saved based on the input tensor T. The calculation formulas for the scaling parameter are as follows: (8) and (9):

[0172] scale = float_max / max_T(8);

[0173] max_T = max(abs(T))(9);

[0174] Wherein, float_max (corresponding to the maximum input value in the above embodiment) is the maximum value currently using the FP8 format, such as the maximum value of the E5M2 format being 57334.0, and max_T (corresponding to the maximum type value in the above embodiment) is the maximum value of the absolute value of the tensor.

[0175] 2. Multiply the current input by the calculated scaling parameter, and then cast it to an FP8 type; then, FP8 data type operations (convolution, matrix multiplication, etc.) can be performed within this layer.

[0176] Figure 8 This is a schematic diagram of tensor transformation during forward training provided in an embodiment of this application. Figure 8 As shown, for a linear layer, the initial input tensor 801 (corresponding to the input tensor of the second data type in the above embodiment) is of type FP16. The scaling parameter 802 can be calculated from the initial input tensor 801. The initial input tensor is multiplied by the scaling parameter to obtain the intermediate input tensor 803 (corresponding to the target input tensor in the above embodiment). The intermediate input tensor 803 is of type FP16. The intermediate input tensor is converted into the target input tensor 804 of type FP8 (corresponding to the input tensor of the first data type in the above embodiment) by the cast method. Then, FP8 data type operations can be performed in this layer (the target input tensor 804 of type FP8 is convolved with the weights & biases of type FP8, matrix multiplication, etc.) to obtain the target output tensor 806 of type FP8. The target output tensor 806 of type FP8 (corresponding to the output tensor of the first data type in the above embodiment) is converted into the target input tensor 807 of type FP16 (corresponding to the output tensor of the second data type in the above embodiment) by the cast method and input into the next model layer.

[0177] Specifically, when the target output tensor 806 is input to the activation layer (the activation layer is usually a non-linear layer, such as ReLU, GeLU, Logits, etc., which do not involve weights; specific requirements are not specified here), it determines whether the current input is of type FP8. If so, the FP8 type target output tensor 806 needs to be cast to the FP16 type target input tensor 807 to ensure computational accuracy. For example... Figure 8 As shown, the next layer after the linear layer is the activation layer (GeLU layer). The target output tensor 806 of type FP8 in the linear layer is cast to the target input tensor 807 of type FP16 and input to the GeLU layer. The GeLU layer outputs the target output tensor 808 of type FP16.

[0178] For two adjacent linear layers, the output tensor of the preceding linear layer is of type FP8. This output tensor needs to be cast to an input tensor of type FP16, and this FP16 input tensor is used as the input to the following linear layer. Within the following linear layer, the computation process of the preceding linear layer is repeated: first, the scaling parameter of the layer is calculated based on the FP16 input tensor; then, the scaling parameter is multiplied by the input tensor to obtain an intermediate input tensor, which is also of type FP16. This intermediate input tensor is then converted to an FP8 target input tensor using a cast method, after which FP8 type operations can be performed within that layer.

[0179] In some embodiments, during the backpropagation process of FP8 model training, based on the forward process until the loss value is calculated, the backpropagation process is performed in the same way as that of FP32 model training. Gradients typically need to be stored in FP32 data format. However, using the training method proposed in this application, gradients can be calculated and stored in FP16 format (corresponding to the second data type gradient in the above embodiments). Specifically, the initial loss value is of type FP16. For the aforementioned calculation of gradients for model layers without parameters (such as activation layers, output logits layers, etc.), FP16 type is used to calculate and store the gradients to ensure training accuracy. For model layers with FP8 parameters (such as convolutional layers (conv), linear layers (Linear), when calculating the gradient, the model parameters need to be cast to FP16 type first, then divided by the scale parameter stored in that layer, and then the gradient of the model layer is calculated normally.

[0180] Figure 9 This is a schematic diagram of tensor transformation during reverse training provided in an embodiment of this application. Figure 9 As shown, within the linear layer, the initial weights and biases 901 (corresponding to the model parameters of the first data type in the above embodiment) are of type FP8. The weights and biases are then first cast to type FP16 to obtain intermediate weights and biases 902 (corresponding to the target model parameters in the above embodiment). These are then divided by the scaling parameter 903 to obtain the adjusted target weights and biases 904 of type FP16 (corresponding to the model parameters of the second data type in the above embodiment). Simultaneously, the activation layer (GeLU layer) connected to the linear layer receives the initial gradient tensor 905 of type FP16 transmitted from the next layer and, based on the initial gradient tensor, outputs an intermediate gradient tensor 906 of type FP16, which is then transmitted to the linear layer. The linear layer calculates the gradients of the weights and biases based on the received intermediate gradient tensor 906 and the target weights and biases 904 of type FP16, saves them, and finally outputs the target gradient tensor 907 of type FP16 to the next layer.

[0181] In some embodiments, during the parameter update process of FP8 model training, after fully calculating the gradient, the optimizer needs to update the gradient onto the existing weights and biases, then restore the gradient, and proceed with the next training iteration (forward + backward process). In the stochastic gradient descent algorithm, after each weight update, the gradient is not directly retained for the next update; instead, a new gradient is recalculated in the next iteration based on the difference between the current model prediction and the true value. In some more complex optimizers, gradients may be accumulated or adaptively adjusted, but before each new training epoch, there will also be gradient-related operations such as "resetting" or "initializing" to ensure that each gradient calculation and update is based on the current model state and data.

[0182] In this embodiment, the optimizer stores gradients of type FP16, while the model parameters are of type FP8. Therefore, when updating the weights, the update formula changes from formula (10) to formula (11):

[0183] W_1 = W_0 - lr * grad (10);

[0184] W_1 = W_0 - lr * Cast(grad * scale) (11);

[0185] Where W_1 is the updated FP8 type weight, W_0 is the original FP8 type weight, lr is the learning rate, grad is the gradient, scale is the scale parameter stored in this layer, and Cast(*) is the type conversion.

[0186] Specifically, the calculation formulas for grad in the Adam optimizer are shown in formulas (12) to (16):

[0187] (12);

[0188] (13);

[0189] (14);

[0190] (15);

[0191] (16);

[0192] in, These are the first-order statistics of the optimizer. These are the second-order statistics of the optimizer. The current gradient calculated for this weight. Let the second-order momentum be the momentum at the current time step. The second momentum of the previous time step. Let the first-order momentum be the momentum at the current time step. The first-order momentum of the previous time step. The adjusted first-order momentum, Let be the adjusted second-order momentum, and grad be the adjusted gradient. It is a very small number.

[0193] In some embodiments, any model can be selected, such as the currently open-source large language model. After model initialization, the model is first loaded onto the computing device, and then the model's weights and biases are directly converted to FP8 data types. This type conversion is similar to the regular conversion from FP32 to FP16, and does not involve specific calculations; it is a direct conversion of the hardware's internal data types. Based on the converted FP8 model, an FP8 optimizer is set up, and then data can be loaded to start training. After training, a model based on the FP8 data type is obtained.

[0194] The embodiments of this application can utilize the FP8 data type for model training, while maintaining (approximately maintaining) the same prediction accuracy as models trained using FP32 / FP16, thereby improving training efficiency and reducing dependence on and consumption of computing resources.

[0195] The embodiments of this application can make full use of the FP8 data type to directly train the model; while maintaining the model accuracy, the model training efficiency is improved; and the requirements for computing power and storage during model training are reduced.

[0196] Based on the foregoing embodiments, this application provides a model training device, which includes various units and modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0197] Figure 10 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of this application, as shown below. Figure 10 As shown, the model training device 1000 includes: a first conversion module 1010, a second conversion module 1020, and an update module 1030, wherein: the model includes a linear layer, and the linear layer includes model parameters of a first data type;

[0198] The first conversion module 1010 is used to perform type conversion on the input tensor of the second data type based on the scaling parameter corresponding to the linear layer during the forward process to obtain the input tensor of the first data type; after the linear layer completes the calculation, it converts the output tensor of the first data type into the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type;

[0199] The second conversion module 1020 is used to, during the reverse process, perform type conversion on the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type; determine the gradient of the second data type based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained after completing the forward process;

[0200] The update module 1030 is used to update the model parameters of the model using the gradient of the second data type to obtain the trained model.

[0201] In some embodiments, the first conversion module 1010 is further configured to determine the maximum input value based on the input tensor of the second data type; and to determine the scaling parameter based on the maximum input value and the maximum value of the type corresponding to the first data type.

[0202] In some embodiments, the first conversion module 1010 is further configured to determine a target input tensor based on the scaling parameter and the input tensor of the second data type; and to perform type conversion on the target input tensor to obtain an input tensor of the first data type.

[0203] In some embodiments, the second conversion module 1020 is further configured to perform type conversion on the model parameters of the first data type to obtain target model parameters; and determine the model parameters of the second data type based on the target model parameters and the scaling parameters.

[0204] In some embodiments, the update module 1030 is further configured to determine a target gradient based on the gradient of the second data type and the scaling parameter; perform type conversion on the target gradient to obtain a gradient of the first data type; and update the model parameters of the model based on the gradient of the first data type to obtain the trained model.

[0205] In some embodiments, the second conversion module 1020 is further configured to: determine the current gradient based on the model parameters of the second data type and the loss value of the second data type; determine the current first-order momentum and the current second-order momentum based on the current gradient, the first-order statistical value, and the second-order statistical value; adjust the current first-order momentum based on the first-order statistical value to obtain an adjusted first-order momentum, and adjust the current second-order momentum based on the second-order statistical value to obtain an adjusted second-order momentum; and determine the gradient of the second data type based on the adjusted first-order momentum and the adjusted second-order momentum.

[0206] In some embodiments, the first data type is a floating-point type with a first precision, the second data type is a floating-point type with a second precision, and the first precision is less than the second precision.

[0207] In some embodiments, the first data type is an 8-bit floating-point number FP8, and the second data type is a 16-bit floating-point number FP16 or a 32-bit floating-point number FP32.

[0208] In some embodiments, the initialized model parameters are of type FP32, and the first conversion module 1010 is further used to perform type conversion on the initialized model parameters to obtain model parameters of type FP8.

[0209] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0210] It should be noted that, in the embodiments of this application, if the above-described model training method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0211] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0212] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0213] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0214] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0215] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0216] Figure 11 This application provides a hardware entity diagram of a computer device as an embodiment of the present application, such as... Figure 11 As shown, the hardware entity of the computer device 1100 includes a processor 1101 and a memory 1102, wherein the memory 1102 stores a computer program that can run on the processor 1101, and the processor 1101 executes the program to implement the steps in the method of any of the above embodiments.

[0217] The memory 1102 stores computer programs that can run on the processor. The memory 1102 is configured to store instructions and applications that can be executed by the processor 1101. It can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1101 and various modules in the computer device 1100. It can be implemented by flash memory or random access memory (RAM).

[0218] The processor 1101 executes the program to implement the steps of any of the above-mentioned model training methods. The processor 1101 typically controls the overall operation of the computer device 1100.

[0219] This application provides a computer storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the model training method as described in any of the above embodiments.

[0220] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0221] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0222] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0223] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The model includes linear layers, each containing model parameters of a first data type, and the model training method includes: During the forward pass, the input tensor of the second data type is converted based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type; after the linear layer completes the calculation, the output tensor of the first data type is converted into the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; During the reverse process, the model parameters of the first data type are converted based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type; the gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained after completing the forward process; The model parameters of the model are updated using the gradient of the second data type to obtain the trained model; The model is an image processing model for image processing; the input tensors of the first data type and the second data type are image samples of different precision; the output tensors of the first data type and the second data type are feature data of different precision carrying the image processing results.

2. The model training method according to claim 1, characterized in that, The model training method also includes: Based on the input tensor of the second data type, determine the maximum input value; The scaling parameter is determined based on the maximum input value and the maximum value corresponding to the first data type.

3. The model training method according to claim 1, characterized in that, The step of converting the input tensor of the second data type based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type includes: Based on the scaling parameters and the input tensor of the second data type, determine the target input tensor; The target input tensor is type-converted to obtain an input tensor of the first data type.

4. The model training method according to claim 1, characterized in that, The step of converting the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type includes: The model parameters of the first data type are converted to obtain the target model parameters; Based on the target model parameters and the scaling parameters, the model parameters for the second data type are determined.

5. The model training method according to claim 1, characterized in that, The step of updating the model parameters of the model using the gradient of the second data type to obtain the trained model includes: The target gradient is determined based on the gradient of the second data type and the scaling parameter; The target gradient is type-converted to obtain a gradient of a first data type; The model parameters of the model are updated based on the gradient of the first data type to obtain the trained model.

6. The model training method according to claim 1, characterized in that, Determining the gradient of the second data type based on the model parameters and loss value of the second data type includes: The current gradient is determined based on the model parameters of the second data type and the loss value of the second data type; Based on the current gradient, first-order statistics, and second-order statistics, determine the current first-order momentum and the current second-order momentum; The current first-order momentum is adjusted based on the first-order statistical value to obtain the adjusted first-order momentum, and the current second-order momentum is adjusted based on the second-order statistical value to obtain the adjusted second-order momentum. The gradient of the second data type is determined based on the adjusted first-order momentum and the adjusted second-order momentum.

7. The model training method according to any one of claims 1 to 6, characterized in that, The first data type is a floating-point type with a first precision, and the second data type is a floating-point type with a second precision, wherein the first precision is less than the second precision.

8. The model training method according to any one of claims 1 to 6, characterized in that, The first data type is an 8-bit floating-point number (FP8), and the second data type is a 16-bit floating-point number (FP16) or a 32-bit floating-point number (FP32).

9. The model training method according to any one of claims 1 to 6, characterized in that, The initialized model parameters are of type FP32, and the method further includes: The initialized model parameters are converted to FP8 type model parameters.

10. A model training device, characterized in that, The model includes a linear layer, and the linear layer includes model parameters of a first data type; the model training device includes: The first conversion module is used to perform type conversion on the input tensor of the second data type based on the scaling parameter corresponding to the linear layer during the forward process to obtain the input tensor of the first data type; after the linear layer completes the calculation, it converts the output tensor of the first data type into the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; The second conversion module is used to, during the reverse process, perform type conversion on the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type; determine the gradient of the second data type based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained after completing the forward process; The update module is used to update the model parameters of the model using the gradient of the second data type to obtain the trained model. The model is an image processing model for image processing; the input tensors of the first data type and the second data type are image samples of different precision; the output tensors of the first data type and the second data type are feature data of different precision carrying the image processing results.

11. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the model training method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the model training method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps in the model training method according to any one of claims 1 to 9.

Images