Model training method and device, equipment, storage medium and program product

Abstract

The invention discloses a model training method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the type conversion of an input tensor of a second data type based on a zooming parameter corresponding to a linear layer in a forward process, and obtaining an input tensor of a first data type; after the calculation of the linear layer is completed, converting the output tensor of the first data type into an output tensor of a second data type; in a reverse process, performing type conversion on the model parameter of the first data type based on a scaling parameter corresponding to a linear layer to obtain a model parameter of a second data type; determining a gradient of the second data type based on the model parameter of the second data type and the loss value of the second data type; and updating model parameters of the model by using the gradient of the second data type to obtain a trained model. Therefore, under the condition that the model precision is guaranteed, the model training efficiency is improved, and the requirements for computing power and storage during model training are reduced.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a model training method, apparatus, device, storage medium, and program product. Background Art

[0002] Current deep learning models primarily utilize FP32 and FP16 data types for training. As model parameters continue to grow, their demand for computing power is also increasing. While the latest graphics processing units (GPUs) support lower data types, reducing computing power requirements and achieving higher training efficiency, their representation range and accuracy remain limited, often preventing direct model training. Therefore, training larger models with limited computing resources has become a key area of industry research. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide at least one model training method, apparatus, device, storage medium, and program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows: On the one hand, an embodiment of the present application provides a model training method, wherein the model includes a linear layer, and the linear layer includes model parameters of a first data type. The method includes: in a forward process, performing type conversion on an input tensor of a second data type based on a scaling parameter corresponding to the linear layer to obtain an input tensor of the first data type; after the linear layer completes the calculation, converting the output tensor of the first data type to an output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; in a reverse process, performing type conversion on the model parameters of the first data type based on the scaling parameter corresponding to the linear layer to obtain model parameters of the second data type; determining the gradient of the second data type based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained by completing the forward process; and updating the model parameters of the model using the gradient of the second data type to obtain a trained model.

[0005] In some embodiments, the method further includes: determining an input maximum value based on an input tensor of the second data type; and determining a scaling parameter based on the input maximum value and a type maximum value corresponding to the first data type.

[0006] In some embodiments, the input tensor of the second data type is type-converted based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type, including: determining the target input tensor based on the scaling parameter and the input tensor of the second data type; and type-converting the target input tensor to obtain the input tensor of the first data type.

[0007] In some embodiments, the model parameters of the first data type are type-converted based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type, including: type-converting the model parameters of the first data type to obtain target model parameters; and determining the model parameters of the second data type based on the target model parameters and the scaling parameters.

[0008] In some embodiments, the model parameters of the model are updated using the gradient of the second data type to obtain a trained model, including: determining a target gradient based on the gradient of the second data type and a scaling parameter; performing type conversion on the target gradient to obtain a gradient of the first data type; and updating the model parameters of the model based on the gradient of the first data type to obtain a trained model.

[0009] In some embodiments, the gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type, including: determining the current gradient based on the model parameters of the second data type and the loss value of the second data type; determining the current first-order momentum and the current second-order momentum based on the current gradient, the first-order statistical value and the second-order statistical value; adjusting the current first-order momentum based on the first-order statistical value to obtain an adjusted first-order momentum, and adjusting the current second-order momentum based on the second-order statistical value to obtain an adjusted second-order momentum; determining the gradient of the second data type based on the adjusted first-order momentum and the adjusted second-order momentum.

[0010] In some embodiments, the first data type is a floating point type of a first precision, the second data type is a floating point type of a second precision, and the first precision is smaller than the second precision.

[0011] In some embodiments, the first data type is an 8-bit floating point number FP8 type, and the second data type is a 16-bit floating point number FP16 or a 32-bit floating point number FP32 type.

[0012] In some embodiments, the initialized model parameters are of FP32 type, and the method further includes: performing type conversion on the initialized model parameters to obtain FP8 type model parameters.

[0013] On the other hand, an embodiment of the present application provides a model training device, wherein the model includes a linear layer, and the linear layer includes model parameters of a first data type; the device includes: a first conversion module, which is used to perform type conversion on the input tensor of the second data type based on the scaling parameters corresponding to the linear layer in a forward process to obtain the input tensor of the first data type; after the linear layer completes the calculation, the output tensor of the first data type is converted into the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; a second conversion module, which is used to perform type conversion on the model parameters of the first data type based on the scaling parameters corresponding to the linear layer in a reverse process to obtain the model parameters of the second data type; the gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained by completing the forward process; an update module, which is used to update the model parameters of the model using the gradient of the second data type to obtain a trained model.

[0014] On the other hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0015] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0016] On the other hand, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implements some or all of the steps in the above method when the computer program or instructions are executed by a processor.

[0017] In an embodiment of the present application, in the forward process, the input tensor of the second data type is converted based on the scaling parameter to obtain the input tensor of the first data type, thereby achieving the data type unification of the input tensor and the model parameter in the linear layer, and obtaining an accurate output tensor. After the linear layer completes the calculation, the output tensor of the first data type is converted to the output tensor of the second data type, which facilitates the calculation of the subsequent model layer. In the reverse process, the model parameters of the first data type are converted based on the scaling parameter to obtain the model parameters of the second data type, thereby achieving the data type unification of the model parameters and the loss value, and thereby obtaining an accurate gradient. The model parameters are updated using the gradient of the second data type, and the trained model can be finally obtained. Compared with the related art, the embodiment of the present application can realize model training based on the model parameters and input tensors of the first data type, and the precision of the first data type is less than that of the second data type. Therefore, while ensuring the accuracy of the model, the efficiency of model training is improved, and the requirements for computing power and storage during model training are reduced.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0020] Figure 1 Schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 1 ; Figure 2 Schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 2 ; Figure 3 Schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 3 ; Figure 4 Schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 4 ; Figure 5 Schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 5 ; Figure 6 Schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 6 ; Figure 7 A schematic diagram of model parameter conversion provided in an embodiment of the present application; Figure 8 A schematic diagram of tensor conversion during forward training provided in an embodiment of the present application; Figure 9 A schematic diagram of tensor conversion during reverse training provided in an embodiment of the present application; Figure 10 A schematic diagram of the structure of a model training device provided in an embodiment of the present application; Figure 11 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or sequence of "first / second / third" may be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing this application only and are not intended to limit this application.

[0024] The present application provides a model training method that can be executed by a processor of a computer device. The computer device may refer to a device with data processing capabilities, such as a server, a laptop, a tablet, a desktop computer, a smart TV, a set-top box, or a mobile device (e.g., a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, or a portable gaming device). The processor may be, but is not limited to, a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC).

[0025] Figure 1 Schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes the following steps S101 to S103: Step S101: In the forward process, the input tensor of the second data type is converted based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type; after the linear layer completes the calculation, the output tensor of the first data type is converted to the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type.

[0026] The model can be any neural network model that needs to be trained, and the model includes at least one linear layer, and each linear layer includes model parameters of the first data type. The number of model parameters of each linear layer can be arbitrary, and the embodiments of the present application are not limited thereto. The linear layer is a layer that performs linear operations. Exemplarily, the linear layer can be a convolution layer (convolution) or a regularization layer (layer_norm), and the first data type can be an FP8 data type. Model parameters may include, but are not limited to, weights and biases.

[0027] Different data types have different numerical ranges and precision. A scaling parameter is an adjustment factor corresponding to a model layer. It is used to adjust model parameters or other tensors when converting between different data types, so that the range of the adjusted values is more reasonable and the numerical precision is close to the original precision.

[0028] In some embodiments, the numerical ranges of different data types are different, and the numerical precision is also different. If the data type conversion is performed directly, the numerical values of the converted data type may be missing or the numerical distribution may be unreasonable. For example, directly converting the FP16 data type to the FP8 data type may reduce the numerical range and may result in missing values; directly converting the FP8 data type to the FP16 data type may increase the numerical range and may result in unreasonable numerical distribution. Therefore, it is necessary to consider the scaling parameter when performing data type conversion.

[0029] The input tensor is the tensor consisting of the input data of the model layer, and the output tensor is the tensor consisting of the output data of the model layer.

[0030] In some embodiments, the latest GPU hardware architecture already supports lower data types (for example, the FP8 data type). Compared with data types such as FP32 and FP16, using FP8 in deep learning large model training scenarios can reduce the demand for computing resources and achieve higher training efficiency. However, because the representation range and accuracy of the FP8 data type are still relatively low, the FP8 data type cannot usually be used directly for model training.

[0031] In some embodiments, the linear layer includes model parameters of a first data type, while the input tensor of the linear layer is a second data type. In order to be able to perform effective operations on the input tensor, it is necessary to unify the data types of the model parameters and the input tensor. Therefore, during the forward training process of the model, it is necessary to perform type conversion on the input tensor of the second data type to obtain the input tensor of the first data type. In this way, within the linear layer, based on the model parameters of the first data type and the input tensor of the first data type, calculations can be performed to obtain an output tensor of the first data type. After the linear layer completes the calculation, the output tensor of the first data type needs to be converted to an output tensor of the second data type for calculation in the next model layer.

[0032] The precision of the first data type is smaller than the precision of the second data type; the data range and precision of the first data type are smaller than the data range and precision of the second data type. For example, the first data type is an FP8 data type and the second data type is an FP16 / FP32 data type. Accordingly, the storage space occupied by data of the first data type is smaller than the storage space occupied by data of the second data type.

[0033] In some embodiments, the input tensor of the second data type is multiplied by the scaling parameter to obtain an intermediate input tensor of the second data type, and the intermediate input tensor of the second data type is converted to an input tensor of the first data type using a hardware-provided cast method. The cast method is a data type conversion function commonly used in the art.

[0034] In some embodiments, after the linear layer completes the calculation, the output tensor of the first data type is converted to the output tensor of the second data type by a cast method.

[0035] For example, the input tensor input tensor1 of the second data type is of FP16 data type, and the scaling parameter is scale. Input tensor1 and scale are multiplied to obtain the target input tensor input tensor2. Input tensor2 is then converted to input tensor3 of the first data type using the cast method. Input tensor3 is of FP8 data type. Based on input tensor3 and the model parameters weight and bias of the FP8 data type, calculations are performed within the linear layer to output the output tensor input tensor4 of FP8 data type. After the linear layer completes the calculations, the FP8 data type input tensor4 is converted to the FP16 data type input tensor5 using the cast method.

[0036] In some embodiments, the model also includes an activation layer. In the forward process, when passing through the activation layer, the output tensor of the first data type output by the previous layer needs to be converted into an input tensor of the second data type through a cast method, and then input into the activation layer to ensure calculation accuracy.

[0037] The above model is a model for image processing, that is, an image processing model. During the training process, the initial image processing model can be trained using an image sample set, which includes multiple image samples and a label corresponding to each image sample.

[0038] In some embodiments, the input tensor of the second data type is an image sample (such as RGB pixel values) loaded directly from an image sample set, which is stored in a high-precision format (FP32) after preprocessing (normalization, cropping, flipping, etc.). Exemplarily, the format of the input tensor of the second data type can be [Batch_size, Channels, Height, Width], for example, [32, 3, 224, 224], where the numerical range can be a normalized pixel value, such as [0, 1] or [-1, 1], and the data type is torch.float32. Here, due to the use of high-precision data, image details and complete numerical information after preprocessing can be retained, avoiding information loss in the early stages. Correspondingly, the input tensor of the first data type is a low-precision image sample obtained by reducing the image sample in the high-precision format.

[0039] In some embodiments, the output tensor of the first data type may be the direct output of a linear layer performing a matrix operation (e.g., y = Wx + b) using the input tensor of the first data type (FP16) and the model parameters of the first data type (FP16). For example, the linear layer may be a convolutional layer, a fully connected layer, or the like. For example, using the fully connected layer of an image classification model, the output tensor of the first data type may be feature data carrying the classification result. Correspondingly, the output tensor of the second data type may be feature data carrying the classification result in a high-precision format obtained by amplifying the low-precision feature data carrying the classification result.

[0040] Step S102: In the reverse process, the model parameters of the first data type are converted based on the scaling parameters corresponding to the linear layer to obtain model parameters of the second data type; the gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained by completing the forward process.

[0041] In some embodiments, the model performs calculations on the input tensor through multiple model layers, ultimately outputting a model prediction value. A loss function is used to calculate the difference between the model prediction value and the true label value to obtain a loss value. Since both the model prediction value and the true label value are of the second data type, the obtained loss value is also of the second data type.

[0042] In some embodiments, after obtaining the loss value, the model needs to undergo backpropagation. This backpropagation process begins at the output layer and propagates the gradient of the loss function (i.e., the partial derivative of the loss value with respect to the output layer parameters) backward through each layer of the network. During this process, the chain rule is used to calculate the gradient of each layer's parameters. During this backward process, when calculating the gradient of the activation layer, the second data type is used for calculation and storage to ensure training accuracy.

[0043] In some embodiments, during the backward process, the model parameters of the linear layer are of a first data type, while the loss value is of a second data type. Therefore, when calculating the gradient of the linear layer, the model parameters of the first data type are first converted to obtain the model parameters of the second data type. The gradient of the second data type is then determined based on the model parameters of the second data type and the loss value of the second data type.

[0044] In some embodiments, the model parameters of the first data type are converted into intermediate model parameters of the second data type using a cast method, and then the intermediate model parameters of the second data type are divided by a scaling parameter corresponding to the linear layer to obtain the model parameters of the second data type. The gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type.

[0045] Exemplarily, the model parameters weight1 and bias1 of the first data type are of FP8 data type, and weight1 and bias1 are converted to model parameters weight2 and bias2 of the second data type by the cast method, and weight2 and bias2 are of FP16 data type. Dividing weight2, bias2 and the scaling parameter scale, the model parameters weight3 and bias3 of the second data type are obtained, and weight3 and bias3 are of FP16 data type. The loss value of the second data type is also of FP16 data type. Based on weight3, bias3 and the loss value of the second data type, the gradient of the second data type is determined.

[0046] In some embodiments, the above loss value is obtained by calculating the difference between the result of the forward process output corresponding to the image sample and the label corresponding to the image sample through a preset loss function.

[0047] Step S103: Use the gradient of the second data type to update the model parameters of the model to obtain a trained model.

[0048] In some embodiments, after calculating the gradients in the backward pass, the model parameters of the linear layer need to be updated using the gradients. The gradients are then restored for the next round of forward and backward training. The gradients calculated and stored in the backward pass are of the second data type, while the model parameters are of the first data type. Therefore, when updating the parameters, the data types of the gradients and model parameters need to be unified.

[0049] In some embodiments, the gradient of the second data type is multiplied by a scaling parameter to obtain an intermediate gradient of the second data type, the intermediate gradient of the second data type is converted into a gradient of the first data type through a cast method, and then the model parameters of the first data type are updated based on the gradient of the first data type to obtain a trained model.

[0050] In some embodiments, when updating parameters, after obtaining the gradient of the first data type, the gradient of the first data type is multiplied by the learning rate to obtain an intermediate result, and the intermediate result is subtracted from the original model parameters to obtain the updated model parameters.

[0051] Exemplarily, the gradient of the second data type is grad1, the model parameter of the first data type is w0, the scaling parameter is scale, and the learning rate is lr. Grad1 is converted by the cast method to obtain the gradient of the first data type cast(grad1*scale). Based on cast(grad1*scale) and lr, the updated model parameter w1=w0-lr*cast(grad1*scale) is determined.

[0052] In some embodiments, after the model is trained using the above-mentioned image sample set to obtain a trained model, the trained model can be used to perform the corresponding image processing task. The embodiments of the present application can accelerate the model training process corresponding to the image processing task. At the same time, due to the use of input tensors and output tensors of the second data type (high precision), the parameters of the image processing model can be updated with the precision of the second data type during the training process, thereby making the trained model more consistent with the task requirements of the image processing task and improving the efficiency of model training.

[0053] During the above-described model training process, the model parameters of the first data type and the gradients of the second data type are both stored in a storage component within the computer device. This storage component may be temporary storage or volatile memory; illustratively, the storage component may be video memory within a graphics processor or system memory. To achieve high-precision data training (i.e., utilizing the gradients of the second data type to ensure training accuracy), in order to reduce the use of storage resources within the storage component, the above-described embodiment utilizes model parameters of the first data type to complete model training, compared to prior art methods that directly utilize model parameters of the second data type to complete model training, thereby reducing the use of storage resources within the storage component.

[0054] In some embodiments, the method further includes: determining a target strategy between the first training strategy and the second training strategy in response to a training instruction; and implementing a training process for the accuracy of the second data type based on the target strategy to obtain a trained model.

[0055] The first training strategy is configured to train a model whose linear layer includes model parameters of a first data type, and to allocate a first cache space in a storage component during training; the first cache space is configured to store the model parameters of the first data type and the gradients of the second data type. In some embodiments, the first cache space is further configured to store momentum, such as first-order momentum and second-order momentum, used in determining the gradients of the second data type.

[0056] The second training strategy is configured to train a model whose linear layer includes model parameters of a second data type, and to allocate a second cache space in the storage component during training; the second cache space is configured to store the model parameters of the second data type and the gradients of the second data type. In some embodiments, the second cache space is further configured to store momentum, such as first-order momentum and second-order momentum, used in determining the gradients of the second data type.

[0057] Here, the first cache space is smaller than the second cache space.

[0058] In an embodiment of the present application, in the forward process, the input tensor of the second data type is converted based on the scaling parameter to obtain the input tensor of the first data type, thereby achieving the data type unification of the input tensor and the model parameter in the linear layer, and obtaining an accurate output tensor. After the linear layer completes the calculation, the output tensor of the first data type is converted to the output tensor of the second data type, which facilitates the calculation of the subsequent model layer. In the reverse process, the model parameters of the first data type are converted based on the scaling parameter to obtain the model parameters of the second data type, thereby achieving the data type unification of the model parameters and the loss value, and thereby obtaining an accurate gradient. The model parameters are updated using the gradient of the second data type, and the trained model can be finally obtained. Compared with the related art, the embodiment of the present application can realize model training based on the model parameters and input tensors of the first data type, and the precision of the first data type is less than that of the second data type. Therefore, while ensuring the accuracy of the model, the efficiency of model training is improved, and the requirements for computing power and storage during model training are reduced.

[0059] Figure 2 This is a schematic diagram of the implementation process of a model training method provided in the embodiment of the present application. Figure 2 , the method can be executed by a processor of a computer device. Figure 1 , the method also includes step S201 and step S202.

[0060] Step S201: Determine an input maximum value based on the input tensor of the second data type.

[0061] In some embodiments, the scaling parameter of the linear layer is related to the input tensor of the linear layer, and the scaling parameter of the linear layer is determined based on the maximum value of the tensor input to the linear layer. Therefore, it is necessary to determine the input maximum value based on the input tensor of the second data type.

[0062] The input tensor includes all the data input to the linear layer.

[0063] For example, the input tensor T is a one-dimensional array. First, the absolute value abs(T) of the input tensor T is taken, and then the maximum value of the input tensor after taking the absolute value is max(abs(T)) as the input maximum value. The values in the input tensor T can be positive, negative, or 0.

[0064] Step S202: Determine the scaling parameter based on the input maximum value and the type maximum value corresponding to the first data type.

[0065] The first data type has a corresponding data range, and the corresponding type maximum value can be directly obtained based on the first data type. For example, the first data type is an FP8 data type, and when the FP8 data type is E5M2, the type maximum value is 57334.

[0066] In some embodiments, the type maximum value corresponding to the first data type is divided by the input maximum value to obtain a scaling parameter. It is understood that when the type maximum value is less than the input maximum value, the scaling parameter is less than 1, and when the type maximum value is greater than the input maximum value, the scaling parameter is greater than 1.

[0067] In some embodiments, the model includes at least two linear layers, each linear layer has a corresponding scaling parameter, and different scaling parameters are independent of each other. The scaling parameter is a 32-bit floating point number initialized to 1.0.

[0068] In the embodiment of the present application, the maximum input value is determined based on the input tensor of the second data type, and the scaling parameter is determined based on the maximum input value and the type maximum value corresponding to the first data type. In this way, an accurate scaling parameter can be obtained based on the input tensor of the linear layer.

[0069] Figure 3 This is a schematic diagram of the implementation process of a model training method provided in the embodiment of the present application. Figure 3 , the method can be executed by a processor of a computer device. Figure 1 , the step S101 of "converting the input tensor of the second data type based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type" can be updated to S301 to S302, which will be combined with Figure 3 The steps shown are explained.

[0070] Step S301: Determine a target input tensor based on a scaling parameter and an input tensor of a second data type.

[0071] In some embodiments, before performing type conversion on the input tensor of the second data type, the input tensor of the second data type needs to be adjusted using a scaling parameter to obtain a target input tensor.

[0072] The scaling parameter is multiplied by the input tensor of the second data type to obtain the target input tensor. It can be understood that the scaling parameter is multiplied by each input data in the input tensor.

[0073] Step S302: Perform type conversion on the target input tensor to obtain an input tensor of the first data type.

[0074] The target input tensor is type-converted by using the cast method provided by the hardware to obtain an input tensor of the first data type.

[0075] Exemplarily, the input tensor T is of FP16 data type, and the scaling parameter is scale. First, T is multiplied by scale to obtain the target input tensor T*scale. The target input tensor is converted to the first data type using the cast method to obtain the input tensor cast(T*scale). cast(T*scale) is of FP8 data type.

[0076] In the embodiment of the present application, a target input tensor is determined based on a scaling parameter and an input tensor of a second data type; the target input tensor is type-converted to obtain an input tensor of the first data type. In this way, the input tensor can be adjusted based on the scaling parameter to obtain an accurately type-converted input tensor.

[0077] Figure 4 This is a schematic diagram of the implementation process of a model training method provided in the embodiment of the present application. Figure 4 , the method can be executed by a processor of a computer device. Figure 1 , the step S102 of "converting the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type" can be updated to S401 to S402, combined with Figure 4 The steps shown are explained.

[0078] Step S401: Perform type conversion on the model parameters of the first data type to obtain target model parameters.

[0079] In some embodiments, when converting model parameters of a first data type to model parameters of a second data type, it is necessary to first perform type conversion on the model parameters of the first data type using a cast method to obtain target model parameters of the second data type. The target model parameters are then adjusted to obtain model parameters of the second data type.

[0080] Step S402: Determine model parameters of the second data type based on the target model parameters and the scaling parameters.

[0081] After obtaining the target model parameters of the second data type, the target model parameters need to be adjusted to better match the second data type. Here, the target model parameters are divided by the scaling parameter to obtain the model parameters of the second data type.

[0082] Exemplarily, the model parameter weight is of FP8 data type, and the model parameter weight is converted to the target model parameter cast(weight) through the cast method, cast(weight) is of FP16 data type, the scaling parameter is scale, and cast(weight) and scale are divided to obtain the model parameter cast(weight) / scale of the second data type, and cast(weight) / scale is of FP16 data type.

[0083] In the embodiment of the present application, the model parameters of the first data type are converted to obtain target model parameters; and the model parameters of the second data type are determined based on the target model parameters and the scaling parameters. In this way, by first converting the model parameters and then adjusting them based on the scaling parameters, accurate model parameters of the second data type can be obtained.

[0084] Figure 5 This is a schematic diagram of the implementation process of a model training method provided in the embodiment of the present application. Figure 5 , the method can be executed by a processor of a computer device. Figure 1 , Figure 1 S103 in can be updated to S501 to S503, which will be combined Figure 5 The steps shown are explained.

[0085] Step S501: Determine a target gradient based on the gradient of the second data type and the scaling parameter.

[0086] In some embodiments, the calculated gradient is of the second data type, while the model parameters are of the first data type. Therefore, when updating the model parameters based on the gradient, the gradient must be converted from the second data type to the first data type. When converting the gradient type, the gradient of the second data type must first be adjusted based on a scaling parameter to obtain a target gradient.

[0087] The target gradient is obtained by multiplying the gradient of the second data type by the scaling parameter. The target gradient is the second data type.

[0088] Step S502: Perform type conversion on the target gradient to obtain a gradient of a first data type.

[0089] After obtaining the target gradient, the target gradient of the second data type is directly converted by the cast method to obtain the gradient of the first data type.

[0090] Step S503: Update the model parameters of the model based on the gradient of the first data type to obtain a trained model.

[0091] After obtaining the gradient of the first data type, the model parameters are also the first data type. Therefore, the model parameters of the model can be updated based on the gradient of the first data type to obtain a trained model.

[0092] In some embodiments, the gradient of the first data type is multiplied by the learning rate to obtain an intermediate result, which is then subtracted from the current model parameters to obtain updated model parameters. For each model parameter, the model parameter is updated based on the corresponding gradient of the first data type, ultimately obtaining a trained model.

[0093] For example, the current model parameter is weight W0, the learning rate is lr, the scaling parameter is scale, the gradient of the second data type is grad, the target gradient is grad*scale, the gradient of the first data type is cast(grad*scale), and the updated weight W1 is W0-lr*cast(grad*scale).

[0094] In this embodiment of the present application, a target gradient is determined based on the gradient of the second data type and a scaling parameter; the target gradient is converted to obtain the gradient of the first data type; and the model parameters of the model are updated based on the gradient of the first data type to obtain a trained model. This unifies the data types of the model gradient and the model parameters, and accurately updates the model parameters based on the converted model gradient.

[0095] Figure 6 This is a schematic diagram of the implementation process of a model training method provided in the embodiment of the present application. Figure 6 , the method can be executed by a processor of a computer device. Figure 1 , Figure 1 The step of “determining the gradient of the second data type based on the model parameters of the second data type and the loss value of the second data type” can be updated to S601 to S604, which will be combined with Figure 6 The steps shown are explained.

[0096] Step S601: Determine a current gradient based on the model parameters of the second data type and the loss value of the second data type.

[0097] The current gradient is the gradient at the current time step (current iteration) during model training. After obtaining the model parameters of the second data type and the loss value of the second data type, the current gradient corresponding to the model parameters can be determined using the backpropagation algorithm and the chain rule. The current gradient is of the second data type, and each model parameter corresponds to a current gradient.

[0098] Step S602: Determine a current first-order momentum and a current second-order momentum based on the current gradient, the first-order statistics, and the second-order statistics.

[0099] During model training, an optimizer is used to update model parameters, optimizing the model's weights and biases to make the model's predictions more accurate. The optimizer determines the method and rate of parameter updates, which in turn affects the model's convergence speed and ultimate performance.

[0100] In some embodiments, during model training, the optimizer needs to save first-order and second-order statistics, which are used to determine first-order momentum and second-order momentum. First-order momentum introduces a momentum term during the iteration process, accelerating the gradient descent process by accumulating past gradient information; second-order momentum typically refers to optimization algorithms that take into account past squared gradients and can adaptively adjust the learning rate of each parameter.

[0101] In some embodiments, the current first-order momentum is determined based on the current gradient, the first-order momentum of the previous time step, and the first-order statistics. The current first-order momentum is determined by formula (1): (1); in, is the current first-order momentum, is the first-order momentum of the previous time step, is the first-order statistic, is the current gradient.

[0102] In some embodiments, the current second-order momentum is determined based on the current gradient, the second-order momentum of the previous time step, and the second-order statistics. The current second-order momentum is determined by formula (2): (2); in, is the current second-order momentum, is the second-order momentum of the previous time step, is the second-order statistic, is the current gradient.

[0103] Step S603: Adjust the current first-order momentum based on the first-order statistical value to obtain an adjusted first-order momentum, and adjust the current second-order momentum based on the second-order statistical value to obtain an adjusted second-order momentum.

[0104] Among them, the initial first-order momentum and the initial second-order momentum are initialized to 0. In the initial stage, and It will be biased towards 0. In order to correct this deviation, a deviation correction term is introduced to adjust the current first-order momentum and the current second-order momentum.

[0105] In some embodiments, an adjusted first-order momentum is determined based on the first-order statistics and the current first-order momentum. The adjusted first-order momentum is determined by formula (3): (3); in, is the adjusted first-order momentum.

[0106] In some embodiments, based on the second-order statistics and the current second-order momentum, an adjusted second-order momentum is determined. The adjusted second-order momentum is determined by formula (4): (4); in, is the adjusted second-order momentum.

[0107] Step S604: Determine the gradient of the second data type based on the adjusted first-order momentum and the adjusted second-order momentum.

[0108] After obtaining the adjusted first-order momentum and the adjusted second-order momentum, the gradient of the second data type may be determined based on the adjusted first-order momentum and the adjusted second-order momentum.

[0109] In some embodiments, the gradient of the second data type is determined by formula (5): (5); in, is the adjusted first-order momentum, is the adjusted second-order momentum, grad is the adjusted gradient, and ϵ is a very small number.

[0110] In the embodiment of the present application, the current gradient is first determined. Based on the current gradient, the first-order statistics, and the second-order statistics, the current first-order momentum and second-order momentum are determined. The first-order momentum and second-order momentum are then adjusted to determine the gradient of the second data type. In this way, an accurate gradient can be determined based on the momentum statistics, and model parameters can be optimized to achieve more accurate model prediction results.

[0111] In some embodiments, the first data type is a floating point type of a first precision, the second data type is a floating point type of a second precision, and the first precision is smaller than the second precision.

[0112] The model parameter is of a first data type, which is a floating-point type of first precision, and illustratively, the first data type is an FP8 data type. The model input tensor is of a second data type, which is a floating-point type of second precision, and illustratively, the second data type is an FP16 data type.

[0113] The mantissa of a floating-point type can represent the precision of the floating-point type. More mantissa bits can bring higher numerical precision. The first precision is smaller than the second precision. For example, the first data type is the FP8 data type. The mantissa of the E5M2 format in the FP8 data type has 2 bits, and the mantissa of the E4M3 format has 3 bits. The second data type is the FP16 data type. The mantissa of the FP16 data type has 10 bits. Therefore, the first precision is smaller than the second precision.

[0114] Among them, the first data type and the second data type are both floating-point types. In this way, the design circuit of the first data type and the design circuit of the second data type can be reused to a certain extent, and the conversion circuit between the first data type and the second data type can be designed more simply and directly without the overhead of multiplication and addition required for conversion from integer to floating-point type.

[0115] Among them, the first precision is smaller than the second precision. In this way, the bit width of the first data type is smaller than the bit width of the second data type. During model training, training can be based on the first data type with a smaller bit width. Compared with the second data type, using the first data type during model training can reduce the demand for computing resources and achieve higher training efficiency.

[0116] In the embodiment of the present application, the first data type is a floating-point type of first precision, and the second data type is a floating-point type of second precision, where the first precision is less than the second precision. This simplifies the conversion circuit between different data types, reduces the demand for computing resources during model training, and achieves higher training efficiency.

[0117] In some embodiments, the first data type is an 8-bit floating point number FP8 type, and the second data type is a 16-bit floating point number FP16 or a 32-bit floating point number FP32 type.

[0118] Among them, in related technologies, deep learning models mainly use FP32 or FP16 data types for model training. As the number of model parameters increases, its consumption of computing resources is also becoming higher and higher. At present, many of the latest GPU hardware architectures already support the FP8 data type. Under the same hardware transmission bandwidth, it can transmit more data than FP16 and 16-bit brain floating point BF16 data types, and has higher calculation accuracy than the 8-bit integer INT8 data type. Therefore, the first data type is FP8 type and the second data type is FP16 type, which can improve training efficiency and reduce resource consumption while meeting the current training data.

[0119] The FP8 data type has two formats: E5M2 and E4M3. (ExMx here indicates the number of bits used to represent the exponent and mantissa of 8-bit data; E5M2 uses 5 bits for the exponent, 2 bits for the mantissa, and 1 sign bit for a total of 8 bits.) These two formats differ in their range and precision. Generally speaking, more exponent bits result in a wider range of values, while more mantissa bits result in higher precision. For large language models, the FP8 format is set to E5M2, while for vision models, the FP8 format is set to E4M3.

[0120] Among them, the FP16 data type includes 1 sign bit, 5 exponent bits, and 10 mantissa bits. FP16 is a half-precision floating-point data type that uses 16 binary bits to represent floating-point numbers. It has lower precision and numerical range, but has significant memory usage and computing time advantages in deep learning and high-performance computing.

[0121] FP32 uses 32-bit binary to represent floating-point numbers, which includes a sign bit, 8-bit exponent bits, and 23-bit mantissa bits. This representation method enables FP32 to provide high precision and dynamic range, suitable for most scientific computing and general computing tasks.

[0122] In the embodiment of the present application, the first data type is FP8 type, and the second data type is FP16 or FP32 type. In this way, converting between the first data type and the second data type can meet the requirements of input training data, improve training efficiency, and reduce resource consumption.

[0123] In some embodiments, the initialized model parameters are of FP32 type, and the method further includes: performing type conversion on the initialized model parameters to obtain FP8 type model parameters.

[0124] For the initialized model, its model parameters are usually FP32 data types. To enable training based on the FP8 data type, the model parameters must first be converted to the FP8 data type. This conversion can be performed directly using the cast method, without involving actual hardware calculations. The converted FP8 data type is the first data type.

[0125] Exemplarily, the initialized model parameters include weight weight0 and bias bias0, the initialized weight weight0 and bias are of FP32 data type, weight0 and bias0 are converted to weight1 and bias1 through the cast method, weight1 and bias1 are of FP8 data type, and then training is performed based on weight1 and bias1 of FP8 data type.

[0126] In the embodiment of the present application, the model parameters are initialized to FP32 type, and the initialized model parameters are converted to FP8 type model parameters. In this way, when the model is initialized, the model parameters are converted and training is performed based on the converted model parameters, which can improve training efficiency and reduce resource consumption.

[0127] The following describes the application of the model training method provided in the embodiments of the present application in actual scenarios.

[0128] Currently, deep learning models primarily utilize FP32 and FP16 data types for model training. As the number of model parameters continues to grow, the computing power consumption is also increasing. Therefore, how to train larger models with limited computing resources has become a research topic in the industry.

[0129] Currently, many of the latest GPU hardware architectures already support the FP8 data type (8-bit floating point). Under the same hardware transmission bandwidth, it can transmit more data per unit time and has higher calculation accuracy than the INT8 (8-bit integer) data type. However, compared with the currently common FP32, FP16 and other data types, the representation range and accuracy of FP8 are still insufficient for direct model training. Therefore, the embodiment of this application proposes a model training method for the FP8 data type (corresponding to the first data type in the above embodiment), by: 1. Dynamically scale the activation value to adjust its range to make it more suitable for FP8 data type; 2. A new optimizer is designed for FP8 data type training to minimize video memory usage while ensuring training accuracy.

[0130] In this way, the model can make full use of the FP8 data type during training, achieve higher training efficiency, and reduce resource consumption during the training process.

[0131] Solutions for training low-precision models typically rely on quantization-aware training using the INT8 data type. This involves inserting observation points into the model, recording changes in activation values during training, and performing INT8 quantization on weights based on these statistics. This approach doesn't significantly reduce computing resource consumption, and it requires dynamic data type conversion of tensors like weights and activation values, resulting in low training efficiency.

[0132] The latest GPU hardware architecture already supports the FP8 data type. Compared with data types such as FP32 and FP16, using FP8 in large deep learning model training scenarios can reduce the demand for computing resources and achieve higher training efficiency. However, because the representation range and precision of the FP8 data type are still lower than those of FP16 and BF16, the FP8 data type cannot usually be used directly for model training.

[0133] The model training method based on the FP8 data type in the embodiment of the present application has higher training efficiency, and the FP8 data type has a better dynamic range and accuracy than the INT8 format used in the former, and the training effect of the final model will also be better.

[0134] The FP8 data type has a performance improvement of nearly 2 times compared to 16-bit data types such as FP16. As an 8-bit data type, FP8 has some special advantages over INT8: 1. FP8 is a floating-point type. The design circuit of FP8 MAC can be reused with FP16 to some extent. 2. The conversion circuit between FP8 to FP16 / FP32 / BF16 can be designed to be simpler and more direct, without the overhead of multiplication and addition required for conversion from INT8 / UINT8 to FP.

[0135] Among them, the hardware architecture supports two formats of FP8, including E5M2 and E4M3. The FP8 data type model training method designed in the embodiment of this application sets the FP8 format to E5M2 for large language models and to E4M3 for visual models.

[0136] In some embodiments, for the initialized model, its weights and biases are usually of FP32 data type, so the model weights and biases need to be converted to FP8 data type first. The conversion here can be directly performed through the cast method provided by the hardware, which does not involve actual hardware calculations and is not required here. After the model is converted, a scaling parameter scale needs to be added to all model layers that need to be trained (such as convolution layer, linear layer, batch normalization layer batch_norm, regularization layer layer_norm, etc., which are not required in detail here). This parameter is a 32-bit floating point number initialized to 1.0. It should be noted that each model layer here has a scaling parameter, and each parameter is independent of each other. The scaling parameter represents the scaling factor when converting between different data types. The scale parameter can make the data distribution corresponding to the converted data type more reasonable.

[0137] Figure 7 A schematic diagram of model parameter conversion provided in the embodiment of the present application. Figure 7As shown, the weights 701 and biases 702 in the convolutional layer (Conv2d layer) and the linear layer (Linear layer) in the initialized model are of FP32 data type. The weights 701 and biases 702 in the convolutional layer and the linear layer are converted to weights 703 and biases 704 of FP8 data type through the cast method, and a scaling parameter 705 is added to each of the convolutional layer and the linear layer.

[0138] In some embodiments, during model training, the optimizer needs to save model parameters, model gradients, and momentum statistics at the same time. Model parameters refer to the size of model parameters. For example, a large language model of 7B size has 7 billion model parameters. If the FP32 data type is used, approximately 28GB of storage space is required (each parameter is stored in FP32, and one FP32 data occupies 4 bytes). The size of the model gradient is the same as the model parameter, while the momentum statistics vary depending on the optimizer: for the most commonly used Adam optimizer, first-order momentum and second-order momentum need to be stored, which consumes twice the storage of the model parameters. Therefore, if Adam and FP32 data types are used to train a large language model of any size, the video memory is calculated using formula (6): (4+4+4+4)*model_size = 16*model_size(6); Among them, model_size is the size of model parameters.

[0139] The FP8 training method of the embodiment of the present application converts the model parameters into FP8 data type in advance, reducing the model parameter storage overhead by 75%. For model gradients, it is necessary to keep the gradients in FP32 data type during normal model training to ensure training accuracy. The embodiment of the present application uses FP8 model parameters, and using FP16 as gradients is sufficient to represent parameter information (the numerical precision that FP32 can represent far exceeds FP8, and gradients can be stored in FP16). Therefore, the embodiment of the present application uses FP16 data type to store gradients, reducing the gradient storage requirements by half. For the first-order statistical value (α) in Adam, FP16 is used for storage, and the second-order statistical value (β) still maintains the FP32 data format due to the higher accuracy required. In summary, the video memory required by the new optimizer to train a large language model of any size is calculated by formula (7): (1+2+2+4)*model_size = 9*model_size(7); Among them, the video memory consumption is only 56.25% of the previous level.

[0140] In some embodiments, during the forward pass of FP8 model training, training data is typically preprocessed into FP32 / FP16 data types (corresponding to the second data type of input tensors in the above embodiments). In the present embodiment, the training data type is maintained and is not required. After training is started, before inputting into each model layer with parameters (model layers with parameters are all added with a scale parameter), it is determined whether the input is of FP8 type. If so, the following operations are performed: 1. First, you need to calculate and save the scaling parameters of the layer based on the tensor T input to the model layer. The calculation formulas for the scaling parameters are as follows: scale = float_max / max_T(8); max_T = max(abs(T))(9); Among them, float_max (corresponding to the maximum input value in the above embodiment) is the maximum value of the current FP8 format, such as the maximum value of the E5M2 format is 57334.0, and max_T (corresponding to the maximum type value in the above embodiment) is the maximum absolute value of the tensor.

[0141] 2. Multiply the current input by the calculated scaling factor and then cast it to FP8 type; then you can perform FP8 data type operations (convolution, matrix multiplication, etc.) within this layer.

[0142] Figure 8 This is a schematic diagram of tensor conversion during forward training provided in an embodiment of the present application. Figure 8 As shown, for the linear layer, the initial input tensor 801 (corresponding to the input tensor of the second data type in the above embodiment) is of FP16 type. The scaling parameter 802 can be calculated through the initial input tensor 801. The initial input tensor is multiplied by the scaling parameter to obtain an intermediate input tensor 803 (corresponding to the target input tensor in the above embodiment). The intermediate input tensor 803 is of FP16 type. The intermediate input tensor is converted into a target input tensor 804 of FP8 type (corresponding to the input tensor of the first data type in the above embodiment) through the cast method. After that, FP8 data type operations can be performed in the layer (the FP8 type target input tensor 804 and the FP8 type weight & bias 805 perform convolution, matrix multiplication and other operations) to obtain the FP8 type target output tensor 806. The FP8 type target output tensor 806 (corresponding to the output tensor of the first data type in the above embodiment) is converted into the FP16 type target input tensor 807 (corresponding to the output tensor of the second data type in the above embodiment) through the cast method and input into the next model layer.

[0143] Among them, when the target output tensor 806 is input to the activation layer (the activation layer is usually a nonlinear layer, such as ReLU, GeLU, Logits, etc., which does not involve weight model layers and is not specifically required here), it is determined whether the current input is of FP8 type. If so, the FP8 type target output tensor 806 needs to be cast to the FP16 type target input tensor 807 to ensure calculation accuracy. Figure 8 As shown, the next layer of the linear layer (Linear layer) is the activation layer (GeLU layer), and the FP8 type target output tensor 806 of the linear layer is cast to the FP16 type target input tensor 807, which is input to the GeLU layer. The GeLU layer outputs the FP16 type target output tensor 808.

[0144] For two adjacent linear layers, if the output tensor of the previous linear layer is of FP8 type, it is necessary to cast the output tensor to an FP16 input tensor, and use the FP16 input tensor as the input of the next linear layer. Within the next linear layer, the operation process of the previous linear layer is repeated: first, the scaling parameter of the layer is calculated based on the FP16 input tensor of the layer, and the scaling parameter is multiplied by the input tensor to obtain an intermediate input tensor of FP16 type. The intermediate input tensor is converted to the target input tensor of FP8 type through the cast method, and then FP8 type operations can be performed within the layer.

[0145] In some embodiments, in the reverse process of FP8 model training, based on the forward process until the loss value is calculated, the back propagation process will be performed in the same way as FP32 model training. Usually the gradient needs to be stored in FP32 data format. Using the training method proposed in the embodiment of the present application, the gradient can be calculated and stored in FP16 format (corresponding to the gradient of the second data type in the above embodiment). The specific method is: the initial loss value loss is of FP16 type. For the above-mentioned calculation of the gradient of the model layer without parameters (such as the activation layer, the output logical layer Logits, etc.), the gradient is calculated and stored using FP16 type to ensure the training accuracy. For model layers with FP8 parameters (such as convolutional layer conv, linear layer Linear, etc.), when calculating the gradient, the model parameters need to be cast to FP16 type first, and then divided by the scale parameter saved by the layer, and then the gradient of the model layer is calculated normally.

[0146] Figure 9 This is a schematic diagram of tensor conversion during reverse training provided in an embodiment of the present application. Figure 9As shown, within the Linear layer, the initial weights and biases 901 (corresponding to the model parameters of the first data type in the above embodiment) are of FP8 type. These weights and biases are first cast to FP16 type, resulting in intermediate weights and biases 902 (corresponding to the target model parameters in the above embodiment). These weights and biases are then divided by a scaling parameter 903 to obtain adjusted target weights and biases 904 of FP16 type (corresponding to the model parameters of the second data type in the above embodiment). Simultaneously, the activation layer (GeLU layer) connected to the Linear layer receives the initial FP16 gradient tensor 905 transmitted from the subsequent layer. Based on this initial gradient tensor, it outputs an intermediate FP16 gradient tensor 906, which is then transmitted to the Linear layer. The Linear layer calculates the gradients of the weights and biases based on the received intermediate gradient tensor 906 and the FP16 target weights and biases 904, stores them, and ultimately outputs an FP16 target gradient tensor 907 to the next layer.

[0147] In some embodiments, during the parameter update process of FP8 model training, after the gradient is fully calculated, the optimizer needs to update the gradient to the existing weights and biases, and then restore the gradient for the next training (forward + reverse process). In the stochastic gradient descent algorithm, after each weight update, the gradient is not directly retained for the next update, but the new gradient is recalculated based on the difference between the current model prediction and the true value at the next iteration. In some more complex optimizers, the gradient may be accumulated or adaptively adjusted, but before each new training round begins, there will also be operations related to "resetting" or "initializing" the gradient to ensure that each gradient calculation and update is based on the current model state and data.

[0148] The gradient stored by the optimizer in the embodiment of the present application is of FP16 type, while the model parameters are of FP8 type. Therefore, when updating the weights, the update formula changes from formula (10) to formula (11): W_1 = W_0 - lr * grad (10); W_1 = W_0 - lr * Cast(grad * scale) (11); Among them, W_1 is the FP8 type weight after the update, W_0 is the FP8 type weight before the update, lr is the learning rate, grad is the gradient, scale is the scale parameter saved by the layer, and Cast(*) is the type conversion.

[0149] Specifically, the calculation formula of grad in the Adam optimizer is as follows: (12); (13); (14); (15); (16); in, is the first-order statistic of the optimizer, is the second-order statistics of the optimizer, The current gradient calculated for this weight, is the second-order momentum of the current time step, is the second-order momentum of the previous time step, is the first-order momentum of the current time step, is the first-order momentum of the previous time step, is the adjusted first-order momentum, is the adjusted second-order momentum, grad is the adjusted gradient, For a very small number.

[0150] In some embodiments, you can select any model, such as the currently open-source large language model. After the model is initialized, it is first loaded into the computing device, and then the model's weights and biases are directly converted to FP8 data types. The type conversion here is similar to the conventional FP32 type conversion to FP16 type. It does not involve specific operations and is a direct conversion of data types within the hardware. The FP8 optimizer is set based on the converted FP8 model, and then the data can be loaded to start training. After the training is completed, a model based on the FP8 data type is obtained.

[0151] The embodiment of the present application can use the FP8 data type for model training, while maintaining the same prediction accuracy (approximately the same) as the model trained using FP32 / FP16, improving the training efficiency and reducing the dependence on and consumption of computing resources.

[0152] The embodiments of the present application can fully utilize the FP8 data type to directly perform model training; improve the efficiency of model training while maintaining model accuracy; and reduce the requirements for computing power and storage during model training.

[0153] Based on the foregoing embodiments, an embodiment of the present application provides a model training device, which includes the various units included and the various modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0154] Figure 10 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 10 As shown, the model training device 1000 includes: a first conversion module 1010, a second conversion module 1020, and an update module 1030, wherein: the model includes a linear layer, and the linear layer includes model parameters of the first data type; A first conversion module 1010 is configured to, in a forward process, convert an input tensor of the second data type based on a scaling parameter corresponding to the linear layer to obtain an input tensor of the first data type; after the linear layer completes calculation, convert an output tensor of the first data type to an output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; A second conversion module 1020 is configured to, in a backward process, convert the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain model parameters of a second data type; determine the gradient of the second data type based on the model parameters of the second data type and a loss value of the second data type; the loss value of the second data type is obtained by completing the forward process; The updating module 1030 is configured to update the model parameters of the model using the gradient of the second data type to obtain a trained model.

[0155] In some embodiments, the first conversion module 1010 is further used to determine an input maximum value based on the input tensor of the second data type; and determine the scaling parameter based on the input maximum value and the type maximum value corresponding to the first data type.

[0156] In some embodiments, the first conversion module 1010 is further used to determine a target input tensor based on the scaling parameter and the input tensor of the second data type; and perform type conversion on the target input tensor to obtain an input tensor of the first data type.

[0157] In some embodiments, the second conversion module 1020 is further configured to perform type conversion on the model parameters of the first data type to obtain target model parameters; and determine the model parameters of the second data type based on the target model parameters and the scaling parameters.

[0158] In some embodiments, the update module 1030 is further used to determine a target gradient based on the gradient of the second data type and the scaling parameter; perform type conversion on the target gradient to obtain a gradient of the first data type; and update the model parameters of the model based on the gradient of the first data type to obtain a trained model.

[0159] In some embodiments, the second conversion module 1020 is further used to determine the current gradient based on the model parameters of the second data type and the loss value of the second data type; determine the current first-order momentum and the current second-order momentum based on the current gradient, the first-order statistical value and the second-order statistical value; adjust the current first-order momentum based on the first-order statistical value to obtain an adjusted first-order momentum, and adjust the current second-order momentum based on the second-order statistical value to obtain an adjusted second-order momentum; determine the gradient of the second data type based on the adjusted first-order momentum and the adjusted second-order momentum.

[0160] In some embodiments, the first data type is a floating point type of a first precision, the second data type is a floating point type of a second precision, and the first precision is smaller than the second precision.

[0161] In some embodiments, the first data type is an 8-bit floating point number FP8 type, and the second data type is a 16-bit floating point number FP16 or a 32-bit floating point number FP32 type.

[0162] In some embodiments, the initialized model parameters are of FP32 type, and the first conversion module 1010 is further configured to perform type conversion on the initialized model parameters to obtain FP8 type model parameters.

[0163] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0164] It should be noted that in the embodiments of the present application, if the above-mentioned model training method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0165] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0166] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0167] An embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0168] An embodiment of the present application provides a computer program product, comprising a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, the computer program implements some or all of the steps of the above-described method. The computer program product can be implemented in hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0169] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between the various embodiments, and their similarities or similarities can be referenced to each other. The descriptions of the above device, storage medium, computer program, and computer program product embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the description of the method embodiments of this application for understanding.

[0170] Figure 11 A hardware entity diagram of a computer device provided in an embodiment of the present application is shown as follows: Figure 11 As shown, the hardware entity of the computer device 1100 includes: a processor 1101 and a memory 1102, wherein the memory 1102 stores a computer program that can be run on the processor 1101, and the processor 1101 implements the steps in the method of any of the above embodiments when executing the program.

[0171] The memory 1102 stores computer programs that can be run on the processor. The memory 1102 is configured to store instructions and applications executable by the processor 1101. It can also cache data to be processed or processed by the processor 1101 and various modules in the computer device 1100 (for example, image data, audio data, voice communication data, and video communication data). It can be implemented through flash memory (FLASH) or random access memory (RAM).

[0172] When the processor 1101 executes the program, the steps of any of the above-mentioned model training methods are implemented. The processor 1101 generally controls the overall operation of the computer device 1100.

[0173] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the model training method of any of the above embodiments.

[0174] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0175] The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device that implements the functions of the processor may also be other electronic devices, which are not specifically limited in the embodiments of the present application.

[0176] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface storage device, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0177] The above is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The model includes a linear layer, the linear layer includes model parameters of a first data type, and the model training method includes: In the forward process, the input tensor of the second data type is converted based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type; after the linear layer completes the calculation, the output tensor of the first data type is converted to the output tensor of the second data type; the precision of the first data type is less than the precision of the second data type; In the reverse process, the model parameters of the first data type are converted based on the scaling parameters corresponding to the linear layer to obtain model parameters of the second data type; the gradient of the second data type is determined based on the model parameters of the second data type and the loss value of the second data type; the loss value of the second data type is obtained by completing the forward process; The model parameters of the model are updated using the gradient of the second data type to obtain a trained model.

2. The model training method according to claim 1, characterized in that The model training method further includes: Determine an input maximum value based on the input tensor of the second data type; The scaling parameter is determined based on the input maximum value and a type maximum value corresponding to the first data type.

3. The model training method according to claim 1, characterized in that The performing type conversion on the input tensor of the second data type based on the scaling parameter corresponding to the linear layer to obtain the input tensor of the first data type includes: determining a target input tensor based on the scaling parameter and the input tensor of the second data type; Perform type conversion on the target input tensor to obtain an input tensor of the first data type.

4. The model training method according to claim 1, characterized in that The performing type conversion on the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain the model parameters of the second data type includes: Performing type conversion on the model parameters of the first data type to obtain target model parameters; Model parameters of a second data type are determined based on the target model parameters and the scaling parameters.

5. The model training method according to claim 1, characterized in that The updating of the model parameters of the model using the gradient of the second data type to obtain a trained model includes: determining a target gradient based on the gradient of the second data type and the scaling parameter; Performing type conversion on the target gradient to obtain a gradient of a first data type; The model parameters of the model are updated based on the gradient of the first data type to obtain a trained model.

6. The model training method according to claim 1, characterized in that The determining the gradient of the second data type based on the model parameters of the second data type and the loss value of the second data type includes: determining a current gradient based on the model parameters of the second data type and the loss value of the second data type; Determining a current first-order momentum and a current second-order momentum based on the current gradient, the first-order statistical value, and the second-order statistical value; Adjusting the current first-order momentum based on the first-order statistical value to obtain an adjusted first-order momentum, and adjusting the current second-order momentum based on the second-order statistical value to obtain an adjusted second-order momentum; A gradient of the second data type is determined based on the adjusted first-order momentum and the adjusted second-order momentum.

7. The model training method according to any one of claims 1 to 6, characterized in that: The first data type is a floating point type of a first precision, the second data type is a floating point type of a second precision, and the first precision is smaller than the second precision.

8. The model training method according to any one of claims 1 to 6, characterized in that: The first data type is an 8-bit floating point number FP8 type, and the second data type is a 16-bit floating point number FP16 type or a 32-bit floating point number FP32 type.

9. The model training method according to any one of claims 1 to 6, characterized in that: The initialized model parameters are of FP32 type, and the method further includes: Perform type conversion on the initialized model parameters to obtain FP8 type model parameters.

10. A model training device, characterized in that: The model includes a linear layer, and the linear layer includes model parameters of a first data type; the model training device includes: a first conversion module configured to, in a forward process, convert an input tensor of the second data type based on a scaling parameter corresponding to the linear layer to obtain an input tensor of the first data type; and after the linear layer completes calculation, convert an output tensor of the first data type to an output tensor of the second data type; wherein the precision of the first data type is less than the precision of the second data type; a second conversion module configured to, in a reverse process, convert the model parameters of the first data type based on the scaling parameters corresponding to the linear layer to obtain model parameters of a second data type; and determine the gradient of the second data type based on the model parameters of the second data type and a loss value of the second data type, wherein the loss value of the second data type is obtained by completing the forward process; An updating module is used to update the model parameters of the model using the gradient of the second data type to obtain a trained model.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the model training method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the model training method according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instructions are executed by a processor, the steps in the model training method described in any one of claims 1 to 9 are implemented.

Images