Model training methods, devices, electronic equipment and storage media

Abstract

This application discloses a model training method, apparatus, electronic device, and storage medium. The model training method includes: during the forward propagation process of the world model to be trained, quantizing a forward propagation tensor of a first data format into a forward propagation tensor of a second data format for processing; and during the backward propagation process of the world model to be trained, quantizing the backward propagation tensor of the first data format into a backward propagation tensor of a third data format for processing; wherein the numerical precision of the first data format is higher than that of the second data format, the numerical precision of the second data format is higher than that of the third data format, and the numerical range of the third data format is larger than that of the second data format. In the embodiments of this application, the training efficiency and accuracy of the world model can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, world models, as key models capable of understanding and predicting environmental dynamics, have shown great potential in fields such as video generation, autonomous driving, and robot simulation.

[0003] However, world models typically have a large number of parameters and complex computational graphs, requiring massive amounts of computing resources and lengthy training times, which severely restricts their research, development, and application iteration efficiency. Therefore, improving the training efficiency of world models is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] The purpose of this application is to provide a model training method, apparatus, electronic device, and storage medium that can solve the problem of low training efficiency of world models.

[0005] In a first aspect, embodiments of this application provide a model training method, the method comprising: During the forward propagation process of the world model to be trained, the forward propagation tensor in the first data format is quantized into the forward propagation tensor in the second data format for processing. During the backpropagation process of the training process of the world model to be trained, the backpropagation tensor in the first data format is quantized into the backpropagation tensor in the third data format for processing. The numerical precision of the first data format is higher than that of the second data format, the numerical precision of the second data format is higher than that of the third data format, and the numerical range of the third data format is greater than that of the second data format.

[0006] Optionally, the step of quantizing the forward propagation tensor of the first data format into a forward propagation tensor of the second data format includes: determining a first scaling factor based on the numerical range of the forward propagation tensor of the first data format and the second data format; scaling the forward propagation tensor of the first data format using the first scaling factor; and converting the scaled forward propagation tensor into a forward propagation tensor of the second data format for processing. The step of quantizing the backpropagation tensor of the first data format into a backpropagation tensor of the third data format for processing includes: determining a second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format; scaling the backpropagation tensor of the first data format using the second scaling factor; and converting the scaled backpropagation tensor into a backpropagation tensor of the second data format for processing.

[0007] Optionally, determining the first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format includes: taking the maximum absolute value of the elements in the forward propagation tensor of the first data format as the first target value, and determining the ratio of the upper limit of the numerical range of the second data format to the first target value as the first scaling factor. Determining the second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format includes: taking the maximum absolute value of the elements in the backpropagation tensor of the first data format as the second target value, and determining the ratio of the upper limit of the numerical range of the third data format to the second target value as the second scaling factor.

[0008] Optionally, determining the first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format includes: dividing the forward propagation tensor of the first data format into multiple forward element groups along the output channel dimension, determining a third scaling factor based on the forward element groups and the numerical range of the second data format, and scaling the forward element groups using the third scaling factor. The step of determining the second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format includes: dividing the backpropagation tensor of the first data format into multiple backpropagation element groups along the output channel dimension, determining a fourth scaling factor based on the backpropagation element groups and the numerical range of the third data format, and scaling the backpropagation element groups using the fourth scaling factor.

[0009] Optionally, the process of quantizing the forward propagation tensor of the first data format into a forward propagation tensor of the second data format includes: in the first network layer, processing the forward propagation tensor of the first data format; and in the second network layer, processing the forward propagation tensor of the first data format into a forward propagation tensor of the second data format. The process of quantizing the backpropagation tensor of the first data format into the backpropagation tensor of the third data format includes: in the first network layer, processing the backpropagation tensor of the first data format; and in the second network layer, processing the backpropagation tensor of the first data format into the backpropagation tensor of the third data format. The first network layer is more sensitive to quantization than the second network layer.

[0010] Optionally, the first network layer includes at least one of an input layer and an output layer.

[0011] Optionally, the second data format is the E4M3 data format of 8-bit floating-point numbers, and the third data format is the E5M2 data format of 8-bit floating-point numbers.

[0012] Secondly, embodiments of this application provide a model training apparatus, the apparatus comprising: The first processing module is used to quantize the forward propagation tensor of the first data format into the forward propagation tensor of the second data format for processing during the forward propagation process of the training process of the world model to be trained. The second processing module is used to quantize the backpropagation tensor of the first data format into the backpropagation tensor of the third data format for processing during the backpropagation process of the training process of the world model to be trained. The numerical precision of the first data format is higher than that of the second data format, the numerical precision of the second data format is higher than that of the third data format, and the numerical range of the third data format is greater than that of the second data format.

[0013] Optionally, the first processing module includes: a first quantization unit, configured to determine a first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format, scale the forward propagation tensor of the first data format using the first scaling factor, and convert the scaled forward propagation tensor into a forward propagation tensor of the second data format for processing; The second processing module includes: a second quantization unit, configured to determine a second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format, scale the backpropagation tensor of the first data format using the second scaling factor, and convert the scaled backpropagation tensor into a backpropagation tensor of the second data format for processing.

[0014] Optionally, the first quantization unit is specifically used to take the maximum absolute value of the elements in the forward propagation tensor of the first data format as the first target value, and to determine the ratio of the upper limit of the numerical range of the second data format to the first target value as the first scaling factor. The second quantization unit is specifically used to take the maximum absolute value of the elements in the backpropagation tensor of the first data format as the second target value, and to determine the ratio of the upper limit of the numerical range of the third data format to the second target value as the second scaling factor.

[0015] Optionally, the first quantization unit is specifically used to divide the forward propagation tensor of the first data format into multiple forward element groups along the output channel dimension, determine a third scaling factor based on the forward element groups and the numerical range of the second data format, and scale the forward element groups using the third scaling factor. The second quantization unit is specifically used to divide the backpropagation tensor of the first data format into multiple inverse element groups along the output channel dimension, determine a fourth scaling factor based on the numerical range of the inverse element groups and the third data format, and scale the inverse element groups using the fourth scaling factor.

[0016] Optionally, the first processing module is specifically used to process the forward propagation tensor of the first data format in the first network layer; and to quantize the forward propagation tensor of the first data format into the forward propagation tensor of the second data format in the second network layer. The second processing module is specifically used to process the backpropagation tensor of the first data format in the first network layer; and to quantize the backpropagation tensor of the first data format into a backpropagation tensor of the third data format in the second network layer. The first network layer is more sensitive to quantization than the second network layer.

[0017] Optionally, the first network layer includes at least one of an input layer and an output layer.

[0018] Optionally, the second data format is the E4M3 data format of 8-bit floating-point numbers, and the third data format is the E5M2 data format of 8-bit floating-point numbers.

[0019] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the model training method as described in any of the preceding claims.

[0020] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the model training method as described in any of the preceding claims.

[0021] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the model training method as described in any of the preceding claims.

[0022] In this embodiment, on the one hand, during the training of the world model to be trained, high-precision tensors are quantized into low-precision tensors for processing. Low-precision computation can reduce memory usage and improve training efficiency. On the other hand, different low-precision data formats are used for the forward propagation and backpropagation processes. The data in the forward propagation process requires higher numerical precision, so the forward propagation uses a second data format with higher numerical precision, which can better preserve detailed information and reduce reconstruction errors. The data in the backpropagation process is not sensitive to loss of numerical precision, but requires a higher dynamic range. Therefore, the backpropagation uses a third data format with a larger numerical range, which can accommodate outliers, prevent gradient vanishing, and ensure data stability. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some drawings of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a model training method according to an embodiment of this application; Figure 2 This is a schematic diagram of a different data format in an embodiment of this application; Figure 3 This is a structural block diagram of a model training device according to an embodiment of this application; Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application; Figure 5 This is a structural block diagram of a readable storage medium according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In this embodiment, addressing the issues of the massive parameter scale and complex computational graph of world models, which require enormous computational resources and lengthy training times for training, a low-numerical-precision training technique is proposed for training the world model. However, low-numerical-precision training techniques typically focus on LLMs (Large Language Models) and lack exploration for world models. Since world models are based on diffusion model design, their computational characteristics differ significantly from LLMs, leading to several drawbacks when directly applying low-numerical-precision training techniques for LLMs to world models. For example, the dynamic range of numerical distribution in world models is larger than that of LLMs; directly applying low-numerical-precision training techniques for LLMs can result in overflow or underflow issues, ultimately manifesting as model non-convergence or a severe degradation in generation quality. Furthermore, the different requirements for forward and backward propagation in world models mean that low-numerical-precision training techniques exhibit specificity across different models and datasets, making direct transfer of LLM-based low-numerical-precision training techniques to world models ungeneralizable.

[0027] To address the aforementioned issues, this application proposes a low-numerical-precision model training method for world models. By constructing a systematic low-numerical-precision training framework, the method quickly finds the optimal quantization configuration for the world model on different tasks and datasets, achieving the best balance between performance and efficiency. The details are described below.

[0028] Example 1: Reference Figure 1 The diagram shows a flowchart of a model training method according to an embodiment of this application.

[0029] like Figure 1 As shown, the model training method may include the following steps: Step 101: During the forward propagation process of the world model to be trained, the forward propagation tensor in the first data format is quantized into the forward propagation tensor in the second data format for processing.

[0030] In this embodiment, the dataset required for model training is prepared in advance. For example, sample data (such as image data, video data, etc.) is acquired, and the sample data is preprocessed (such as normalization, outlier handling, data augmentation, etc.) to adapt it to the input structure of the world model. The preprocessed sample data constitutes the dataset. Of course, the dataset can also be divided into a training set, a validation set, and a test set, at least one of these.

[0031] In this embodiment, a world model architecture is pre-built. The specific world model architecture can be set according to actual needs, and this embodiment does not impose any restrictions on it. For example, the network layers in the world model may include, but are not limited to, input layers, convolutional layers, attention layers, modulation layers, output layers, and so on.

[0032] In this embodiment of the application, the training process of the world model to be trained includes a forward propagation process and a backward propagation process.

[0033] In this embodiment, during the forward propagation process of the world model to be trained, each network layer has its own forward propagation tensor. The forward propagation tensor refers to the tensor that participates in the computation within the network layer during the forward propagation process. During the forward propagation process of the world model to be trained, the forward propagation tensor of the network layer in its first data format is quantized into a forward propagation tensor in its second data format for processing within the network layer of the world model to be trained.

[0034] For example, during the forward propagation process of the training process of the world model to be trained, the forward propagation tensor of the network layer of the world model to be trained is quantized into a forward propagation tensor of a second data format in the network layer of the network layer. The forward propagation tensor of the second data format is processed by the forward propagation calculation (such as convolution calculation, attention calculation, etc.) corresponding to the network layer to obtain the forward output tensor of the network layer in the second data format. The forward output tensor of the second data format is dequantized into a forward output tensor of the first data format, and then the forward output tensor of the first data format is output.

[0035] For example, during the forward propagation process of training the world model to be trained, sample data from the training set is input into the world model. This sample data is in a first data format, serving as the input activation tensor of the first network layer (e.g., the input layer) in the forward propagation of the world model. The input activation tensors of subsequent network layers in the forward propagation are the forward output tensors of the previous network layer in the first data format. Additionally, each network layer has its own weight tensor. The forward propagation tensor of any network layer can include both the input activation tensor and the weight tensor of that layer.

[0036] Step 102: During the backpropagation process of the training process of the world model to be trained, the backpropagation tensor of the first data format is quantized into a backpropagation tensor of the third data format for processing; wherein, the numerical precision of the first data format is higher than that of the second data format, the numerical precision of the second data format is higher than that of the third data format, and the numerical range of the third data format is greater than that of the second data format.

[0037] In this embodiment, during the backpropagation process of the world model to be trained, each network layer has its own backpropagation tensor. The backpropagation tensor refers to the tensor that participates in the computation within the network layer during the backpropagation process. During the backpropagation process of the world model to be trained, the backpropagation tensor of the network layer in its first data format is quantized into a backpropagation tensor in its third data format for processing within the network layer of the world model to be trained.

[0038] For example, during the backpropagation process of the training process of the world model to be trained, the backpropagation tensor of the first data format of the network layer of the world model to be trained is quantized into the backpropagation tensor of the third data format. The backpropagation tensor of the third data format is subjected to the backpropagation calculation processing corresponding to the network layer to obtain the backpropagation output tensor of the third data format of the network layer. The backpropagation output tensor of the third data format is dequantized into the backpropagation output tensor of the first data format, and then the backpropagation output tensor of the first data format is output.

[0039] For example, during the backpropagation process of the world model to be trained, the gradient tensor of the first data format is calculated based on the prediction result of the output layer (the forward output tensor of the first data format output by the output layer) and the true label. This gradient tensor of the first data format is used as the input gradient tensor of the first data format of the first network layer in the backpropagation of the world model (such as the network layer before the output layer). The input gradient tensors of the first data format of each subsequent network layer in the backpropagation are the backward output tensors of the first data format output by the network layer before that network layer. The backpropagation tensor of any network layer may include the input gradient tensor of that network layer.

[0040] In this embodiment, numerical precision refers to the fineness of a data format when representing a numerical value. It determines the magnitude of rounding error; the higher the precision, the smaller the rounding error. Numerical precision is determined by the number of mantissas. The more mantissas, the richer the decimal details that can be represented, and the higher the numerical precision.

[0041] In this embodiment, the numerical range refers to the span between the maximum and minimum values ​​that the data format can represent. The numerical range is determined by the exponent; the more exponents, the wider the numerical range that can be covered.

[0042] In this embodiment, on the one hand, during the training of the world model to be trained, high-precision tensors are quantized into low-precision tensors for processing. Low-precision computation can reduce memory usage and improve training efficiency. On the other hand, different low-precision data formats are used for the forward propagation and backpropagation processes. The data in the forward propagation process requires higher numerical precision, so the forward propagation uses a second data format with higher numerical precision, which can better preserve detailed information and reduce reconstruction errors. The data in the backpropagation process is not sensitive to loss of numerical precision, but requires a higher dynamic range. Therefore, the backpropagation uses a third data format with a larger numerical range, which can accommodate outliers, prevent gradient vanishing, and ensure data stability.

[0043] Example 2: In this embodiment of the application, considering that the world model contains a large number of heterogeneous attention operations and modulation operations, which are prone to introducing quantization errors and causing computational instability, a scaling strategy is proposed to scale the tensors.

[0044] For example, scaling strategies can include delayed scaling, current scaling, etc.

[0045] The delayed scaling strategy calculates a scaling factor for each tensor based on the maximum absolute value (amax) observed in previous iteration windows, rather than relying solely on the current batch of data. This historical data-based approach smooths outliers and reduces the risk of training instability caused by scaling abrupt changes. Instead of determining the scaling factor based on the current data batch, the delayed scaling strategy examines the maximum absolute value across multiple recent batches and continuously records historical data for these maximum absolute values. When selecting a new scaling factor, the system reviews historical records and selects values ​​that reflect recent trends, rather than relying solely on the latest batch. This approach helps smooth sudden fluctuations in data values, making the training process more stable and reducing the risk of abrupt changes harming model performance. The effectiveness of the delayed scaling strategy relies on a key assumption: the statistical distribution of tensor values ​​remains relatively stable throughout training iterations. However, outliers can dominate the history of maximum absolute values, leading to suboptimal scaling that causes underflow or overflow, which can disrupt training stability, especially at large scales.

[0046] The instant scaling strategy provides a solution that prioritizes real-time adaptability by determining a scaling factor for each tensor based on its statistical properties during the current forward or backward propagation. The instant scaling strategy dynamically adjusts the scaling ratio according to the current data range. This instantaneous responsiveness helps optimize data representation and improves model convergence during training. This strategy operates in real-time and does not rely on historical batch data.

[0047] In one optional implementation, the process of quantizing the forward propagation tensor of the first data format into a forward propagation tensor of the second data format for processing may include: determining a first scaling factor based on the numerical range of the forward propagation tensor of the first data format and the second data format; scaling the forward propagation tensor of the first data format using the first scaling factor; and converting the scaled forward propagation tensor into a forward propagation tensor of the second data format for processing.

[0048] In one optional implementation, the process of quantizing the backpropagation tensor of the first data format into a backpropagation tensor of the third data format for processing may include: determining a second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format, scaling the backpropagation tensor of the first data format using the second scaling factor, and converting the scaled backpropagation tensor into a backpropagation tensor of the second data format for processing.

[0049] In this embodiment of the application, by employing an instant scaling strategy to calculate the scaling factor, the broad dynamic range of high numerical precision data can be mapped to the limited representation range of low numerical precision format, thereby effectively controlling the dynamic range of data during training, improving numerical stability, effectively handling sparsely distributed but numerically large gradient outliers that occur during world model training, and preventing quantization overflow.

[0050] Example 3: In one optional implementation, the process of determining the first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format may include: taking the maximum absolute value of the elements in the forward propagation tensor of the first data format as the first target value, and determining the ratio of the upper limit of the numerical range of the second data format to the first target value as the first scaling factor.

[0051] In one optional implementation, the process of determining the second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format may include: taking the maximum absolute value of the elements in the backpropagation tensor of the first data format as the second target value, and determining the ratio of the upper limit of the numerical range of the third data format to the second target value as the second scaling factor.

[0052] In this embodiment, by using the ratio of the upper limit of the numerical range to the maximum absolute value as a scaling factor, linear uniform quantization can be achieved. The numerical range of the original high numerical precision tensor can be mapped losslessly and linearly to the numerical range of the low numerical precision format, which can improve the utilization of dynamic range and make the calculation process simple and efficient.

[0053] After calculating the first scaling factor, the forward propagation tensor of the first data format is scaled using the first scaling factor to obtain a scaled forward propagation tensor. For example, the forward propagation tensor of the first data format is multiplied by the first scaling factor to obtain the scaled forward propagation tensor. Each element of the forward propagation tensor of the first data format is multiplied by the first scaling factor.

[0054] After scaling to obtain the scaled forward propagation tensor, the scaled forward propagation tensor is converted into a forward propagation tensor in the second data format. Specifically, the scaled forward propagation tensor is a scaled forward propagation tensor in the first data format. The specific process of converting the scaled forward propagation tensor in the first data format into the forward propagation tensor in the second data format, that is, the specific process of converting the first data format into the second data format, can be handled based on practical experience, and will not be discussed in detail in this embodiment.

[0055] After calculating the second scaling factor, the backpropagation tensor of the first data format is scaled using the second scaling factor to obtain a scaled backpropagation tensor. For example, the backpropagation tensor of the first data format is multiplied by the second scaling factor to obtain the scaled backpropagation tensor. Each element of the backpropagation tensor of the first data format is multiplied by the second scaling factor.

[0056] After scaling to obtain the scaled backpropagation tensor, the scaled backpropagation tensor is converted into the backpropagation tensor of the third data format. Specifically, the scaled backpropagation tensor is the scaled backpropagation tensor of the first data format. The specific process of converting the scaled backpropagation tensor of the first data format into the backpropagation tensor of the third data format, that is, the specific process of converting the first data format into the third data format, can be handled according to practical experience, and will not be discussed in detail in this embodiment.

[0057] Example 4: In this embodiment of the application, considering the accuracy of the scaling process, a scaling granularity strategy is proposed for scaling processing.

[0058] For example, scaling granularity strategies can be per-tensor, per-block, per-group, etc.

[0059] Tensor-wise scaling granularity strategies often face limitations when dealing with block-level variability within a single tensor. This challenge arises when different regions or slices within a tensor exhibit disparate numerical ranges and statistical behaviors. For example, in large Transformer models, different attention heads within a single weight matrix may have different magnitudes. When a single scaling factor is applied to diverse tensors, it either sacrifices numerical accuracy in dense, small numerical regions or causes extreme outliers to be truncated. Both situations reduce training stability and compromise the final model quality.

[0060] The block-by-block scaling granularity strategy divides each tensor into smaller, more manageable contiguous blocks and assigns a dedicated scaling factor to each block. This mechanism dynamically adjusts based on the local statistical characteristics of each block, rather than being dominated by a single extreme value. This means that high-order regions are accurately represented while ensuring that the fidelity of more typical low-order values ​​within the same tensor is not affected.

[0061] Group-by-group scaling is a finer-grained quantization method. Its core lies in further dividing the tensor along a specific dimension into multiple consecutive, smaller groups of elements, and independently calculating and applying a set of quantization parameters to each group. Specifically, each output channel is further divided into multiple consecutive groups of elements. Therefore, the elements within each group share a set of scaling factors tailored to that group. By significantly narrowing the scope of the scaling factors, the dynamic range of the values ​​within each group is highly concentrated. The resulting scaling factors can more closely fit the actual distribution of the data in that group, thereby reducing quantization rounding errors to extremely low levels.

[0062] In one optional implementation, the process of determining a first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format may include: dividing the forward propagation tensor of the first data format into multiple forward element groups along the output channel dimension, determining a third scaling factor based on the forward element groups and the numerical range of the second data format, and scaling the forward element groups using the third scaling factor.

[0063] In one optional implementation, the process of determining the second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format may include: dividing the backpropagation tensor of the first data format into multiple backpropagation element groups along the output channel dimension, determining a fourth scaling factor based on the backpropagation element groups and the numerical range of the third data format, and scaling the backpropagation element groups using the fourth scaling factor.

[0064] In this embodiment, by adopting a group-by-group scaling granularity strategy, a balance is achieved between numerical accuracy and computational overhead, and the local features of data distribution can be better captured, while significantly reducing storage and transmission overhead while maintaining expressive power.

[0065] For example, the process of determining a third scaling factor based on the preceding element group and the numerical range of the second data format, and scaling the preceding element group using the third scaling factor, may include: for each preceding element group, taking the maximum absolute value of the elements in the preceding element group as a third target value, determining the ratio of the upper limit of the numerical range of the second data format to the third target value as the third scaling factor corresponding to the preceding element group, and multiplying the preceding element group by the third scaling factor. Each element in the preceding element group is multiplied by the third scaling factor.

[0066] For example, the process of determining a fourth scaling factor based on the inverted element group and the numerical range of the third data format, and scaling the inverted element group using the fourth scaling factor, may include: for each inverted element group, taking the maximum absolute value of the elements in the inverted element group as a fourth target value, determining the ratio of the upper limit of the numerical range of the third data format to the fourth target value as the fourth scaling factor corresponding to the inverted element group, and multiplying the inverted element group by the fourth scaling factor. Each element in the inverted element group is multiplied by the fourth scaling factor.

[0067] For example, in the process of dequantizing the forward output tensor of the second data format into the forward output tensor of the first data format, the forward output tensor of the second data format is divided by the scaling factor corresponding to the forward output tensor of the second data format (that is, the scaling factor corresponding to the forward propagation tensor of the second data format used when calculating the forward output tensor of the second data format), and the scaled forward output tensor of the second data format is converted into the forward output tensor of the first data format.

[0068] For example, in the process of dequantizing the inverse output tensor of the third data format into the inverse output tensor of the first data format, the inverse output tensor of the third data format is divided by the scaling factor corresponding to the inverse output tensor of the third data format (that is, the scaling factor corresponding to the backpropagation tensor of the third data format used when calculating the inverse output tensor of the third data format), and the scaled inverse output tensor of the third data format is converted into the inverse output tensor of the first data format.

[0069] Example 5: In one optional implementation, quantizing the forward propagation tensor of the first data format into a forward propagation tensor of the second data format includes: in a first network layer, processing the forward propagation tensor of the first data format while retaining it (that is, performing the forward propagation calculation corresponding to the network layer on the forward propagation tensor of the first data format without quantizing the forward propagation tensor of the first data format); and in a second network layer, quantizing the forward propagation tensor of the first data format into a forward propagation tensor of the second data format for processing.

[0070] In one optional implementation, quantizing the backpropagation tensor of the first data format into a backpropagation tensor of the third data format includes: in the first network layer, retaining the backpropagation tensor of the first data format for processing (that is, performing backpropagation calculation processing corresponding to the network layer on the backpropagation tensor of the first data format without quantizing the backpropagation tensor of the first data format); and in the second network layer, quantizing the backpropagation tensor of the first data format into a backpropagation tensor of the third data format for processing.

[0071] The first network layer is more sensitive to quantization than the second network layer.

[0072] In this embodiment, considering that the first network layer is highly sensitive to quantization, it retains high numerical accuracy to avoid negative impact on the model's expressive ability due to the accumulation of quantization errors, thereby further improving the model quality.

[0073] Example 6: In one alternative implementation, the first network layer includes at least one of an input layer and an output layer. In this approach, considering that the input and output layers are highly sensitive to quantization, they can be used as the first network layer.

[0074] Example 7: In one alternative implementation, the second data format is an 8-bit floating-point number (FP8) E4M3 data format, and the third data format is an 8-bit floating-point number E5M2 data format.

[0075] The E4M3 data format consists of 1 sign bit, 4 exponent bits, and 3 mantissa bits, and can store values ​​ranging from ±448 to NaN (Not-a-Number). The E5M2 data format contains 1 sign bit, 5 exponent bits, and 2 mantissa bits, and can store values ​​ranging from ±57344 to ± infinity and nanometers.

[0076] In this approach, the forward propagation stage uses the E4M3 data type because its higher numerical precision can better preserve image detail information, reduce reconstruction errors, and balance numerical precision and efficiency; the backward propagation stage uses the E5M2 data format, which utilizes its larger dynamic range to accommodate outliers, prevent gradient vanishing, and ensure the dynamic range and expressive power in gradient calculation.

[0077] For example, the first data format may be a Brain Floating Point 16-bit (BF16) data format.

[0078] Figure 2 This is a schematic diagram illustrating a different data format in an embodiment of this application. For example... Figure 2 As shown, BF16 is a 16-bit floating-point format, consisting of 1 sign bit, 8 exponent bits, and 7 mantissa bits. FP8 E4M3 is an 8-bit floating-point format, consisting of 1 sign bit, 4 exponent bits, and 3 mantissa bits. FP8 E5M2 is an 8-bit floating-point format, consisting of 1 sign bit, 5 exponent bits, and 2 mantissa bits.

[0079] In this embodiment, a multi-dimensional model vector framework is constructed. Mechanisms such as instantaneous scaling, group-by-group scaling granularity, different data types for forward and backward propagation, and preservation of high numerical precision for quantization-sensitive network layers are employed on the world model. A configuration combination for the world model is proposed and solidified into a reusable training recipe. Based on this recipe, a significant acceleration of the end-to-end training process is achieved on the world model structure. Actual tests show that compared to the BF16 mixed numerical precision scheme, it significantly shortens the model iteration cycle and reduces computational resource requirements.

[0080] In this embodiment of the application, the test set can also be used to calculate the FID (Fréchet Inception Distance) and PSNR (Peak Signal-to-Noise Ratio) of the world model, thereby evaluating the quality of the world model so as to improve training efficiency while ensuring the quality of the generated content.

[0081] It should be noted that the model training methods involved in the above embodiments may include at least one of Embodiments 1 to 7. For example, Embodiment 1 can be implemented as an independent embodiment, Embodiment 1+2 can be implemented as an independent embodiment, Embodiment 1+2+3 can be implemented as an independent embodiment, Embodiment 1+2+4 can be implemented as an independent embodiment, Embodiment 1+5 can be implemented as an independent embodiment, Embodiment 1+2+5 can be implemented as an independent embodiment, Embodiment 1+2+3+5+5 can be implemented as an independent embodiment, Embodiment 1+2+4+5 can be implemented as an independent embodiment, Embodiment 1+5+6 can be implemented as an independent embodiment, Embodiment 1+2+5+6 can be implemented as an independent embodiment, Embodiment 1+2+3+5+6 can be implemented as an independent embodiment, Embodiment 1+2+4+5+6 can be implemented as an independent embodiment, and Embodiment 1+7 can be implemented as an independent embodiment. Examples 1+2+7 can be implemented as independent embodiments, examples 1+2+3+7 can be implemented as independent embodiments, examples 1+2+4+7 can be implemented as independent embodiments, examples 1+5+7 can be implemented as independent embodiments, examples 1+2+5+7 can be implemented as independent embodiments, examples 1+2+3+5+7 can be implemented as independent embodiments, examples 1+2+4+5+7 can be implemented as independent embodiments, examples 1+5+6+7 can be implemented as independent embodiments, examples 1+2+5+6+7 can be implemented as independent embodiments, examples 1+2+3+5+6+7 can be implemented as independent embodiments, examples 1+2+4+5+6+7 can be implemented as independent embodiments, and so on, but are not limited to these, and will not be listed one by one here.

[0082] It should be noted that the model training method provided in this application embodiment can be executed by a model training device or a control module within that model training device for executing the loading model training method. This application embodiment uses the execution of the loading model training method by a model training device as an example to illustrate the model training method provided in this application embodiment.

[0083] Reference Figure 3 The diagram shows a structural block diagram of a model training device according to an embodiment of this application.

[0084] like Figure 3 As shown, the model training device may include the following modules: The first processing module 301 is used to quantize the forward propagation tensor of the first data format into the forward propagation tensor of the second data format for processing during the forward propagation process of the training process of the world model to be trained. The second processing module 302 is used to quantize the backpropagation tensor of the first data format into the backpropagation tensor of the third data format for processing during the backpropagation process of the training process of the world model to be trained. The numerical precision of the first data format is higher than that of the second data format, the numerical precision of the second data format is higher than that of the third data format, and the numerical range of the third data format is greater than that of the second data format.

[0085] Optionally, the first processing module 301 includes: a first quantization unit, configured to determine a first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format, scale the forward propagation tensor of the first data format using the first scaling factor, and convert the scaled forward propagation tensor into a forward propagation tensor of the second data format for processing; The second processing module 302 includes: a second quantization unit, configured to determine a second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format, scale the backpropagation tensor of the first data format using the second scaling factor, and convert the scaled backpropagation tensor into a backpropagation tensor of the second data format for processing.

[0086] Optionally, the first quantization unit is specifically used to take the maximum absolute value of the elements in the forward propagation tensor of the first data format as the first target value, and to determine the ratio of the upper limit of the numerical range of the second data format to the first target value as the first scaling factor. The second quantization unit is specifically used to take the maximum absolute value of the elements in the backpropagation tensor of the first data format as the second target value, and to determine the ratio of the upper limit of the numerical range of the third data format to the second target value as the second scaling factor.

[0087] Optionally, the first quantization unit is specifically used to divide the forward propagation tensor of the first data format into multiple forward element groups along the output channel dimension, determine a third scaling factor based on the forward element groups and the numerical range of the second data format, and scale the forward element groups using the third scaling factor. The second quantization unit is specifically used to divide the backpropagation tensor of the first data format into multiple inverse element groups along the output channel dimension, determine a fourth scaling factor based on the numerical range of the inverse element groups and the third data format, and scale the inverse element groups using the fourth scaling factor.

[0088] Optionally, the first processing module 301 is specifically used to process the forward propagation tensor of the first data format in the first network layer; and to quantize the forward propagation tensor of the first data format into the forward propagation tensor of the second data format in the second network layer. The second processing module 302 is specifically used to process the backpropagation tensor of the first data format in the first network layer; and to quantize the backpropagation tensor of the first data format into a backpropagation tensor of the third data format in the second network layer. The first network layer is more sensitive to quantization than the second network layer.

[0089] Optionally, the first network layer includes at least one of an input layer and an output layer.

[0090] Optionally, the second data format is the E4M3 data format of 8-bit floating-point numbers, and the third data format is the E5M2 data format of 8-bit floating-point numbers.

[0091] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0092] The model training device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0093] The model training device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0094] The model training apparatus provided in this application embodiment can implement the various processes implemented by the model training apparatus in any of the above-mentioned method embodiments. To avoid repetition, these processes will not be described again here.

[0095] Optionally, embodiments of this application also provide an electronic device.

[0096] Reference Figure 4 The diagram shows a structural block diagram of an electronic device according to an embodiment of this application.

[0097] like Figure 4 As shown, the electronic device 11 includes a processor 111, a memory 112, and a program or instruction 1121 stored in the memory and executable on the processor. When the program or instruction 1121 is executed by the processor 111, it implements the various processes of any of the above-described model training method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0098] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0099] Optionally, embodiments of this application also provide a readable storage medium.

[0100] Reference Figure 5 The diagram shows a structural block diagram of a readable storage medium according to an embodiment of this application.

[0101] like Figure 5 As shown, a program or instruction 211 is stored on the readable storage medium 21. When the program or instruction 211 is executed by the processor, it implements the various processes of any of the above model training method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0102] The processor may be the processor in the electronic device described in the above embodiments.

[0103] Optionally, this application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model training method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0104] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0105] The processors mentioned above may include, but are not limited to: Central Processing Unit (CPU), Network Processor (NP), Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0106] The aforementioned readable storage media include computer-readable storage media, which may include, but are not limited to: read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), electronically erasable programmable read-only memory (EEPROM), hard disk, floppy disk, flash memory, etc.

[0107] The various embodiments in this specification are related to each other and are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0108] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0109] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0110] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0112] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms. The functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0113] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0114] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method includes: During the forward propagation process of the world model to be trained, the forward propagation tensor in the first data format is quantized into the forward propagation tensor in the second data format for processing. During the backpropagation process of the training process of the world model to be trained, the backpropagation tensor in the first data format is quantized into the backpropagation tensor in the third data format for processing. The numerical precision of the first data format is higher than that of the second data format, the numerical precision of the second data format is higher than that of the third data format, and the numerical range of the third data format is greater than that of the second data format.

2. The method according to claim 1, characterized in that, The process of quantizing the forward propagation tensor in the first data format into a forward propagation tensor in the second data format includes: A first scaling factor is determined based on the forward propagation tensor of the first data format and the numerical range of the second data format. The forward propagation tensor of the first data format is scaled using the first scaling factor, and the scaled forward propagation tensor is converted into a forward propagation tensor of the second data format for processing. The process of quantizing the backpropagation tensor of the first data format into a backpropagation tensor of the third data format includes: A second scaling factor is determined based on the backpropagation tensor of the first data format and the numerical range of the third data format. The backpropagation tensor of the first data format is scaled using the second scaling factor, and the scaled backpropagation tensor is converted into a backpropagation tensor of the second data format for processing.

3. The method according to claim 2, characterized in that, Determining the first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format includes: The maximum absolute value of the elements in the forward propagation tensor of the first data format is taken as the first target value, and the ratio of the upper limit of the numerical range of the second data format to the first target value is determined as the first scaling factor. Determining the second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format includes: The maximum absolute value of the elements in the backpropagation tensor of the first data format is taken as the second target value, and the ratio of the upper limit of the numerical range of the third data format to the second target value is determined as the second scaling factor.

4. The method according to claim 2, characterized in that, Determining the first scaling factor based on the forward propagation tensor of the first data format and the numerical range of the second data format includes: The forward propagation tensor of the first data format is divided into multiple forward element groups along the output channel dimension. A third scaling factor is determined based on the numerical range of the forward element groups and the second data format. The forward element groups are then scaled using the third scaling factor. Determining the second scaling factor based on the backpropagation tensor of the first data format and the numerical range of the third data format includes: The backpropagation tensor of the first data format is divided into multiple inverse element groups along the output channel dimension. A fourth scaling factor is determined based on the numerical range of the inverse element groups and the third data format. The inverse element groups are then scaled using the fourth scaling factor.

5. The method according to any one of claims 1 to 4, characterized in that, The process of quantizing the forward propagation tensor in the first data format into a forward propagation tensor in the second data format includes: In the first network layer, the forward propagation tensor of the first data format is retained for processing; In the second network layer, the forward propagation tensor of the first data format is quantized into the forward propagation tensor of the second data format for processing; The process of quantizing the backpropagation tensor of the first data format into a backpropagation tensor of the third data format includes: In the first network layer, the backpropagation tensor of the first data format is retained for processing; In the second network layer, the backpropagation tensor of the first data format is quantized into the backpropagation tensor of the third data format for processing; The first network layer is more sensitive to quantization than the second network layer.

6. The method according to claim 5, characterized in that, The first network layer includes at least one of an input layer and an output layer.

7. The method according to any one of claims 1 to 6, characterized in that, The second data format is the E4M3 data format of 8-bit floating-point numbers, and the third data format is the E5M2 data format of 8-bit floating-point numbers.

8. A model training device, characterized in that, The device includes: The first processing module is used to quantize the forward propagation tensor of the first data format into the forward propagation tensor of the second data format for processing during the forward propagation process of the training process of the world model to be trained. The second processing module is used to quantize the backpropagation tensor of the first data format into the backpropagation tensor of the third data format for processing during the backpropagation process of the training process of the world model to be trained. The numerical precision of the first data format is higher than that of the second data format, the numerical precision of the second data format is higher than that of the third data format, and the numerical range of the third data format is greater than that of the second data format.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the model training method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the model training method as described in any one of claims 1 to 7.

Images