Model training method and device, electronic device, storage medium, and program product

Abstract

The present disclosure provides a model training method and device, an electronic device, a storage medium and a program product. The method comprises: during quantization training of a model, obtaining a first weight tensor and a second weight tensor of a Gated Linear Unit of the model after completion of current training step training; determining a target quantization granularity strategy of the model in a subsequent training step from a plurality of quantization granularity strategies according to the first weight tensor and the second weight tensor; and continuing to quantization train the model in the subsequent training step according to the target quantization granularity strategy to obtain a trained model. According to embodiments of the present disclosure, a balance between precision retention, operation overhead and storage cost of quantization training can be achieved, and model training efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method and apparatus, electronic equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] As the parameter size and training data size of Large Language Models (LLMs) continue to grow, the need to control training efficiency and deployment costs becomes increasingly prominent. Quantization training, as a core method to reduce computational overhead and storage usage, is widely used in the model training process.

[0003] Typically, during quantization training of a model, the quantization granularity strategy used is fixed, meaning a fixed quantization granularity strategy is preset before model training and remains unchanged throughout the training cycle. This model training method may increase computational overhead and storage burden, affecting model training efficiency. Summary of the Invention

[0004] This disclosure provides a model training method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] Firstly, this disclosure provides a model training method, which includes:

[0006] During the quantization training of the model, the first weight tensor and the second weight tensor of the gated linear unit of the model are obtained after training in the current training step; wherein, the quantization training includes, in each training step, performing numerical quantization processing on the first weight tensor and the second weight tensor of the gated linear unit based on the quantization granularity strategy of the training step, and training the model based on the quantized first weight tensor and the second weight tensor.

[0007] Based on the first weight tensor and the second weight tensor, the target quantization granularity strategy of the model in subsequent training steps is determined from multiple quantization granularity strategies; wherein the quantization granularity of the multiple quantization granularity strategies is different.

[0008] Based on the target quantization granularity strategy, the model is further quantized and trained in subsequent training steps to obtain the trained model.

[0009] Secondly, this disclosure provides a model training apparatus, which includes:

[0010] The acquisition module is used to acquire the first weight tensor and the second weight tensor of the gated linear unit of the model after training in the current training step during the quantization training of the model; wherein, the quantization training includes, in each training step, performing numerical quantization processing on the first weight tensor and the second weight tensor of the gated linear unit based on the quantization granularity strategy of the training step, and performing model training based on the quantized first weight tensor and the second weight tensor.

[0011] The determination module is configured to determine the target quantization granularity strategy of the model in subsequent training steps from multiple quantization granularity strategies based on the first weight tensor and the second weight tensor; wherein the multiple quantization granularity strategies have different quantization granularities.

[0012] The training module is used to continue quantizing the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model.

[0013] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method described above.

[0014] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described model training method.

[0015] Fifthly, this disclosure provides a computer program product that includes computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the model training method described above.

[0016] The embodiments provided in this disclosure, during the quantization training of the model, obtain the first weight tensor and the second weight tensor of the gated linear unit of the model after the training is completed in the current training step; based on the two obtained weight tensors, determine the target quantization granularity strategy for the subsequent training step from multiple quantization granularity strategies with different quantization granularities, and continue to quantize the model according to the target quantization granularity strategy to obtain the trained model.

[0017] Since the two weight tensors of the gated linear unit of the model are continuously updated with each training iteration during model training, this disclosure obtains the weight tensor after the current training step is completed during training. Based on the dynamic changes of these two weight tensors, the target quantization granularity strategy for subsequent training steps can be dynamically adjusted. This ensures that the target quantization granularity used matches the real-time state of the weight tensor of the gated linear unit, avoiding the mismatch between the fixed quantization granularity strategy used in related technologies and the dynamically changing weight tensors. This achieves a balance between preserving the accuracy of quantization training and reducing computational and storage costs, thereby improving model training efficiency.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 This is a flowchart of a model training method provided in an embodiment of the present disclosure.

[0021] Figure 2 A first flowchart of a strategy for determining the quantization granularity of a target provided in an embodiment of this disclosure.

[0022] Figure 3 A flowchart for determining a first amplitude index provided in an embodiment of this disclosure.

[0023] Figure 4 A second flowchart illustrating the strategy for determining the quantization granularity of the target provided in this embodiment of the disclosure.

[0024] Figure 5 A first flowchart of quantitative training provided for embodiments of this disclosure.

[0025] Figure 6 A second flowchart for quantitative training provided in an embodiment of this disclosure.

[0026] Figure 7 This is a block diagram of a model training apparatus provided in an embodiment of the present disclosure.

[0027] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0030] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0032] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0033] As mentioned above, with the continuous growth of the parameter scale and training data scale of large language models, quantization training has become a core means to reduce the computational overhead and storage consumption of the model. However, existing quantization training often adopts a fixed quantization granularity strategy, that is, a fixed quantization granularity strategy is preset before training and remains unchanged throughout the training cycle. However, since the values ​​of weight tensors or activation tensors will change dynamically with the model training iteration during the model training process, this method may increase the computational overhead and storage burden, affecting the efficiency of model training.

[0034] Taking FP8 training in quantization training as an example, FP8 is a low-precision numerical format. By further reducing the numerical precision, it can significantly improve the computational throughput while maintaining a certain degree of precision loss during model training. FP8 quantization training has been widely used in large model training architectures such as Transformer.

[0035] In FP8 quantization training, the choice of quantization granularity strategy directly affects training efficiency and accuracy preservation. Existing quantization granularity strategies generally include per-tensor quantization and per-block quantization.

[0036] Tensor quantization strategy refers to using a uniform quantization scale for the entire weight tensor or activation tensor. This quantization granularity strategy can improve the computational efficiency during model training. However, it is less adaptable to tensors with uneven numerical distribution, which can easily lead to quantization overflow, resulting in decreased training accuracy or even training failure.

[0037] Block quantization strategy refers to dividing the tensor into several sub-blocks and independently calculating the quantization granularity of each sub-block during model training. This quantization granularity strategy can avoid quantization overflow problems, but it has the problems of large computational and storage overhead.

[0038] Currently, existing frameworks, such as Megatron-LM and Transformer-Engine, typically employ a fixed quantization granularity strategy during training. This means that a fixed quantization granularity is preset before training begins and is not dynamically adjusted throughout the entire training iteration process.

[0039] However, the applicant found that existing large language models typically include a Transformer feedforward network, and the Gated Linear Unit (GLU), as the core of the Transformer feedforward network, usually consists of two weight tensors W1 and W2 that are continuously updated with model training iterations, and the amplification of its output amplitude changes constantly. Taking the SwishGLU gated linear unit as an example, its basic form can be expressed as follows: ,in, , Let the weight tensor be the two linear transformations. This represents element-wise multiplication. Let y represent the activation function, such as the sigmoid activation function, x represent the input data, and y represent the output data of the SwishGLU gated linear unit. During model quantization training, such as FP8 quantization training, due to the multiplicative structure of SwishGLU, the magnitude of its output data y will gradually increase as the training progresses.

[0040] Therefore, when using a fixed quantization granularity strategy, if a per-tensor quantization strategy is used, quantization overflow may occur at any time during quantization training, leading to model training failure and affecting model training efficiency. On the other hand, if a per-block quantization strategy is used, although model training failure can be avoided, unnecessary computational and storage overhead will be continuously increased, affecting model training efficiency.

[0041] In implementing this disclosure, to address the aforementioned issues, the applicant discovered that modifying the model structure or operators could be considered, for example, by using scaling before and after weight operations to adapt to dynamic changes in weights, i.e., pre-setting scaling factors to compensate for weights or outputs. However, this method may have the following problems: On the one hand, the selection of scaling factors is usually empirical and lagging, making it difficult to accurately match the real-time fluctuations of weights during training. Excessive scaling factors can easily lead to the loss of feature information, while insufficient scaling factors cannot solve the overflow problem, potentially still causing model training failure. On the other hand, modifying the model structure or introducing dedicated scaling operators can compromise the model's versatility, thereby increasing deployment difficulty. Furthermore, additional scaling operations themselves typically introduce new computational overhead, affecting model training efficiency.

[0042] In view of this, the present disclosure provides a model training method that does not require modification of the model structure or the introduction of complex weight scaling logic. During the quantization training of the model, the first weight tensor and the second weight tensor of the gated linear unit of the model after the current training step are obtained; based on the obtained two weight tensors, the target quantization granularity strategy for the subsequent training step is determined from multiple quantization granularity strategies with different quantization granularities, and the model is continued to be quantized and trained according to the target quantization granularity strategy to obtain the trained model.

[0043] Based on the method provided in this disclosure, since the two weight tensors of the gated linear unit of the model are continuously updated with training iterations during model training, this disclosure obtains the weight tensor after the current training step is completed during training. It can dynamically adjust the target quantization granularity strategy for subsequent training steps based on the dynamic changes of the two weight tensors. This allows the target quantization granularity to match the real-time state of the weight tensor of the gated linear unit, avoiding the mismatch between the fixed quantization granularity strategy and the dynamically changing weight tensors that exists in related technologies when using a fixed quantization granularity strategy for quantization training. This achieves a balance between preserving the accuracy of quantization training and computational and storage costs, thereby improving model training efficiency.

[0044] The model training method according to embodiments of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0045] Figure 1 A flowchart illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method may include the following steps S11-S13, which are described in detail below.

[0046] Step S11: During the quantization training of the model, obtain the first weight tensor and the second weight tensor of the gated linear unit of the model after the training is completed in the current training step.

[0047] The quantization training includes, in each training step, performing numerical quantization on the first and second weight tensors of the gated linear unit based on the quantization granularity strategy of the training step, and training the model based on the quantized first and second weight tensors.

[0048] Specifically, in this embodiment of the disclosure, model training based on the quantized first and second weight tensors can involve performing training processes such as forward propagation, backward propagation, and parameter updates based on the quantized first and second weight tensors. By quantizing and training the model based on the quantized first and second weight tensors, numerical precision can be reduced, model computational efficiency can be improved, and memory usage can be reduced, thereby improving model training efficiency.

[0049] This quantization training, for example, involves quantizing tensors in the model that were originally of high-precision data types such as FP16, FP32, or BF16 to low-precision data types such as FP8, in order to improve the training speed of the model.

[0050] In this embodiment, the model can be any neural network model including gated linear units (GLUs). This model can be used to perform any of the following tasks: image processing, speech processing, text processing, and video processing.

[0051] As described above, a gated linear unit is typically a computational unit in the Transformer feedforward network that uses two weights—a first weight tensor and a second weight tensor—to implement a gating mechanism. During model training, the values ​​of these first and second weight tensors usually change dynamically as the training iterations progress. This gated linear unit can be, for example, any of SwishGLU (Swish-Gated Linear Unit), GEGLU (GELU-Gated Linear Unit), and ReGLU, etc. This disclosure does not impose any specific limitation on this; in the following description, this disclosure uses SwishGLU as an example to illustrate the gated linear unit.

[0052] Step S12: Based on the first weight tensor and the second weight tensor, determine the target quantization granularity strategy for the model in subsequent training steps from multiple quantization granularity strategies.

[0053] The multiple quantization granularity strategies can be preset strategies with different quantization granularities. These multiple quantization granularity strategies may include, for example, the tensor quantization strategy and block quantization strategy mentioned above, and may also include other quantization strategies of any type as needed; this disclosure does not impose any special limitations on this.

[0054] Step S13: Based on the target quantization granularity strategy, continue to quantize and train the model in subsequent training steps to obtain the trained model.

[0055] In some embodiments, step S11 may include: during the quantization training of the model, if the current training step meets the preset training step period, obtaining the first weight tensor and the second weight tensor of the gated linear unit of the model after the training is completed in the current training step.

[0056] The preset training step period can be set as needed, for example, it can be 100 steps. That is, during model training, considering that the values ​​of the first weight tensor and the second weight tensor usually do not change significantly within a certain training step, in order to reduce computational complexity, for example, the latest first weight tensor and the second weight tensor can be obtained every 100 training steps to determine the target quantization granularity strategy in subsequent training steps. It should be noted that the preset training step period can be set as needed, and this disclosure does not impose any special limitations on it.

[0057] In other words, in this embodiment, it is not necessary to fix the quantization granularity strategy used for model quantization training before model training, nor is it necessary to change the original computational structure of the model's gated linear unit, introduce additional operators, or modify the inference computation path. Instead, during model training, the target quantization granularity strategy for subsequent training steps can be adaptively determined based on the first and second weight tensors after the current training step is completed. This ensures that the quantization granularity of each training step can match the real-time state of the values ​​of the first and second weight tensors, thereby reducing computational overhead and storage costs while ensuring quantization training accuracy and improving model training efficiency. The trained model can then be used directly in practical application scenarios.

[0058] As can be seen, based on the method provided in this disclosure, since the two weight tensors of the gated linear unit of the model are continuously updated with training iterations during model training, this disclosure obtains the weight tensor after the current training step is completed during training. It can dynamically adjust the target quantization granularity strategy for subsequent training steps based on the dynamic changes of the two weight tensors, thereby enabling the target quantization granularity used to match the real-time state of the weight tensor of the gated linear unit. This avoids the situation in related technologies where a fixed quantization granularity strategy is used for quantization training and does not match the two dynamically changing weight tensors. It can achieve a balance between the preservation of accuracy in quantization training and computational and storage costs, thereby improving model training efficiency.

[0059] The model training method according to the embodiments of this disclosure will now be described in detail.

[0060] Please refer to Figure 2 This is a first flowchart of the strategy for determining the quantization granularity of the target provided in the embodiments of this disclosure. Figure 2 As shown, in some embodiments, the step S11, which involves determining the target quantization granularity strategy of the model in subsequent training steps from multiple quantization granularity strategies based on the first weight tensor and the second weight tensor, may include the following steps S21-S22.

[0061] Step S21: Determine the first magnitude index based on the first weight tensor and the second weight tensor.

[0062] The first amplitude index is used to represent the degree to which the gating linear unit amplifies the amplitude of the output data.

[0063] Specifically, the first amplitude index can be any characteristic parameter used to quantize the degree of amplification of the output data amplitude by the gated linear unit. Its value is positively correlated with the amplification capability of the output amplitude of the gated linear unit. The larger the value, the higher the amplification of the output data amplitude by the gated linear unit, and the easier it is to cause quantization overflow; the smaller the value, the lower the amplification and the lower the risk of quantization overflow.

[0064] Step S22: Based on the first amplitude index, determine the target quantization granularity strategy from multiple quantization granularity strategies.

[0065] That is, as can be seen from the above description, during the quantization training of the model, since the two weights in the gated linear unit of the model's feedforward network are multiplicative, if the result of multiplying the values ​​of the same channel in the two weights is large, the amplitude of the output data will also be significantly amplified, which can easily lead to numerical overflow (i.e., numerical overflow, where the quantized value exceeds the upper limit of the value that the corresponding data type can represent) or precision degradation (i.e., numerical underflow, where the quantized value is, for example, quantized to 0, losing the meaning of feature representation), increasing the risk of quantization overflow.

[0066] Therefore, in this embodiment of the present disclosure, a first magnitude index is determined based on the first weight tensor and the second weight tensor completed in the current training step, which represents the amplification of the output data magnitude by the gated linear unit containing the two weights. Then, based on the first magnitude index, the target quantization granularity strategy to be used in subsequent training steps is determined, that is, a tensor quantization strategy that can reduce computational overhead and storage burden is selected, or a block quantization strategy that can reduce the risk of quantization overflow and avoid precision degradation or numerical overflow is selected.

[0067] By employing the above method, the target quantization granularity strategy in subsequent training steps is matched with the real-time dynamically changing weight parameters of the model. This allows the quantization training process for the model to balance model training efficiency and computational efficiency, achieving a balance between preserving quantization training accuracy and computational and storage costs.

[0068] Please refer to Figure 3 This is a flowchart for determining the first amplitude index provided in an embodiment of this disclosure. For example... Figure 3 As shown, in some embodiments, the first weight tensor and the second weight tensor may include the same number of multiple channels. In this embodiment, the determination of the first amplitude index based on the first weight tensor and the second weight tensor in step S21 may include steps S31-S32.

[0069] Step S31: Obtain the product of the first norm of each channel in the first weight tensor and the second norm of the corresponding channel in the second weight tensor to obtain the norm product of multiple channels.

[0070] The first norm and the second norm can be any one of the L1 norm, L2 norm, and infinite norm. In this embodiment of the disclosure, the first norm is the L2 norm of the corresponding channel in the first weight tensor and the second norm is the L2 norm of the corresponding channel in the second weight tensor, as an example for illustration.

[0071] Step S32: Determine the first amplitude index based on the norm product of the norm products of multiple channels, where the corresponding values ​​satisfy the first preset condition.

[0072] The first preset condition can be set as needed, for example, it can be any one of the maximum value, mean, or 95th percentile value; in this embodiment of the disclosure, since the numerical overflow of the output data is usually dominated by a very small number of channels, it is preferred that the first preset condition is to meet the maximum value condition, that is, to select the norm product with the largest value among the norm products of multiple channels to determine the first amplitude index.

[0073] It is understood that the first amplitude index can be directly the product of the norms with the largest value, or it can be the product of the product of the norms with the largest value and a preset weight coefficient with a fixed value. This disclosure does not make any special restrictions on this.

[0074] Specifically, taking the gated linear unit SwishGLU in the model as an example, as mentioned above, it can be represented as follows: ,in, , Let the weight tensor be the two linear transformations. This represents element-wise multiplication. y represents the activation function, such as the sigmoid activation function, x represents the input data, and y represents the output data of the SwishGLU gated linear unit.

[0075] for The output of any channel i , where i is a positive integer less than or equal to the number of channels of the tensor, and, for The output of any channel i They can be represented as follows:

[0076] Formula 1;

[0077] Formula 2;

[0078] The output of channel i in gated form It can be represented as:

[0079] Formula 3.

[0080] Given that numerical overflow during quantization training is typically dominated by a few channels with large gated input amplitudes, it is possible to satisfy... This is denoted as a risk channel, meaning a channel with the risk of quantization overflow; since the sigmoid activation function's value range is between 0 and 1, and because... Therefore, we can conclude that: Substituting it into formula 3 above, we get:

[0081] Formula 4;

[0082] Furthermore, based on the Cauchy–Schwarz inequality, we can obtain:

[0083] Formula 5;

[0084] Formula 6;

[0085] Substituting formulas 5 and 6 into formula 4, we get:

[0086] Formula 7;

[0087] Therefore, it can be seen that during the quantization training of the model, the product of the norms of the channels of the two weights in the gated linear unit can determine the upper limit of the amplification of the output data amplitude. When the norms of the two weights in the same channel are both large, quantization overflow is more likely to occur during quantization training according to the tensor quantization granularity strategy.

[0088] Therefore, in this embodiment of the disclosure, the first amplitude index can be determined based on the norm product with the largest corresponding value among the norm products of multiple channels obtained in step S31, so as to determine the target quantization granularity strategy based on the first amplitude index.

[0089] Please refer to Figure 4 This is a second flowchart of the strategy for determining the quantization granularity of the target provided in the embodiments of this disclosure. Figure 4 As shown, in some embodiments, the multiple quantization granularity strategies may include tensor quantization strategies and block quantization strategies; in this embodiment, the step S22 of determining the target quantization granularity strategy from the multiple quantization granularity strategies based on the first amplitude index may include the following steps S41-S42.

[0090] Step S41: If the target quantization granularity strategy of the current training step is tensor quantization strategy and the first amplitude index is greater than the first preset uplink threshold, update the target quantization granularity strategy to block quantization strategy.

[0091] Step S42: If the target quantization granularity strategy of the current training step is a block quantization strategy and the first amplitude index is less than the first preset downlink threshold, the target quantization granularity strategy is updated to a tensor quantization strategy.

[0092] Among them, the first preset downlink threshold is less than the first preset uplink threshold.

[0093] The values ​​of the first preset uplink threshold and the first preset downlink threshold can be set as needed, and this disclosure does not impose any special limitations on them.

[0094] Specifically, with This indicates that the first amplitude index is... The first preset uplink threshold is used to... Let denot the first preset downlink threshold, and let per-tensor represent the tensor quantization strategy and per-block represent the block quantization strategy. The above processing can be based on the following pseudocode to determine the target quantization granularity strategy.

[0095] if mode == per-tensor and then

[0096] mode=per-block;

[0097] else if mode==per-block and then

[0098] mode=per-block;

[0099] end if

[0100] As can be seen, based on the method provided in this embodiment, the target quantization granularity strategy for subsequent training steps can be dynamically determined based on the first amplitude index, so that the target quantization granularity strategy in subsequent training steps matches the weight parameters that change dynamically in real time in the model. This allows the quantization training process for the model to take into account both model training efficiency and computational efficiency, and achieve a balance between quantization training accuracy preservation and computational and storage costs.

[0101] Please refer to Figure 5 This is a first flowchart of quantitative training provided in an embodiment of this disclosure. Figure 5As shown, in some embodiments, multiple quantization granularity strategies may include tensor quantization strategies and block quantization strategies; step S13, which involves continuing to quantize and train the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model, may include the following steps S51-S53.

[0102] Step S51: When the target quantization granularity strategy is tensor quantization, set the target safe number of steps, including a preset number of training steps.

[0103] Among them, the target safety step count is a safety verification mechanism introduced to further avoid quantization overflow or accuracy degradation during model quantization training when the target quantization granularity strategy is tensor quantization strategy.

[0104] That is, when the target quantization granularity strategy is determined to be tensor quantization strategy, given that quantization overflow or accuracy degradation may occur during the quantization of the entire tensor, this disclosure also introduces a safety period mechanism to further avoid this risk. The preset number of training steps in the safety period mechanism can be 20, 30, 50 training steps, etc., and this disclosure does not make any special limitations on this.

[0105] Step S52: During the quantization training of the model within the target safe number of steps, if the first amplitude index corresponding to the gated linear unit is greater than the first preset uplink threshold in any training step, the target quantization granularity strategy is updated to the block quantization strategy, and the target safe number of steps is canceled.

[0106] The first amplitude index is used to represent the degree to which the gating linear unit amplifies the amplitude of the output data.

[0107] Step S53: Based on the updated target quantization granularity strategy, continue to quantize and train the model to obtain the trained model.

[0108] After setting the target safety step count, in each subsequent training step, it can be checked whether the first magnitude index of the first weight tensor and the second weight tensor that have undergone parameter updates in the corresponding training step are greater than the first preset uplink threshold. If so, in order to avoid the risk of quantization overflow, the training framework immediately updates the target quantization granularity strategy to the block quantization strategy and cancels the target safety step count.

[0109] For example, if the target safety step count is 5, then if the first amplitude index determined in real time is less than the first preset uplink threshold within the 5 training steps, the model can continue to be quantized and trained based on the target quantization granularity strategy; otherwise, the target quantization granularity strategy is updated to the block quantization strategy, the target safety step count is canceled, and the model continues to be quantized and trained using the updated block quantization strategy to obtain the trained model.

[0110] Let the safety period represent the target safety steps. Taking the pseudocode above as an example, the pseudocode can be supplemented into the following form based on the above processing.

[0111] if mode == per-tensor and then

[0112] mode=per-block;

[0113] else if mode==per-block and then

[0114] mode=per-block;

[0115] Enter safety period; / / Set target safety steps

[0116] end if

[0117] if in a safe period and in real time then

[0118] mode=per-block;

[0119] end if

[0120] until training ends / / Until training is complete and the trained model is obtained.

[0121] As can be seen, the method provided in this disclosure, by introducing a safe step mechanism, can further avoid quantization overflow or accuracy reduction during quantization training of the model based on two quantization strategies, thereby improving model training efficiency.

[0122] Please refer to Figure 6 This is a second flowchart of the quantitative training provided in the embodiments of this disclosure. Figure 6 As shown, in some embodiments, multiple quantization granularity strategies may include tensor quantization strategies and block quantization strategies; step S13, which involves continuing to quantize and train the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model, may include the following steps S61-S64.

[0123] Step S61: When the target quantization granularity strategy is tensor quantization, set the target safe number of steps, including a preset number of training steps.

[0124] Step S62: During the quantization training of the model within the target safe number of steps, determine the changing trend of the first amplitude index corresponding to the gated linear unit.

[0125] The first amplitude index is used to represent the degree to which the gating linear unit amplifies the amplitude of the output data.

[0126] This trend can represent the numerical change trend of the first amplitude index in each training step.

[0127] Step S63: If the trend does not meet the preset trend conditions, update the target quantization granularity strategy to the block quantization strategy and cancel the target safety steps.

[0128] The preset trend conditions can be: the numerical change continues to decrease, and the changed values ​​are all less than the first preset upward threshold, etc.

[0129] Step S64: Based on the updated target quantization granularity strategy, continue to quantize and train the model to obtain the trained model.

[0130] Specifically, in this embodiment of the disclosure, a security mechanism can be set, and within the security mechanism, if the trend of the real-time first amplitude index is a continuous decrease, it can be indicated that the risk of quantization overflow is not likely to occur during quantization training, and the model can continue to be quantized based on the tensor quantization strategy; otherwise, the block quantization strategy is switched.

[0131] Using "safety period" to represent the target safety steps, and still taking the above pseudocode as an example, the pseudocode can be updated to the following form.

[0132] if mode == per-tensor and then

[0133] mode=per-block;

[0134] else if mode==per-block and then

[0135] mode=per-block;

[0136] Enter safety period; / / Set target safety steps

[0137] end if

[0138] if in a safe period and in real time not decreasing then

[0139] mode=per-block; / / If the first amplitude indicator is not continuously decreasing, switch to per-block mode.

[0140] end if

[0141] until training ends / / Until training is complete and the trained model is obtained.

[0142] That is, through a strong verification mechanism, if the first magnitude index does not continue to decrease during the quantization training of the model when switching to tensor quantization strategy, it means that the values ​​of the first weight tensor and the second weight tensor may increase in subsequent training steps, and the first magnitude index may also increase, which may pose a risk of quantization overflow or reduced accuracy. Therefore, the quantization granularity strategy can be switched in time to improve the stability of model training.

[0143] It should be noted that in actual implementation, in the process of continuing to quantize the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model, the model can be quantized based on any of the above steps S51-S53 or S61-S64, or it can be quantized based on other methods as needed. This disclosure does not make any special limitations on this.

[0144] It should be noted that, in some embodiments, the step S12 of determining the target quantization granularity strategy of the model in subsequent training steps from multiple quantization granularity strategies based on the first weight tensor and the second weight tensor may include: determining a first amplitude index and a second amplitude index based on the first weight tensor and the second weight tensor; wherein the first amplitude index and the second amplitude index are used to represent the degree of amplification of the output data amplitude by the gated linear unit from different perspectives; and determining the target quantization granularity strategy from multiple quantization granularity strategies based on the first amplitude index and / or the second amplitude index.

[0145] In this embodiment, the first weight tensor and the second weight tensor include the same number of channels. Determining the second amplitude index based on the first weight tensor and the second weight tensor may include: obtaining the correlation value between each channel in the first weight tensor and the corresponding channel in the second weight tensor to obtain the correlation values ​​of multiple channels; each correlation value is used to represent the degree of similarity between the corresponding channels in the first weight tensor and the second weight tensor; and determining the second amplitude index based on the correlation values ​​of the multiple channels whose corresponding values ​​satisfy a second preset condition.

[0146] Regarding the first amplitude indicator, please refer to the relevant explanations above, and it will not be repeated here.

[0147] As shown above, the second amplitude index can be a correlation value among the correlation values ​​of multiple channels that satisfies a second preset condition. This second preset condition can be set as needed, for example, it can be any one of the maximum value, mean, or 95th percentile value. In this embodiment, since numerical overflow of the output data is usually dominated by a very small number of channels, it is preferable that the second preset condition is the maximum value condition, that is, the correlation value with the largest corresponding value among the correlation values ​​of multiple channels is selected to determine the second amplitude index.

[0148] The correlation values ​​of these multiple channels can be, for example, any one of the following: cosine similarity, Pearson correlation coefficient, normalized inner product, covariance measure, etc., between corresponding channels in the first and second weight tensors.

[0149] In this embodiment, the correlation value is exemplified as the cosine similarity between corresponding channels in the first and second weighted tensors. Of course, in actual implementation, this correlation value can also be obtained using other methods that can represent the similarity between the same channels of two tensors; this disclosure does not impose any special limitations on this.

[0150] Specifically, taking SwishGLU as the gated linear unit in the model as an example, as mentioned above, it can be represented as follows: ,in, , Let the weight tensor be the two linear transformations. This represents element-wise multiplication. y represents the activation function, such as the sigmoid activation function, x represents the input data, and y represents the output data of the SwishGLU gated linear unit.

[0151] Then, for , For any channel i, its channel correlation value For example, it can be obtained based on the following formula 8:

[0152] Formula 8;

[0153] in, It can be a very small positive number to prevent the denominator from being 0, and its value can be set as needed.

[0154] That is, in order to further improve the model training efficiency and avoid the risk of quantization overflow during quantization training, this disclosure also introduces a second amplitude index that represents the amplification degree of the output data amplitude by the gated linear unit from different perspectives. In actual implementation, the target quantization granularity strategy can be determined based on the first amplitude index and / or the second amplitude index.

[0155] It is understandable that the preferred approach is to determine the target quantization granularity strategy based on both the first amplitude indicator and the second amplitude indicator.

[0156] That is, in some embodiments, determining the target quantization granularity strategy from multiple quantization granularity strategies based on the first amplitude index in step S22 may include: updating the target quantization granularity strategy to a block quantization strategy when the target quantization granularity strategy in the current training step is a tensor quantization strategy, and the first amplitude index is greater than a first preset uplink threshold and the second amplitude index is greater than a second preset uplink threshold; updating the target quantization granularity strategy to a tensor quantization strategy when the target quantization granularity strategy in the current training step is a block quantization strategy, and the first amplitude index is less than a first preset downlink threshold and the second amplitude index is less than a second preset downlink threshold; wherein the first preset downlink threshold is less than the first preset uplink threshold, and the second preset downlink threshold is less than the second preset uplink threshold.

[0157] The values ​​of the first preset uplink threshold, the first preset downlink threshold, the second preset uplink threshold, and the second preset downlink threshold can be set as needed, and this disclosure does not impose any special limitations on them.

[0158] That is, in this implementation, when the target quantization granularity strategy for the current training step is tensor quantization, and both the first and second amplitude metrics are greater than their respective uplink thresholds, given that quantization overflow or accuracy degradation may occur during quantization of the entire tensor, if both the first and second amplitude metrics are greater than their respective uplink thresholds in the current training step, the training framework updates the target quantization granularity strategy to block quantization to avoid the risk of quantization overflow. Conversely, when the target quantization granularity strategy for the current training step is block quantization, and both the first and second amplitude metrics are less than their respective downlink thresholds, given that quantization overflow is generally less likely in this case, the training framework can update the target quantization granularity strategy to tensor quantization to reduce computational overhead and storage costs. This achieves a balance between preserving accuracy during quantization training and reducing computational overhead and storage costs, thereby improving model training efficiency.

[0159] Of course, in other cases, the target quantization granularity strategy does not need to be updated. That is, the target quantization granularity strategy of the current training step is still used for quantization training in subsequent training steps. The detailed processing procedure will not be elaborated here.

[0160] It is understood that, in this implementation, when the target quantization granularity strategy is a tensor quantization strategy, a target safe number of training steps, including a preset number of training steps, can also be set. During quantization training of the model within the target safe number of training steps, if the first amplitude index corresponding to the gated linear unit is greater than a first preset uplink threshold and the second amplitude threshold is greater than a second preset uplink threshold in any training step, the target quantization granularity strategy can be updated to a block quantization strategy, and the target safe number of training steps can be canceled. Furthermore, based on the updated target quantization granularity strategy, the model can continue to be quantized to obtain the trained model.

[0161] Alternatively, during the quantization training of the model within the target safe number of steps, a first trend of change of the first amplitude index and a second trend of change of the second amplitude index can be obtained; if either the first trend or the second trend does not meet the preset trend condition, the target quantization granularity strategy is updated to a block quantization strategy, and the target safe number of steps is canceled; and, according to the updated target quantization granularity strategy, the model is continued to be quantized to obtain the trained model.

[0162] That is, with This indicates the first amplitude index. This indicates the second amplitude index, in The first preset uplink threshold is used to... Indicates the first preset downlink threshold, to Indicates the second preset uplink threshold, to The second preset downlink threshold is represented by "per-tensor" to indicate the tensor quantization strategy and "per-block" to indicate the block quantization strategy. The target quantization granularity strategy can then be determined based on the following pseudocode.

[0163] if mode == per-tensor and and then

[0164] mode=per-block;

[0165] else if mode==per-block and and then

[0166] mode=per-block;

[0167] Enter safety period; / / Set target safety steps

[0168] end if

[0169] if in a safe period and in real time or real-time not decreasing then

[0170] mode=per-block; / / If the first or second amplitude indicator is not continuously decreasing, switch to per-block mode.

[0171] end if

[0172] until training ends / / Until training is complete and the trained model is obtained.

[0173] As can be seen, based on the method provided in the embodiments of this disclosure, by introducing a first amplitude index and / or a second amplitude index that can represent the degree of amplification of the output data amplitude by the gated linear unit, and using the first amplitude index and / or the second amplitude index as risk judgment indicators, the target quantization granularity strategy can be dynamically selected during the quantization training of the model, thereby avoiding quantization risks, improving model training efficiency, and reducing computational overhead and storage burden during model training.

[0174] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0175] In addition, this disclosure also provides a model training device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the model training methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the relevant section on methods and will not be repeated here.

[0176] Figure 7 This is a block diagram of a model training apparatus provided in an embodiment of the present disclosure.

[0177] Reference Figure 7 This disclosure provides a model training device, which includes: an acquisition module 701, a determination module 702, and a training module 703.

[0178] The acquisition module 701 is used to acquire the first weight tensor and the second weight tensor of the gated linear unit of the model after training in the current training step during the quantization training of the model; wherein, the quantization training includes, in each training step, performing numerical quantization processing on the first weight tensor and the second weight tensor of the gated linear unit based on the quantization granularity strategy of the training step, and performing model training based on the quantized first weight tensor and the second weight tensor.

[0179] The determining module 702 is used to determine the target quantization granularity strategy of the model in subsequent training steps from multiple quantization granularity strategies based on the first weight tensor and the second weight tensor; wherein the multiple quantization granularity strategies have different quantization granularities.

[0180] The training module 703 is used to continue quantizing the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model.

[0181] In some embodiments, when determining the target quantization granularity policy of the model in subsequent training steps from multiple quantization granularity policies based on the first weight tensor and the second weight tensor, the determining module 702 may be used to:

[0182] A first magnitude index is determined based on the first weight tensor and the second weight tensor; wherein, the first magnitude index is used to represent the degree to which the gated linear unit amplifies the magnitude of the output data;

[0183] Based on the first amplitude index, the target quantization granularity strategy is determined from multiple quantization granularity strategies.

[0184] In some embodiments, the first weight tensor and the second weight tensor include the same number of channels; when determining the first amplitude index based on the first weight tensor and the second weight tensor, the determining module 702 can be used to:

[0185] Obtain the product of the first norm of each channel in the first weight tensor and the second norm of the corresponding channel in the second weight tensor to obtain the norm product of multiple channels;

[0186] The first amplitude index is determined based on the norm product of the multiple channels whose corresponding values ​​satisfy the first preset condition.

[0187] In some embodiments, the plurality of quantization granularity strategies include tensor quantization strategies and block quantization strategies; when determining the target quantization granularity strategy from the plurality of quantization granularity strategies based on the first amplitude index, the determining module 702 can be used to:

[0188] If the target quantization granularity strategy for the current training step is the tensor quantization strategy, and the first amplitude index is greater than the first preset uplink threshold, the target quantization granularity strategy is updated to the block quantization strategy.

[0189] If the target quantization granularity strategy for the current training step is the block quantization strategy, and the first amplitude index is less than the first preset downlink threshold, the target quantization granularity strategy is updated to the tensor quantization strategy; wherein the first preset downlink threshold is less than the first preset uplink threshold.

[0190] In some embodiments, the plurality of quantization granularity strategies include tensor quantization strategies and block quantization strategies; when the training module 703 continues to quantize and train the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model, it can be used for:

[0191] When the target quantization granularity strategy is the tensor quantization strategy, a target safe number of steps is set, including a preset number of training steps.

[0192] During the quantization training of the model within the target safe number of steps, if the first amplitude index corresponding to the gated linear unit is greater than the first preset uplink threshold in any training step, the target quantization granularity strategy is updated to the block quantization strategy, and the target safe number of steps is canceled; wherein, the first amplitude index is used to represent the degree of amplification of the output data amplitude by the gated linear unit;

[0193] Based on the updated target quantization granularity strategy, the model is further quantized and trained to obtain the trained model.

[0194] In some embodiments, the plurality of quantization granularity strategies include tensor quantization strategies and block quantization strategies; when the training module 703 continues to quantize and train the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model, it can be used for:

[0195] When the target quantization granularity strategy is the tensor quantization strategy, a target safe number of steps is set, including a preset number of training steps.

[0196] During the quantization training of the model within the target safe number of steps, the changing trend of the first amplitude index corresponding to the gated linear unit is determined; wherein, the first amplitude index is used to represent the degree to which the gated linear unit amplifies the amplitude of the output data;

[0197] If the trend does not meet the preset trend conditions, the target quantization granularity strategy is updated to the block quantization strategy, and the target safety step is canceled.

[0198] Based on the updated target quantization granularity strategy, the model is further quantized and trained to obtain the trained model.

[0199] In some embodiments, when determining the target quantization granularity policy of the model in subsequent training steps from multiple quantization granularity policies based on the first weight tensor and the second weight tensor, the determining module 702 may be used to:

[0200] Based on the first weight tensor and the second weight tensor, a first amplitude index and a second amplitude index are determined; wherein, the first amplitude index and the second amplitude index are used to represent the degree of amplification of the output data amplitude by the gated linear unit from different perspectives;

[0201] The target quantization granularity strategy is determined from multiple quantization granularity strategies based on the first amplitude index and / or the second amplitude index.

[0202] In some embodiments, the first weight tensor and the second weight tensor include the same number of channels; when determining the second amplitude index based on the first weight tensor and the second weight tensor, the determining module 702 can be used to:

[0203] The correlation value between each channel in the first weight tensor and the corresponding channel in the second weight tensor is obtained to obtain the correlation values ​​of multiple channels; each correlation value is used to represent the degree of similarity between the corresponding channels of the first weight tensor and the second weight tensor.

[0204] The second amplitude index is determined based on the correlation values ​​of the multiple channels that satisfy the second preset condition.

[0205] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0206] Reference Figure 8 This disclosure provides an electronic device, which includes: at least one processor 801; at least one memory 802; and one or more I / O interfaces 803 connected between the processor 801 and the memory 802; wherein the memory 802 stores one or more computer programs that can be executed by the at least one processor 801, and the one or more computer programs are executed by the at least one processor 801 to enable the at least one processor 801 to perform the above-described model training method.

[0207] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the model training method described above. The computer-readable storage medium may be volatile or non-volatile.

[0208] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the model training method described above.

[0209] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0210] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0211] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0212] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0213] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0214] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0215] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0216] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0217] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0218] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A model training method, characterized in that, include: During the quantization training of the model, the first weight tensor and the second weight tensor of the gated linear unit of the model are obtained after training in the current training step; wherein, the quantization training includes, in each training step, performing numerical quantization processing on the first weight tensor and the second weight tensor of the gated linear unit based on the quantization granularity strategy of the training step, and training the model based on the quantized first weight tensor and the second weight tensor. Based on the first weight tensor and the second weight tensor, the target quantization granularity strategy of the model in subsequent training steps is determined from multiple quantization granularity strategies; wherein the quantization granularity of the multiple quantization granularity strategies is different. Based on the target quantization granularity strategy, the model is further quantized and trained in subsequent training steps to obtain the trained model.

2. The method according to claim 1, characterized in that, The step of determining the target quantization granularity strategy for the model in subsequent training steps from multiple quantization granularity strategies based on the first weight tensor and the second weight tensor includes: A first magnitude index is determined based on the first weight tensor and the second weight tensor; wherein, the first magnitude index is used to represent the degree to which the gated linear unit amplifies the magnitude of the output data; Based on the first amplitude index, the target quantization granularity strategy is determined from multiple quantization granularity strategies.

3. The method according to claim 2, characterized in that, The first weight tensor and the second weight tensor both include the same number of channels; The step of determining the first magnitude index based on the first weight tensor and the second weight tensor includes: The product of the first norm of each channel in the first weight tensor and the second norm of the corresponding channel in the second weight tensor is obtained to obtain the norm product of the multiple channels; The first amplitude index is determined based on the norm product of the multiple channels whose corresponding values ​​satisfy the first preset condition.

4. The method according to claim 2, characterized in that, The multiple quantization granularity strategies include tensor quantization strategy and block quantization strategy; The step of determining the target quantization granularity strategy from multiple quantization granularity strategies based on the first amplitude index includes: If the target quantization granularity strategy for the current training step is the tensor quantization strategy, and the first amplitude index is greater than the first preset uplink threshold, the target quantization granularity strategy is updated to the block quantization strategy. If the target quantization granularity strategy for the current training step is the block quantization strategy, and the first amplitude index is less than the first preset downlink threshold, the target quantization granularity strategy is updated to the tensor quantization strategy; wherein the first preset downlink threshold is less than the first preset uplink threshold.

5. The method according to claim 1, characterized in that, The multiple quantization granularity strategies include tensor quantization strategy and block quantization strategy; The step of continuing to quantize and train the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model includes: When the target quantization granularity strategy is the tensor quantization strategy, a target safe number of steps is set, including a preset number of training steps. During the quantization training of the model within the target safe number of steps, if the first amplitude index corresponding to the gated linear unit is greater than the first preset uplink threshold in any training step, the target quantization granularity strategy is updated to the block quantization strategy, and the target safe number of steps is canceled; wherein, the first amplitude index is used to represent the degree of amplification of the output data amplitude by the gated linear unit; Based on the updated target quantization granularity strategy, the model is further quantized and trained to obtain the trained model.

6. The method according to claim 1, characterized in that, The multiple quantization granularity strategies include tensor quantization strategy and block quantization strategy; The step of continuing to quantize and train the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model includes: When the target quantization granularity strategy is the tensor quantization strategy, a target safe number of steps is set, including a preset number of training steps. During the quantization training of the model within the target safe number of steps, the changing trend of the first amplitude index corresponding to the gated linear unit is determined; wherein, the first amplitude index is used to represent the degree to which the gated linear unit amplifies the amplitude of the output data; If the trend does not meet the preset trend conditions, the target quantization granularity strategy is updated to the block quantization strategy, and the target safety step is canceled. Based on the updated target quantization granularity strategy, the model is further quantized and trained to obtain the trained model.

7. The method according to claim 1, characterized in that, The step of determining the target quantization granularity strategy for the model in subsequent training steps from multiple quantization granularity strategies based on the first weight tensor and the second weight tensor includes: Based on the first weight tensor and the second weight tensor, a first amplitude index and a second amplitude index are determined; wherein, the first amplitude index and the second amplitude index are used to represent the degree of amplification of the output data amplitude by the gated linear unit from different perspectives; The target quantization granularity strategy is determined from multiple quantization granularity strategies based on the first amplitude index and / or the second amplitude index.

8. The method according to claim 7, characterized in that, The first weight tensor and the second weight tensor both include the same number of channels; The step of determining the second magnitude index based on the first weight tensor and the second weight tensor includes: The correlation value between each channel in the first weight tensor and the corresponding channel in the second weight tensor is obtained to obtain the correlation values ​​of the multiple channels; each correlation value is used to represent the degree of similarity between the corresponding channels of the first weight tensor and the second weight tensor. The second amplitude index is determined based on the correlation values ​​of the multiple channels that satisfy the second preset condition.

9. The method according to any one of claims 1-8, characterized in that, The model is used to perform any one of the following tasks: image processing, speech processing, text processing, and video processing.

10. A model training device, characterized in that, include: The acquisition module is used to acquire the first weight tensor and the second weight tensor of the gated linear unit of the model after training in the current training step during the quantization training of the model; wherein, the quantization training includes, in each training step, performing numerical quantization processing on the first weight tensor and the second weight tensor of the gated linear unit based on the quantization granularity strategy of the training step, and performing model training based on the quantized first weight tensor and the second weight tensor. The determination module is configured to determine the target quantization granularity strategy of the model in subsequent training steps from multiple quantization granularity strategies based on the first weight tensor and the second weight tensor; wherein the multiple quantization granularity strategies have different quantization granularities. The training module is used to continue quantizing the model in subsequent training steps according to the target quantization granularity strategy to obtain the trained model.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the model training method as described in any one of claims 1-9.

Images