A model training method and device, computer equipment and readable storage medium

Abstract

This application relates to a model training method, apparatus, computer device, and readable storage medium. The method includes: after the current round of training of the target model has ended, determining the current round gradient information of each network layer in the target model based on the current round scaling factors of different training groups in the target model during the current round of training; wherein different training groups are configured with independent initial scaling factors; and predicting the next round scaling factor for each training group in the next round of training based on the current round gradient information, the current round scaling factor of each training group, and the precision format. This method can improve the stability and convergence efficiency of the model during mixed-precision training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, computer device, and readable storage medium. Background Technology

[0002] With the development of artificial intelligence technology, mixed-precision training is often used when training models. This approach balances training speed and accuracy by setting different precision formats for different model layers. For example, some model layers use FP16 (half-precision floating-point numbers) to improve training speed, while others use PF32 (single-precision floating-point numbers) to ensure training accuracy.

[0003] In related technologies, during model training, low-precision formats (such as FP16) have a fixed numerical range limitation. If the gradient of a low-precision layer of the model is less than the lower limit of this numerical range, the gradient will be rounded to 0, and the model will be unable to update its parameters, resulting in model training failure. Usually, a global single scaling factor is set for all layers in the model to avoid the above problem by amplifying the gradient.

[0004] However, the global single scaling factor in related technologies cannot adapt to the significant differences in gradient distribution among different network layers in the model, resulting in gradient underflow or overflow in some layers, which affects the stability and convergence efficiency of the model during mixed precision training. Summary of the Invention

[0005] Based on this, it is necessary to provide a model training method, apparatus, computer equipment, and readable storage medium to address the above-mentioned technical problems. By configuring independent scaling factors for different training groups in the model, the scaling factors of different selected training groups can be adjusted differently to adapt to the significantly different gradient distribution characteristics of network layers in different training groups, significantly reduce the risk of gradient overflow in each training group, and improve the stability and convergence efficiency of the model during mixed precision training.

[0006] Firstly, this application provides a model training method, including:

[0007] When the current round of training of the target model has ended, the current round gradient information of each network layer in the target model is determined based on the current round scaling factor of different training groups in the target model during the current round of training. Among them, different training groups are network layers of the same type and precision format that are divided according to the structure of the target model, and different training groups are configured with independent initial scaling factors.

[0008] Based on the gradient information of the current round, as well as the scaling factor and precision format of each training group in the current round, predict the scaling factor of each training group in the next round of training.

[0009] In one embodiment, the model training method further includes: after the next round of training of the target model is completed, determining the next round gradient information of each network layer in the target model according to the next round scaling factor of each training group, and repeating the above operation until the target model meets the model convergence condition, and determining that the target model training is completed.

[0010] In one embodiment, the current round gradient information of each network layer in the target model is determined based on the current round scaling factor of different training groups in the target model during the current round of training. This includes: scaling the loss value of the target model during the current round of training according to the current round scaling factor of each training group to obtain the scaling loss value corresponding to each training group; and determining the current round gradient information of each network layer in the target model based on the scaling loss value and the parameter information of each network layer in each training group.

[0011] In one embodiment, the loss value for the current training round is determined as follows: the training loss value of the target model is determined based on the prediction results of the target model for the training samples and the labels of the training samples in the current training round; for each training group, the intermediate loss value corresponding to the training group is determined based on the training loss value, the current round scaling factor of the training group, and the preset weights; and the loss value for the current training round is determined based on the sum of the intermediate loss values ​​corresponding to different training groups.

[0012] In one embodiment, based on the current round gradient information and the current round scaling factor and precision format of each training group, the next round scaling factor of each training group in the next training round is predicted, including: for any training group, based on the numerical range corresponding to the current round scaling factor and precision format of the training group, and the current round gradient information of each network layer in the training group, the next round gradient overflow risk of the training group is predicted; based on the next round gradient overflow risk of the training group, the current round scaling factor of the training group is adjusted to obtain the next round scaling factor of the training group in the next training round.

[0013] In one embodiment, the gradient overflow risk in the next round includes the gradient overflow risk and the gradient underflow risk in the next round. Predicting the gradient overflow risk in the next round of the training group based on the current round scaling factor and the numerical range corresponding to the precision format of the training group, as well as the current round gradient information of each network layer in the training group, includes: determining the mean gradient and the maximum absolute gradient value corresponding to the training group based on the current round gradient information of each network layer in the training group; predicting the upper limit of the future gradient corresponding to the training group based on the product of the maximum absolute gradient value and the current round scaling factor of the training group, and predicting the gradient overflow risk in the next round of the training group based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group; predicting the lower limit of the future gradient corresponding to the training group based on the product of the mean gradient and the current round scaling factor of the training group, and predicting the gradient underflow risk in the next round of the training group based on the lower limit of the future gradient and the lower limit of the numerical range.

[0014] In one embodiment, the current round gradient information of each network layer is represented by the current round gradient matrix; based on the current round gradient information of each network layer in the training group, the mean gradient and the maximum absolute gradient of the training group are determined, including: determining the average value of each element in the current round gradient matrix of each network layer in the training group as the mean gradient of the training group; and selecting the maximum value among the absolute values ​​of each element in the current round gradient matrix of each network layer in the training group as the maximum absolute gradient of the training group.

[0015] In one embodiment, predicting the gradient overflow risk of the training group in the next round based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group includes: if the upper limit of the future gradient is greater than a preset upper limit threshold, determining that the gradient overflow risk of the training group in the next round indicates that there is a gradient overflow risk in the next round; wherein, the preset upper limit threshold is determined by the product of the upper limit of the numerical range corresponding to the precision format of the training group and a first coefficient, and the first coefficient is greater than 0 and less than 1.

[0016] In one embodiment, predicting the gradient underflow risk of the training group in the next round based on the future gradient lower bound and the lower bound of the numerical range includes: if the future gradient lower bound is less than a preset lower bound threshold, determining that the gradient overflow risk of the training group in the next round indicates that there is a gradient underflow risk in the next round; wherein the preset lower bound threshold is determined based on the product of the lower bound of the numerical range and a second coefficient, and the second coefficient is greater than 1.

[0017] In one embodiment, the scaling factor of the current round of the training group is adjusted according to the gradient overflow risk of the next round to obtain the scaling factor of the training group in the next round of training. This includes: if the gradient overflow risk representation indicates a gradient overflow risk in the next round, the scaling factor of the current round of the training group is reduced according to a first value to obtain the scaling factor of the training group in the next round of training; wherein the first value is greater than 0 and less than 1; if the gradient overflow risk representation indicates a gradient underflow risk in the next round, the scaling factor of the current round of the training group is increased according to a second value to obtain the scaling factor of the training group in the next round of training; wherein the second value is greater than 1; if the gradient overflow risk representation indicates no gradient overflow risk or gradient underflow risk in the next round, the scaling factor of the training group in the next round of training is predicted according to the number of consecutive rounds in which the gradient overflow risk does not exist.

[0018] In one embodiment, when the gradient overflow risk representation for the next round does not contain gradient overflow risk or gradient underflow risk, the following steps are taken based on the consecutive training rounds without gradient overflow risk or gradient underflow risk: when the consecutive training rounds without gradient overflow risk or gradient underflow risk reach a preset number, the scaling factor of the current round of the training group is amplified according to a third value to obtain the scaling factor of the next round of the training group in the next round of training; wherein the third value is greater than 1 and less than the second value, and the consecutive preset number of training rounds include the current round of training.

[0019] In one embodiment, before predicting the scaling factor of each training group in the next training round based on the current round gradient information, the current round scaling factor and precision format of each training group, the model training method further includes: for any network layer in the target model, determining the ratio of the current round gradient information of the network layer to the current round scaling factor of the training group to which the network layer belongs; and adjusting the model parameters of the network layer according to the ratio.

[0020] In one embodiment, the target model is an object detection model, and different training groups include an embedding layer training group, a convolutional layer training group, and a fully connected layer training group. The model training method further includes: identifying the embedding layer, convolutional layer, and fully connected layer in the target model according to its structure; grouping the identified embedding layers with the same accuracy format into the same embedding layer training group; grouping the identified convolutional layers with the same accuracy format into the same convolutional layer training group; grouping the identified fully connected layers with the same accuracy format into the same fully connected layer training group; and configuring independent initial scaling factors for the embedding layer training group, the convolutional layer training group, and the fully connected layer training group, respectively.

[0021] Secondly, this application also provides a model training apparatus, comprising:

[0022] The gradient determination module is used to determine the gradient information of each network layer in the target model in the current round of training based on the scaling factor of the different training groups in the target model in the current round of training when the current round of training of the target model has ended. The different training groups are network layers of the same type and precision format according to the structure of the target model, and the different training groups are configured with independent initial scaling factors.

[0023] The factor prediction module is used to predict the scaling factor of each training group in the next training round based on the gradient information of the current round, as well as the scaling factor and precision format of each training group in the current round.

[0024] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the method embodiments of the first aspect described above.

[0025] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the methods of the first aspect described above.

[0026] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the methods of the first aspect described above.

[0027] In the aforementioned model training method, apparatus, computer equipment, and readable storage medium, network layers of the same type and precision format are pre-divided into different training groups based on the structure of the target model, and independent initial scaling factors are configured for each training group. Then, during multiple rounds of training of the target model, at the end of the current round, the current round gradient information of each network layer is determined based on the current round scaling factors of different training groups in the current round. Based on the current round gradient information, the current round scaling factors and precision formats of each training group, the scaling factors for the next round of training are predicted for each training group. Thus, by grouping the network layers in the target model and configuring independent scaling factors for different training groups, the scaling factors of different training groups can be independently adjusted based on the gradient information of the network layers in different training groups during the training process. This allows for differentiated adjustment of the scaling factors for different training groups, adapting to the significantly different gradient distribution characteristics of the network layers in different training groups. This enhances the specificity of the scaling factors for different training groups, significantly reduces the risk of gradient overflow in each training group, and improves the stability and convergence efficiency of the model during mixed precision training. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 Internal structural diagrams of a computer device provided in some embodiments of this application;

[0030] Figure 2 A schematic flowchart illustrating the model training method provided in some embodiments of this application;

[0031] Figure 3 This application provides a flowchart illustrating the process of determining the current round gradient information in some embodiments.

[0032] Figure 4 A schematic diagram illustrating the process for determining the loss value of the current training round, provided for some embodiments of this application;

[0033] Figure 5 A flowchart illustrating the prediction of the scaling factor for the next round, provided for some embodiments of this application;

[0034] Figure 6 A flowchart illustrating the process of predicting the risk of gradient overflow in the next round, provided for some embodiments of this application;

[0035] Figure 7 A schematic flowchart illustrating the model training method provided in other embodiments of this application;

[0036] Figure 8 This is a structural block diagram of a model training apparatus provided in some embodiments of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments or any combination of multiple embodiments.

[0039] With the development of artificial intelligence technology, mixed-precision training is often used when training models. This approach balances training speed and accuracy by setting different precision formats for different model layers. For example, some model layers use FP16 (half-precision floating-point numbers) to improve training speed, while others use PF32 (single-precision floating-point numbers) to ensure training accuracy.

[0040] In related technologies, during model training, low-precision formats (such as FP16) have a fixed numerical range limitation. If the gradient of a low-precision layer of the model is less than the lower limit of this numerical range, the gradient will be rounded to 0, and the model will be unable to update its parameters, resulting in model training failure. Usually, a global single scaling factor is set for all layers in the model to avoid the above problem by amplifying the gradient.

[0041] However, the global single scaling factor in related technologies cannot adapt to the significant differences in gradient distribution among different network layers in the model, resulting in gradient underflow or overflow in some layers, which affects the stability and convergence efficiency of the model during mixed precision training.

[0042] Based on this, this application provides a model training method that, by configuring independent scaling factors for different training groups in the model, can differentiate the scaling factors of different selected training groups to adapt to the significantly different gradient distribution characteristics of network layers in different training groups, significantly reduce the risk of gradient overflow in each training group, and improve the stability and convergence efficiency of the model during mixed precision training.

[0043] The model training method provided in this application can be applied to computer devices, which can be terminals or servers. Terminals can be, but are not limited to, various personal computers, laptops, smartphones, tablets, etc.; servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services.

[0044] In one exemplary embodiment, taking the computer device as a server, its internal structure can be as follows: Figure 1As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores triple extraction templates, sample data, and other data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a model training method.

[0045] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0046] Based on the above embodiments, in an exemplary embodiment, such as Figure 2 As shown, a model training method is provided, including:

[0047] S201, when the current round of training of the target model has ended, determine the current round gradient information of each network layer in the target model based on the current round scaling factor of different training groups in the target model during the current round of training.

[0048] Different training groups are formed by dividing network layers of the same type and accuracy format according to the structure of the target model, and each training group is configured with an independent initial scaling factor.

[0049] A target model is any model that needs to be trained using training samples that match the application requirements, so that the output of the trained model meets the application requirements. For example, an image detection model trained using sample images labeled with the object to be detected, a speech recognition model trained using sample speech, an information recommendation model trained using sample recommendation information, or a large language model trained using sample question-answer pairs. In addition, the target model can be any model such as a neural network model (e.g., convolutional neural network model, recurrent neural network model, etc.), a large model (e.g., a large language model, etc.), or a traditional machine learning model (e.g., logistic regression model, decision tree, support vector machine, etc.). There are no specific limitations on the application scenarios (e.g., object detection, speech recognition, information recommendation, etc.) and types of the target model.

[0050] When training a target model with mixed precision, different precision formats can be set for different network layers of the target model. For example, if the target model includes embedding layers, convolutional layers, and fully connected layers, the precision format for the embedding layers and fully connected layers can be set to FP16, while the precision format for the convolutional layers can be set to FP32.

[0051] Based on this, before training the target model, the network layers of the target model are first divided according to the structure of the target model to obtain training groups of the target model. Each training group includes one network layer, or multiple network layers with the same type and precision format (i.e., different network layers in the same training group have the same type and precision format). The so-called network layers with the same type refer to multiple independent network layers that use the same operation logic, implement the same core function, and have the same hierarchical structure. For example, different convolutional layers in the model are network layers with the same type, different fully connected layers are network layers with the same type, while convolutional layers and fully connected layers are network layers with different types. Furthermore, the parameter information of network layers with the same type can be different. For example, different convolutional layers with different convolutional kernels are still network layers with the same type.

[0052] Furthermore, independent initial scaling factors are configured for different training groups. These initial scaling factors can be the same or different across different training groups. For example, the initial scaling factor for training groups containing embedding layers is set to a larger value (e.g., 131072), the initial scaling factor for training groups containing output layers is set to a smaller value (e.g., 32768), and the initial scaling factor for training groups containing intermediate layers is set to a moderate value (e.g., 65536). Optionally, the initial scaling factor for each training group is the upper limit of the numerical range corresponding to the precision format of each network layer in that training group.

[0053] Optionally, the network layers in each training group include all or part of the network layers in the target model that contain trainable parameters (such as weights, bias matrices, etc.). That is, in some cases, the target model is trained as a whole, and the network layers in the target model that contain trainable parameters are divided into training groups; while in other cases, only some network layers in the target model are trained (such as LoRA (Low-Rank Adaptation) fine-tuning for large language models), and the network layers in the target model that contain trainable parameters and need to be trained are divided into training groups.

[0054] After dividing the target model into training groups and defining the initial scaling factor for each group, training of the target model begins. For each training round of the target model, upon completion of the current round, the training results are first used to determine if the target model meets the convergence criteria. If the convergence criteria are not met, the next round of training for the target model needs to be initiated. For example, it is determined whether the target model's loss value is less than a preset loss threshold. If the loss value is not less than the preset loss threshold, and the convergence criteria are still not met, the next round of training for the target model needs to be initiated.

[0055] Therefore, when the current round of training of the target model ends and the target model does not meet the model convergence condition, for each training group, the current gradient information of each network layer in the training group is determined based on the scaling factor of the current round of training for that training group. The scaling factor of the current round of training for that training group is the initial scaling factor configured for that training group.

[0056] The gradient information of each network layer refers to the partial derivative of the loss value of the target model with respect to the trainable parameters of that network layer. Based on the scaling factor of each training group in the current round of training, starting from the loss value of the target model in the current round of training, the partial derivative of the loss value with respect to each network layer is calculated sequentially through backpropagation (in the reverse direction of the network layers in the target model) to obtain the gradient information of each network layer in the current round.

[0057] In one optional embodiment, the model training method further includes, for any network layer in the target model, determining the ratio of the current round gradient information of the network layer to the current round scaling factor of the training group to which the network layer belongs; and adjusting the model parameters of the network layer according to the ratio.

[0058] In this scenario, if the target model fails to meet the convergence condition after the current training round, the model parameters (i.e., trainable parameters) of each network layer in the target model need to be adjusted to obtain the model parameters for each network layer participating in the next training round. Specifically, for any network layer in the target model, the scaling factor of the current round of the training group to which that network layer belongs is used as its current round scaling factor. First, the ratio of the current gradient information of that network layer to its current round scaling factor is determined (i.e., the inverse scaling operation). Then, the model parameters of that network layer are adjusted according to this ratio to obtain the model parameters for that network layer participating in the next training round. That is, the model parameters used by that network layer in the next training round are the adjusted model parameters. Optionally, for any network layer, the product of the learning rate of that network layer and the above ratio is determined. Then, the difference between each current round model parameter of that network layer and the above product is determined as the model parameter for the next round. Here, the current round model parameters are the model parameters of that network layer participating in the current round of training, while the next round model parameters are the model parameters of that network layer participating in the next round of training. This enables differentiated adjustment and dynamic adaptation of model parameters for each network layer, adapting to significant differences in gradient distribution across different network layers, and improving the stability and convergence efficiency of the target model during mixed-precision training.

[0059] S202, based on the gradient information of the current round, as well as the scaling factor and precision format of each training group in the current round, predict the scaling factor of each training group in the next round of training.

[0060] After obtaining the current gradient information of each network layer, since the current gradient information of different network layers is significantly different, the scaling factor adapted to different training groups is different. Furthermore, since the precision format of network layers in different training groups may be different, and the gradient overflow risk of network layers in different training groups is different, for each training group, based on the current gradient information of each network layer in the training group, as well as the current scaling factor and precision format of the training group, the gradient overflow risk of the training group is predicted. Based on the prediction result, the scaling factor of the next round of training for the training group is predicted to ensure that the current gradient information of the network layer does not exceed the numerical range corresponding to the precision format during the training of the target model, thereby improving the training stability and convergence efficiency of the target model.

[0061] In one optional embodiment, the model training method further includes, after the next round of training of the target model is completed, determining the next round gradient information of each network layer in the target model according to the next round scaling factor of each training group, and repeating the above operation until the target model meets the model convergence condition, and determining that the target model training is completed.

[0062] Similar to S201 above, if the target model does not meet the convergence condition after the next round of training, then for each training group, the gradient information of each network layer in the target model is determined based on the scaling factor of that training group in the next round of training. Since the current round of training of the target model is considered the next round of training after the next round of training has ended, the gradient information of each network layer in the target model obtained above is used as the gradient information of each network layer in the target model in the current round, and the scaling factor of each training group in the next round of training is used as the scaling factor of each training group in the current round, thus returning to S202 above. This process is repeated until the target model finishes training in a subsequent round and meets the convergence condition. In this case, it is not necessary to repeat S202; instead, the target model training is directly determined to be complete, resulting in a trained target model that meets the application requirements, which can then be used in subsequent application scenarios.

[0063] Taking the first, second, and third rounds of training as an example, the first round of training for the target model is performed. After the first round of training of the target model is completed, the first-round gradient information of each network layer in the target model is determined based on the initial scaling factor of different training groups in the first round of training. Based on the first-round gradient information of each network layer in the target model, as well as the initial scaling factor and precision format of each training group, the second-round scaling factor of each training group in the second round of training is predicted. The second round of training for the target model is performed. After the second round of training of the target model is completed, the second-round gradient information of each network layer in the target model is determined based on the second-round scaling factor of each training group. Based on the second-round gradient information of each network layer in the target model, as well as the second-round scaling factor and precision format of each training group, the third-round scaling factor of each training group in the third round of training is predicted. The third round of training for the target model is performed, and so on, until the Nth (e.g., 10000th) round of training of the target model is received, at which point the target model meets the model convergence condition.

[0064] Optionally, in response to training instructions for the target model, after the current round of training for the target model has ended, a new scaling factor adjustment class (such as the AutoGradScaler class) is called through a fully compatible API (Application Programming Interface) within the mixed-precision training framework. This allows the model training method to be implemented using methods within the scaling factor adjustment class, such as scale(), step(), and update(). This new scaling factor adjustment class inherits from or mimics the original scaling factor adjustment class (such as the GradScaler class) in related technologies. Therefore, only the class name of the original scaling factor adjustment class in the mixed-precision training framework needs to be replaced with the new scaling factor adjustment class name, without modifying other training code. This achieves a transparent integration design for the mixed-precision training framework, simplifying user operations and improving model training efficiency.

[0065] In one optional embodiment, the model training method further includes, before the first round of training of the target model, identifying each network layer to be trained in the target model according to the structure of the target model; grouping different network layers according to the type and precision format of different network layers to obtain at least one independent training group; and configuring an independent initial scaling factor for each training group.

[0066] Before training the target model, it is necessary to first determine each training group within the target model and configure an independent initial scaling factor for each training group. Therefore, before the first round of training of the target model, the network layers to be trained in the target model are first identified based on the model's structure. This identification can be based on the training objective, training scenario, and application scenario of the target model. Provided that the identified network layers to be trained have trainable parameters, there are no restrictions on the type, number, or model parameters of the network layers to be trained. Optionally, based on the structure of the target model, the model layers are determined, and then network layers with trainable parameters (such as weights, bias matrices, etc.) are identified as the network layers to be trained. Optionally, depending on the training scenario of the target model, all or some network layers with trainable parameters in the target model are identified as the network layers to be trained. Optionally, based on the structure of the target model, network layers of the target type with trainable parameters are identified as the network layers to be trained. For example, at least one type of network layer in the target model, including embedding layers, convolutional layers, and fully connected layers, can be identified as network layers to be trained. Optionally, all or some network layers in the target model that have trainable parameters of the target type can be identified as network layers to be trained.

[0067] Among the identified network layers to be trained, different network layers of the same type can have different precision formats, and different network layers with the same precision format can also have different types. Furthermore, there can be different network layers with both different types and precision formats. Any training group can include one network layer to be trained, or it can include multiple network layers of the same type and precision format. That is, different network layers within the same network layer have the same type and precision format. Therefore, after identifying the network layers to be trained, the different network layers are grouped according to their type and precision format to obtain at least one pair of training groups. Based on this, the number of network layers included in different training groups can be the same or different. Optionally, each network layer can be divided into a training group; or different network layers with the same type and precision format can be divided into the same training group.

[0068] After obtaining each training group, an independent initial scaling factor is configured for each training group. The initial scaling factors for different training groups can be the same or different. The initial scaling factor for each training group can be configured based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in practical applications; no specific limitations are imposed on this. Optionally, the initial scaling factor for each training group is the upper limit of the numerical range corresponding to the accuracy format of each network layer in that training group.

[0069] Optionally, the target model is an object detection model, and different training groups include an embedding layer training group, a convolutional layer training group, and a fully connected layer training group. The model training method then further includes: identifying the embedding layer, convolutional layer, and fully connected layer in the target model based on its structure; grouping the identified embedding layers with the same accuracy format into the same embedding layer training group; grouping the identified convolutional layers with the same accuracy format into the same convolutional layer training group; grouping the identified fully connected layers with the same accuracy format into the same fully connected layer training group; and configuring independent initial scaling factors for each of the embedding layer training group, convolutional layer training group, and fully connected layer training group.

[0070] The target model can be a target detection model for various types of inputs, such as an image target detection model (input is an image), a text target detection model (input is text), or a speech target detection model (input is audio information), etc., without specific limitations. Since target detection models typically include embedding layers, convolutional layers, and fully connected layers, the embedding layers, convolutional layers, and fully connected layers in the target model can be identified based on its structure and used as the network layers to be trained.

[0071] Optionally, the target model can also be other models that include embedding layers, convolutional layers, and fully connected layers, such as speech recognition models, large language models, information recommendation models, classification models, etc. There are no specific limitations on this.

[0072] Optionally, for network layers of the same type (each embedding layer, each convolutional layer, or each fully connected layer), if the accuracy format of each network layer of the same type is the same, all network layers of the same type are divided into the same training group to obtain a training group for network layers of that type (embedding layer training group, convolutional layer training group, or fully connected layer training group); if the accuracy formats of different network layers of the same type are not completely the same, network layers of the same type with the same accuracy format are divided into the same training group to obtain multiple training groups for network layers of that type.

[0073] In this way, for object detection models that include embedding layers, convolutional layers, and fully connected layers, the most likely number of network layers with the same type and precision format can be grouped into the same training group. This allows for differentiated adjustment of scaling factors for different training groups to accommodate the significantly different gradient distribution characteristics of network layers in different training groups. As a result, the scaling factors of different training groups are more targeted, and the risk of gradient overflow in each training group is significantly reduced. At the same time, the number of training groups is minimized, which improves the stability of the model during mixed-precision training and also improves the convergence efficiency of the model during mixed-precision training.

[0074] In the aforementioned model training method, network layers of the same type and precision format are pre-divided into different training groups based on the structure of the target model, and independent initial scaling factors are configured for each training group. Then, during the multi-round training of the target model, at the end of the current round, the current round gradient information of each network layer is determined based on the current round scaling factors of different training groups. Based on the current round gradient information, the current round scaling factors and precision formats of each training group, the scaling factors for the next round of training are predicted. Thus, by grouping the network layers in the target model and configuring independent scaling factors for different training groups, the scaling factors of different training groups can be independently adjusted based on the gradient information of the network layers in different training groups during the training process. This allows for differentiated adjustment of the scaling factors for different training groups, adapting to the significantly different gradient distribution characteristics of the network layers in different training groups. This enhances the specificity of the scaling factors for different training groups, significantly reduces the risk of gradient overflow in each training group, and improves the stability and convergence efficiency of the model during mixed-precision training.

[0075] Based on the above embodiments, in an exemplary embodiment, the determination of the current round gradient information in S201 is further refined; optionally, as... Figure 3 As shown, it includes:

[0076] S301, scale the loss value of the target model in the current round of training according to the scaling factor of each training group to obtain the scaling loss value corresponding to each training group.

[0077] During model training, at the end of each training round, the loss value of the model obtained in that round is usually small. If the loss value of that round is directly used, the gradient information of each network layer in the model in that round is usually also small, which leads to the underflow of the gradient information in the above round, that is, it is lower than the lower limit of the numerical range corresponding to the precision format of each network layer, resulting in the interruption of model training.

[0078] Therefore, to avoid underflow of gradient information in the network layers of the target model due to a small loss value in the current training round, and thus to prevent training interruption, after the current training round of the target model ends, for each training group, the loss value of the target model in the current training round is scaled according to the scaling factor of that training group to obtain the scaled loss value corresponding to that training group. Since the scaling factors of different training groups can be different, the scaled loss values ​​for different training groups can also be different, achieving differentiation of the scaled loss values ​​for different training groups. Optionally, if the scaling factor of each training group is greater than 1, then the product of the scaling factor of the current training round and the loss value of the target model in the current training round is determined as the scaled loss value corresponding to that training group; that is, scaling the loss value is essentially amplifying the loss value.

[0079] Optionally, in the current round of training of the target model, the training samples are input into the target model to obtain the prediction results of the target model for the training samples after performing feature learning and feature analysis on the training samples. Based on the difference between the above prediction results and the labels (i.e., the true values) of the training samples, the training loss value (Loss) of the target model in the current round of training is determined, and the above training loss value is determined as the loss value of the target model in the current round of training.

[0080] For example, taking an image detection model, a sample image with a ground truth bounding box labeled with the object to be detected is input into the target model. The target model outputs a predicted bounding box for the object. The training loss value of the target model is then determined based on the positional difference between the ground truth bounding box and the predicted bounding box. Similarly, in a speech recognition model, a sample speech with corresponding ground truth text is input into the target model. The target model outputs a predicted text for the sample speech. The training loss value of the target model is then determined based on the similarity between the ground truth text and the predicted text. Finally, in an information recommendation model, sample user information with corresponding ground truth recommendation results is input into the target model. The target model outputs a predicted recommendation result for the sample user information. The training loss value of the target model is then determined based on the similarity between the ground truth recommendation result and the predicted recommendation result.

[0081] S302, based on the scaling loss value and the parameter information of each network layer in each training group, determine the current round gradient information of each network layer in the target model.

[0082] Furthermore, after obtaining the scaling loss value corresponding to each training group, for each network layer, the current round gradient information of that network layer is determined based on the scaling loss value corresponding to the training group to which that network layer belongs and the parameter information of that network layer. As mentioned earlier, by differentiating the scaling loss values ​​corresponding to different training groups, the current round gradient information of the network layers in different training groups can be differentiated. Thus, by differentiating the current round gradient information of the network layers in different training groups, the scaling factor for the next round of different training groups can be adjusted differently. Optionally, the parameter information of each network layer includes the trainable parameters in each network layer.

[0083] In this embodiment, the scaling loss value corresponding to different training groups is independently determined by the scaling factor of the current round of different training groups, thereby achieving the differentiation of the scaling loss value corresponding to different training groups. Thus, the differentiation of the scaling loss value leads to the differentiation of the gradient information of the current round, thereby achieving the differentiated adjustment of the scaling factor of different training groups and improving the stability and convergence efficiency of the model during mixed precision training.

[0084] Based on the above embodiments, in an exemplary embodiment, the determination of the loss value for the current training round in S302 is further refined; optionally, as... Figure 4 As shown, it includes:

[0085] S401, Based on the prediction results of the target model for the training samples and the labels of the training samples in the current round of training, determine the training loss value of the target model.

[0086] As mentioned earlier, in the current round of training of the target model, the training samples are input into the target model to obtain the prediction results of the target model for the training samples by performing feature learning and feature analysis on the training samples. Based on the difference between the above prediction results and the labels (i.e., the true values) of the training samples, the training loss value (Loss) of the target model in the current round of training is determined.

[0087] S402, for each training group, determine the intermediate loss value corresponding to the training group based on the training loss value, the current round scaling factor of the training group, and the preset weights.

[0088] Since the network layers in the target model are divided into multiple training groups, and different training groups contain different types of network layers and have different precision formats, their contributions to the training loss value vary. Therefore, based on the contribution of each training group to the training loss value, a preset weight is assigned to each training group in advance. The training group that contributes more to the training loss value has a higher preset weight.

[0089] Therefore, for each training group, the intermediate loss value can be determined based on the aforementioned training loss value, the current round scaling factor of the training group, and the preset weights. Optionally, for each training group, the product of the aforementioned training loss value, the current round scaling factor of the training group, and the preset weights of the training group can be calculated as the intermediate loss value for that training group.

[0090] S403: Determine the loss value for the current training round based on the sum of the intermediate loss values ​​corresponding to different training groups.

[0091] After obtaining the intermediate loss values ​​for different training groups, the sum of the intermediate loss values ​​for different training groups is determined. Based on this sum, the loss value of the target model in the current training round is determined. Optionally, the sum is used as the loss value of the target model in the current training round; or the product of the sum and a preset coefficient is calculated, and the calculated product is used as the loss value of the target model in the current training round. The preset coefficient can be set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in practical applications, and is not specifically limited in this regard.

[0092] Optionally, the loss value of the target model in the current training round can be determined by the following formula:

[0093]

[0094] in, This represents the loss value of the target model during the current training round. This represents the training loss value of the target model in the current training round. For the first Preset weights for each training group For the first The current round scaling factor for each training group This represents the total number of training groups.

[0095] In this embodiment, the training loss value of the target model in the current training round is weighted and summed according to the current round scaling factor and preset weight of different training groups to obtain the training loss value of the target model in the current training round. This can balance the contribution of different training groups to the above training loss value, improve the stability of the target model training, and guide the target model to converge, thereby improving the convergence efficiency of the target model training.

[0096] Based on the above embodiments, in an exemplary embodiment, the prediction of the scaling factor in S202 is further refined; optionally, as... Figure 5 As shown, it includes:

[0097] S501, for any training group, predicts the gradient overflow risk of the next round of the training group based on the numerical range corresponding to the scaling factor and precision format of the current round of the training group, as well as the gradient information of each network layer in the current round of the training group.

[0098] The risk of gradient overflow in the next round refers to the risk of underflow and overflow of the gradient information of each network layer in the target model after the next round of training. Overflow refers to exceeding the upper limit of the numerical range corresponding to the precision format, while underflow refers to exceeding the lower limit of the numerical range corresponding to the precision format.

[0099] To ensure the stability of the target model training and avoid training interruption due to gradient overflow or underflow of network layer information, after the current round of training of the target model has ended, the risk of gradient overflow of the gradient information of each network layer in the next round of training is predicted. That is, it is predicted whether there is a risk of gradient overflow in the gradient information of each network layer in the next round of training. Optionally, it is predicted whether there is a risk of underflow and overflow of the gradient information of each network layer in the next round of training.

[0100] Optionally, for any training set, an upper and lower threshold value is determined based on the numerical range corresponding to the precision format of each network layer in the training set. Then, based on the scaling factor of the current round of the training set and the gradient information of each network layer in the current round, the upper and lower predicted values ​​of the gradient information of each network layer in the next round are predicted. By comparing the predicted upper and upper threshold values, the risk of overflow of the gradient information of each network layer in the next round is determined. Similarly, by comparing the predicted lower and lower threshold values, the risk of underflow of the gradient information of each network layer in the next round is determined. Specifically, if the predicted upper value is greater than the upper threshold value, the risk of overflow of the gradient information of each network layer in the next round is determined; and if the predicted lower threshold value is less than the lower threshold value, the risk of underflow of the gradient information of each network layer in the next round is determined.

[0101] S502, based on the gradient overflow risk of the training group in the next round, adjust the scaling factor of the current round of the training group to obtain the scaling factor of the training group in the next round of training.

[0102] For any training set, after predicting the gradient overflow risk for the next round, the scaling factor for the next training round can be predicted based on the prediction result and the scaling factor of the current round. Specifically, the scaling factor of the current round can be adjusted (e.g., reduced or increased) based on the gradient overflow risk (either gradient overflow or gradient underflow risk). The adjusted scaling factor is then the scaling factor for the next round of training.

[0103] Specifically, if the prediction results indicate that the training group has a risk of underflow (i.e., the risk of gradient overflow in the next round is the risk of gradient underflow in the next round), then the current round scaling factor of the training group is considered to be small. In this case, the current round scaling factor of the training group is increased, and the increased current round scaling factor is used as the scaling factor of the training group in the next round of training. Conversely, if the prediction results indicate that the training group has a risk of overflow (i.e., the risk of gradient overflow in the next round is the risk of gradient overflow in the next round), then the current round scaling factor of the training group is considered to be large. In this case, the current round scaling factor of the training group is decreased, and the decreased current round scaling factor is used as the scaling factor of the training group in the next round of training.

[0104] Optionally, if the prediction results indicate that the training group is at risk of underflow, the step size is increased according to the set parameters, thereby increasing the scaling factor of the current round of the training group; conversely, if the prediction results indicate that the training group is at risk of overflow, the step size is decreased according to the set parameters, thereby decreasing the scaling factor of the current round of the training group. Optionally, the increase in step size and the decrease in compensation may be the same or different.

[0105] Optionally, if the prediction results obtained above indicate that the training group has no risk of underflow and no risk of overflow, the scaling factor of the current round of the training group can be directly determined as the scaling factor of the next round of training; or the scaling factor of the current round of the training group can be increased according to the adjustment step size, and the increased scaling factor of the current round of the training group can be determined as the scaling factor of the next round of training; or the product of the adjustment coefficient and the scaling factor of the current round of the training group can be calculated as the scaling factor of the next round of training; wherein, the adjustment step size is less than the increase step size, and the adjustment coefficient is greater than 0 and less than 1; and the increase step size, decrease step size, adjustment step size and adjustment coefficient can all be set according to empirical values, experimental values ​​from multiple trials and the accuracy requirements of the target model in actual applications, etc., and no specific limitations are imposed on them.

[0106] In this embodiment, by using the next-round gradient overflow risk prediction mechanism, the scaling factor of each training group is adjusted in advance based on the prediction result of the next-round gradient overflow risk. This preventive adjustment of the scaling factor of each training group can reduce the recalculation caused by abnormal gradient information of each network layer, save computing power for training the target model, and improve the training speed and convergence efficiency of the target model.

[0107] Based on the above embodiments, in an exemplary embodiment, the prediction of the next-round gradient overflow risk in S501 is further refined; optionally, the next-round gradient overflow risk includes the next-round gradient overflow risk and the next-round gradient underflow risk, such as... Figure 6 As shown, it includes:

[0108] S601 determines the mean gradient and maximum absolute gradient of the training group based on the current gradient information of each network layer in the training group.

[0109] For any training set, the mean gradient and maximum absolute gradient are first determined based on the current gradient information of each network layer in the training set. The mean gradient characterizes the average update trend and intensity of the overall model parameters of each network layer in the training set, reflecting the average adjustment magnitude of the model parameters in the adjustment direction. The maximum absolute gradient characterizes the extreme fluctuation amplitude of the gradient of the model parameters of each network layer in the training set, reflecting the maximum impact intensity and gradient divergence risk of the model parameter updates in the training set.

[0110] Optionally, for any training group, the arithmetic mean of the current round gradient information of each network layer in the training group is determined as the gradient mean of the training group; or, based on the weights of different network layers in the training group, the current round gradient information of each network layer in the training group is weighted and summed, and the resulting sum is determined as the gradient mean of the training group.

[0111] In one optional embodiment, the current round gradient information of each network layer is represented by the current round gradient matrix. Then, S601 includes determining the average value of each element in the current round gradient matrix of each network layer in the training group as the average gradient value of the training group; and selecting the maximum value among the absolute values ​​of each element in the current round gradient matrix of each network layer in the training group as the maximum absolute gradient value of the training group.

[0112] Specifically, for any training group, the average value of each element in the current round gradient matrix of all network layers in the training group is determined as the average gradient of the training group. For example, if the current round gradient matrix is ​​an N×M matrix and the training group includes m (m≥1) network layers, the sum S of all element values ​​(m×N×M) in the above m N×M matrices is calculated, and then the ratio S / (m×N×M) is calculated to obtain the average gradient of the training group. Correspondingly, the absolute value of each element value (m×N×M) is determined, and the maximum value among the above (m×N×M) absolute values ​​is selected as the maximum absolute gradient of the training group.

[0113] In this way, by utilizing the current round gradient matrix of each network layer in each training group, the mean gradient and maximum absolute gradient of that training group are determined. This fully considers the gradient distribution characteristics of different network layers in that training group, so that the mean gradient of that training group can better reflect the overall gradient strength of that training group, and the maximum absolute gradient of that training group can better reflect the gradient extreme risk of that training group, thereby improving the stability of the target model training.

[0114] S602 predicts the upper limit of the future gradient of the training group based on the product of the maximum absolute value of the gradient and the scaling factor of the current round of the training group, and predicts the risk of gradient overflow in the next round of the training group based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group.

[0115] For any training set, the upper limit of the future gradient corresponding to the training set is used to characterize the maximum value that may be reached in the next round of gradient information of each network layer in the training set.

[0116] Optionally, for any training group, the product of the maximum absolute value of the gradient corresponding to that training group and the scaling factor of the current round of that training group is calculated as the upper limit of the future gradient corresponding to that training group; or the first product of the maximum absolute value of the gradient corresponding to that training group and the scaling factor of the current round of that training group is calculated, and the second product of the first product and the first preset value is calculated, so that the second product is used as the upper limit of the future gradient corresponding to that training group. The first preset value can be a value greater than 1 set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in practical applications, and is not specifically limited in this regard.

[0117] In one optional embodiment, S602 includes determining that the gradient overflow risk characterization of the training group in the next round exists if the future gradient upper limit value is greater than a preset upper limit threshold; wherein the preset upper limit threshold is determined by multiplying the upper limit value of the numerical range corresponding to the precision format of the training group with a first coefficient, and the first coefficient is greater than 0 and less than 1.

[0118] In this embodiment, a first coefficient is predetermined. The first coefficient can be a value greater than 0 and less than 1, set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in actual applications. There is no specific limitation on this value. For example, the first coefficient is 0.9.

[0119] For any training group, a preset upper limit threshold is determined in advance based on the product of the upper limit of the precision format of each network layer in the training group and the aforementioned first coefficient. The precision format of the training group is the precision format of each network layer in that training. Optionally, the preset upper limit threshold is determined by the product of the upper limit of the numerical range corresponding to the precision format of each network layer in the training group and the first coefficient.

[0120] If the upper limit of the future gradient corresponding to the training group is greater than the preset upper limit threshold of the training group, it can be said that the maximum value that the gradient information of each network layer in the training group may reach in the next round is likely to be greater than the upper limit of the numerical range corresponding to the precision format of the training group. Therefore, it can be determined that there is a risk of overflow of the gradient information of each network layer in the training group in the next round, and thus it can be determined that the gradient overflow risk of the training group in the next round is characterized by the risk of gradient overflow in the next round.

[0121] Optionally, for the i-th training group, if the precision format of the i-th training group is FP16 and the first coefficient is 0.9, the upper limit of the numerical range corresponding to the precision format of the i-th training group is 65504. Then, if the following formula holds, it is determined that the gradient overflow risk of the training group in the next round indicates that there is a gradient overflow risk in the next round: max_abs_i × scale_i > 0.9 × 65504; where max_abs_i is the maximum absolute value of the gradient corresponding to the i-th training group, and scale_i is the scaling factor of the current round of the i-th training group.

[0122] In this way, the preset upper limit threshold of each training group is determined by multiplying the upper limit of the numerical range corresponding to the precision format of each training group with the first coefficient. By using the independent preset upper limit thresholds of different training groups, differentiated risk prediction for different training groups can be achieved. This adapts to the gradient distribution characteristics of significantly different network layers in different training groups, significantly reduces the gradient overflow risk of each training group, and improves the stability and convergence efficiency of the model during mixed precision training.

[0123] S603 predicts the future gradient lower bound of the training group based on the product of the gradient mean and the scaling factor of the current round of the training group, and predicts the gradient underflow risk of the training group in the next round based on the future gradient lower bound and the lower bound of the numerical range.

[0124] For any training set, the lower bound of the future gradient corresponding to the training set is used to characterize the minimum value that may be reached in the next round of gradient information of each network layer in the training set.

[0125] Optionally, for any training group, the product of the mean gradient of that training group and the scaling factor of the current round of that training group is calculated as the lower bound of the future gradient for that training group; or the third product of the mean gradient of that training group and the scaling factor of the current round of that training group is calculated, and the fourth product of the third product and the second preset value is calculated, so that the fourth product is used as the lower bound of the future gradient for that training group. The second preset value can be a value greater than 1 set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in practical applications, and is not specifically limited in this regard.

[0126] In one optional embodiment, S603 includes determining that the gradient overflow risk of the training group in the next round exists if the future gradient lower bound value is less than a preset lower bound threshold; wherein the preset lower bound threshold is determined based on the product of the lower bound value of the numerical range and a second coefficient, and the second coefficient is greater than 1.

[0127] In this embodiment, a second coefficient is predetermined. The second coefficient can be a value greater than 1 set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in actual applications. There is no specific limitation on this. For example, the first coefficient is 100.

[0128] For any training group, a preset lower limit threshold is determined in advance based on the product of the lower limit of the precision format of each network layer in the training group and the aforementioned second coefficient. Optionally, the preset lower limit threshold is determined by the product of the lower limit of the numerical range corresponding to the precision format of each network layer in the training group and the second coefficient.

[0129] If the lower bound of the future gradient corresponding to the training group is less than the preset upper bound threshold of the training group, it can be said that the minimum value that the gradient information of each network layer in the training group may reach in the next round is likely to be less than the lower bound of the numerical range corresponding to the precision format of the training group. Therefore, it can be determined that the gradient information of each network layer in the training group is at risk of underflow, and thus the gradient overflow risk of the training group is characterized by the risk of gradient underflow in the next round.

[0130] Optionally, for the i-th training group, if the precision format of the i-th training group is FP16 and the second coefficient is 100, the lower limit of the numerical range corresponding to the precision format of the i-th training group is 5.96e-8. Then, if the following formula holds, it is determined that the gradient overflow risk of the training group in the next round indicates that there is a gradient underflow risk in the next round: mean_i × scale_i < 100 × 5.96e-8; where mean_i is the gradient mean corresponding to the i-th training group, and scale_i is the scaling factor of the i-th training group in the current round.

[0131] In this way, the preset lower limit threshold of each training group is determined by multiplying the lower limit of the numerical range corresponding to the precision format of each training group with the second coefficient. By utilizing the independent preset upper limit thresholds of different training groups, differentiated risk prediction for different training groups can be achieved. This adapts to the gradient distribution characteristics of significantly different network layers in different training groups, significantly reduces the gradient underflow risk of each training group, and improves the stability and convergence efficiency of the model during mixed precision training.

[0132] There is no restriction on the execution order of S602 and S603, as long as S602 and S603 are executed after S601 is completed. That is, S602 can be executed first and then S603; S603 can be executed first and then S602; or S602 and S603 can be executed simultaneously.

[0133] In this embodiment, the gradient mean and maximum average gradient of each training group are used to predict the gradient overflow risk in the next round. Preset upper and lower thresholds are configured independently for different training groups to predict the gradient overflow risk and gradient underflow risk in the next round for different training groups. This enables differentiated risk prediction for different training groups, adapting to the significantly different gradient distribution characteristics of network layers in different training groups, significantly reducing the gradient underflow risk of each training group, and improving the stability and convergence efficiency of the model during mixed precision training.

[0134] Based on the above embodiments, in an exemplary embodiment, the determination of the scaling factor in S502 is further refined; optionally, it includes the following steps:

[0135] Step 1: If there is a risk of gradient overflow in the next round, reduce the scaling factor of the current round of the training group according to the first value to obtain the scaling factor of the next round of training; wherein the first value is greater than 0 and less than 1.

[0136] The first value is predetermined. The first value can be a value greater than 0 and less than 1, which is set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in actual applications. There is no specific limitation on this value. For example, the first value is 0.5.

[0137] If, for any training set, there is a risk of gradient overflow in the next round of training, it indicates that the maximum value that the gradient information of each network layer in the next round may reach is likely to exceed the upper limit of the numerical range corresponding to the precision format of the training set. Therefore, it is necessary to reduce the scaling factor of the next round of training. This can be done by reducing the scaling factor of the current round of training based on the first value, and then determining the reduced scaling factor of the current round as the scaling factor of the next round of training. Optionally, the product of the first value and the scaling factor of the current round of training can be calculated as the scaling factor of the next round of training.

[0138] Step 2: If there is a risk of gradient underflow in the next round, the scaling factor of the current round of the training group is amplified according to the second value to obtain the scaling factor of the next round of training; wherein the second value is greater than 1.

[0139] The second value is predetermined. The second value can be a value greater than 1, which can be set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in actual applications. There is no specific limitation on this value. For example, the second value is 1.1.

[0140] For any training set, if the gradient overflow risk representation for the next round of training sets indicates a risk of gradient underflow, it means that the minimum possible value of the gradient information of each network layer in the next round of training sets is likely to be less than the lower limit of the numerical range corresponding to the precision format of the training set. Therefore, it is necessary to increase the scaling factor for the next round of training sets. This can be done by amplifying the scaling factor of the current round of training sets based on the second value, and then determining the amplified current round scaling factor as the scaling factor for the next round of training. Optionally, the product of the second value and the current round scaling factor of the training set can be calculated as the scaling factor for the next round of training.

[0141] Step 3: If there is no gradient overflow risk or gradient underflow risk in the next round, predict the scaling factor of the training group in the next round of training based on the consecutive occurrences of the gradient overflow risk in the next round when there is no gradient overflow risk or gradient underflow risk.

[0142] For any training set, if there is no gradient overflow or underflow risk in the next round of gradient overflow risk representation, it indicates that the gradient information of each network layer in the training set is likely to be within the numerical range corresponding to the precision format of the training set. Therefore, it can be determined that the scaling factor of the current round of the training set is a scaling factor that is well-suited to each network layer in the training set. Thus, in order to further improve the fit between the scaling factor of the training set and each network layer in the training set, and to find the most suitable scaling factor, the scaling factor of the next round of training can be predicted based on the consecutive rounds in which there is no gradient overflow or underflow risk.

[0143] Optionally, based on the number of consecutive rounds where there is no risk of gradient overflow or gradient underflow in the next round, it can be determined whether to continue using the current round scaling factor of the training group as the next round scaling factor for the training group, or to appropriately increase the next round scaling factor of the training group.

[0144] In one optional embodiment, step 3 above includes, when the number of consecutive training rounds without gradient overflow risk or gradient underflow risk reaches a preset number, scaling factor of the current round of the training group is amplified according to a third value to obtain the scaling factor of the training group in the next round of training; wherein, the third value is greater than 1 and less than the second value, and the consecutive preset number of training rounds include the current round of training.

[0145] The third value is predetermined. The third value can be a value greater than 1 and less than the second value, which is set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in actual applications. There is no specific limitation on this value. For example, if the second value is 1.1, the third value can be 1.05.

[0146] For any training set, if, during a predetermined number of consecutive training rounds (including the current round), the predicted gradient overflow risk for the next round is zero (meaning there is no risk of gradient overflow or underflow), then to maximize the gradient information of each network layer in the training set without causing overflow or underflow, the scaling factor for the next round of training can be appropriately increased. Based on this, the scaling factor for the current round of training can be amplified according to a third value, and this amplified scaling factor is determined as the scaling factor for the next round of training. Optionally, the product of the third value and the scaling factor for the current round of training can be calculated as the scaling factor for the next round of training.

[0147] Optionally, if the number of consecutive training rounds without gradient overflow risk or gradient underflow risk in the next round is less than a preset number, the scaling factor of the current round of the training group is determined as the scaling factor of the next round of training for the training group.

[0148] For any training group, if the number of consecutive training rounds in which the predicted gradient overflow risk for the next round is not less than a preset number (i.e., the number of consecutive training rounds in which the predicted gradient overflow risk is not greater than the preset number), then the scaling factor of the training group can be temporarily left unchanged to avoid excessive scaling factor adjustments causing overflow or underflow. In this case, the scaling factor of the current round of the training group can be directly determined as the scaling factor of the next round of training.

[0149] The aforementioned preset number can be set based on empirical values, experimental values ​​from multiple trials, and the accuracy requirements of the target model in practical applications, without specific limitations. For example, the preset number could be 3. For instance, assuming that after the first, second, and third rounds of training, the gradient overflow risk for a certain training group indicates the absence of gradient overflow or underflow risk in the next round, then the initial scaling factor of the training group from the first round of training is used as the scaling factor for the second and third rounds, and the scaling factor for the fourth round is determined based on the third value and the initial scaling factor.

[0150] Thus, for any training group, based on the consecutive occurrences of gradient overflow risk in the next round where there is no risk of gradient overflow or underflow, the scaling factor for the next round of the training group is predicted. This can maximize the gradient information of each network layer in the training group without overflow or underflow, and avoid excessive adjustment of the scaling factor that could cause overflow or underflow, thereby improving the accuracy of the obtained scaling factor for the next round of the training group.

[0151] In this embodiment, based on the different predicted gradient overflow risks of each training group in the next round, the scaling factor of each training group in the next round of training is predicted in a targeted manner. This can improve the adaptability of the predicted scaling factor of each training group in the next round of training with each network layer in each training group, thereby improving the accuracy of the predicted scaling factor of each training group in the next round. This significantly reduces the gradient overflow risk of each training group and improves the stability and convergence efficiency of the model during mixed precision training.

[0152] Based on the above embodiments, in an exemplary embodiment, such as Figure 7 As shown, the model training method includes:

[0153] S701 identifies the network layers to be trained in the target model based on its structure before the first round of training of the target model.

[0154] S702, based on the type and precision format of different network layers, groups different network layers to obtain at least one independent training group, and configures an independent initial scaling factor for each training group.

[0155] S703, when the current round of training of the target model has ended, determine the training loss value of the target model based on the prediction results of the target model for the training samples and the labels of the training samples in the current round of training, and for each training group, determine the corresponding intermediate loss value of the training group based on the training loss value, the current round scaling factor of the training group and the preset weight.

[0156] S704 determines the loss value for the current training round based on the sum of the intermediate loss values ​​corresponding to different training groups.

[0157] S705: Scale the loss value of the target model in the current round of training according to the scaling factor of each training group to obtain the scaling loss value corresponding to each training group. And determine the current round gradient information of each network layer in the target model according to the scaling loss value and the parameter information of each network layer in each training group.

[0158] S706: For any network layer in the target model, determine the ratio of the current round gradient information of the network layer to the current round scaling factor of the training group to which the network layer belongs, and adjust the model parameters of the network layer according to the ratio.

[0159] S707, for any training group, determine the average value of each element in the current round gradient matrix of each network layer in the training group as the average gradient of the training group; and select the maximum value among the absolute values ​​of each element in the current round gradient matrix of each network layer in the training group as the maximum absolute value of the gradient of the training group.

[0160] In this context, the gradient information of the current round of each network layer is represented by the gradient matrix of the current round;

[0161] S708 predicts the upper limit of the future gradient of the training group based on the product of the maximum absolute value of the gradient and the scaling factor of the current round of the training group. If the upper limit of the future gradient is greater than the preset upper limit threshold, it determines that there is a risk of gradient overflow in the next round of the training group.

[0162] The preset upper limit threshold is determined by the product of the upper limit of the numerical range corresponding to the precision format of the training group and the first coefficient, where the first coefficient is greater than 0 and less than 1.

[0163] S709: Based on the product of the gradient mean and the scaling factor of the current round of the training group, predict the future gradient lower bound value of the training group, and if the future gradient lower bound value is less than the preset lower bound threshold, determine that the gradient overflow risk of the training group in the next round indicates that there is a risk of gradient underflow in the next round.

[0164] The preset lower threshold is determined by multiplying the lower limit of the numerical range with the second coefficient, where the second coefficient is greater than 1.

[0165] S710: In the case that there is a risk of gradient overflow in the next round, the scaling factor of the current round of the training group is reduced according to the first value to obtain the scaling factor of the next round of training.

[0166] The first value is greater than 0 and less than 1;

[0167] S711, in the case that there is a risk of gradient underflow in the next round, the scaling factor of the current round of the training group is amplified according to the second value to obtain the scaling factor of the next round of training.

[0168] The second value is greater than 1;

[0169] S712, if the number of consecutive training rounds without gradient overflow risk or gradient underflow risk reaches a preset number, the scaling factor of the current round of the training group is amplified according to the third value to obtain the scaling factor of the next round of training for the training group; if the number of consecutive training rounds without gradient overflow risk or gradient underflow risk is less than a preset number, the scaling factor of the current round of the training group is determined as the scaling factor of the next round of training for the training group.

[0170] Among them, the third value is greater than 1 and less than the second value, and the number of consecutive preset training rounds includes the current training round.

[0171] S713, after the next round of training of the target model is completed, determine the next round gradient information of each network layer in the target model according to the next round scaling factor of each training group, and repeat the above operation until the target model meets the model convergence condition, and determine that the target model training is completed.

[0172] The specific implementation methods of S701-S713 are the same as those in the above method embodiments, and will not be repeated here.

[0173] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0174] Based on the same inventive concept, this application also provides a model training apparatus for implementing the model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more model training apparatus embodiments provided below can be found in the limitations of the model training method described above, and will not be repeated here.

[0175] In one exemplary embodiment, such as Figure 8 As shown, a model training apparatus is provided, including: a gradient determination module 810 and a factor prediction module 820, wherein:

[0176] The gradient determination module 810 is used to determine the gradient information of each network layer in the target model in the current round of training based on the scaling factor of the current round of training of different training groups in the target model when the current round of training of the target model has ended. The different training groups are network layers of the same type and precision format according to the structure of the target model, and the different training groups are configured with independent initial scaling factors.

[0177] The factor prediction module 820 is used to predict the scaling factor of each training group in the next training round based on the gradient information of the current round, as well as the scaling factor and precision format of each training group in the current round.

[0178] In an exemplary embodiment, the gradient determination module 810 is further configured to determine the next round gradient information of each network layer in the target model according to the next round scaling factor of each training group when the next round of training of the target model is completed, and trigger the factor prediction module 820 until the target model meets the model convergence condition and the target model training is determined to be completed.

[0179] In an exemplary embodiment, the gradient determination module 810 is specifically used to: scale the loss value of the target model in the current round of training according to the scaling factor of each training group to obtain the scaling loss value corresponding to each training group; and determine the current round gradient information of each network layer in the target model according to the scaling loss value and the parameter information of each network layer in each training group.

[0180] In one exemplary embodiment, the model training apparatus further includes a loss value determination module, wherein:

[0181] The loss value determination module is used to determine the training loss value of the target model based on the prediction results of the target model for the training samples and the labels of the training samples in the current training round; for each training group, it determines the intermediate loss value corresponding to the training group based on the training loss value, the current round scaling factor of the training group and the preset weights; and determines the loss value of the current training round based on the sum of the intermediate loss values ​​corresponding to different training groups.

[0182] In an exemplary embodiment, the factor prediction module 820 includes a risk prediction submodule and a factor prediction submodule, wherein:

[0183] The risk prediction submodule is used to predict the gradient overflow risk of the next round of the training group for any training group, based on the numerical range corresponding to the scaling factor and precision format of the current round of the training group, as well as the gradient information of each network layer in the current round of the training group.

[0184] The factor prediction submodule is used to adjust the scaling factor of the current round of the training group based on the gradient overflow risk of the next round, so as to obtain the predicted scaling factor of the training group in the next round of training.

[0185] In an exemplary embodiment, the next round gradient overflow risk includes the next round gradient overflow risk and the next round gradient underflow risk; the risk prediction submodule includes a gradient determination unit, a first prediction unit, and a second prediction unit, wherein:

[0186] The gradient determination unit is used to determine the mean gradient and maximum absolute value of the gradient for each network layer in the training group based on the gradient information of the current round.

[0187] The first prediction unit is used to predict the upper limit of the future gradient of the training group based on the product of the maximum absolute value of the gradient and the scaling factor of the current round of the training group, and to predict the risk of gradient overflow in the next round of the training group based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group.

[0188] The second prediction unit is used to predict the future gradient lower bound of the training group based on the product of the gradient mean and the scaling factor of the current round of the training group, and to predict the gradient underflow risk of the training group in the next round based on the future gradient lower bound and the lower bound of the numerical range.

[0189] In an exemplary embodiment, the current round gradient information of each network layer is represented by the current round gradient matrix; the gradient determination unit is specifically used to: determine the average value of each element in the current round gradient matrix of each network layer in the training group as the average gradient value of the training group; and select the maximum value among the absolute values ​​of each element in the current round gradient matrix of each network layer in the training group as the maximum absolute value of the gradient of the training group.

[0190] In an exemplary embodiment, the first prediction unit is specifically configured to: determine that the gradient overflow risk representation of the training group in the next round exists in the next round if the upper limit value of the future gradient is greater than a preset upper limit threshold; wherein, the preset upper limit threshold is determined by multiplying the upper limit value of the numerical range corresponding to the precision format of the training group with a first coefficient, and the first coefficient is greater than 0 and less than 1.

[0191] In an exemplary embodiment, the second prediction unit is specifically configured to: determine that the gradient overflow risk representation of the training group in the next round exists in the next round if the future gradient lower bound value is less than a preset lower bound threshold; wherein, the preset lower bound threshold is determined based on the product of the lower bound value of the numerical range and a second coefficient, and the second coefficient is greater than 1.

[0192] In an exemplary embodiment, the factor prediction submodule includes a first prediction unit, a second prediction unit, and a third prediction unit, wherein:

[0193] The first prediction unit is used to reduce the scaling factor of the current round of the training group according to the first value when there is a risk of gradient overflow in the next round of gradient overflow risk representation, so as to obtain the scaling factor of the training group in the next round of training; wherein the first value is greater than 0 and less than 1.

[0194] The second prediction unit is used to amplify the current round scaling factor of the training group according to the second value when there is a risk of gradient underflow in the next round of gradient overflow risk representation, so as to obtain the next round scaling factor of the training group in the next round of training; wherein the second value is greater than 1;

[0195] The third prediction unit is used to predict the scaling factor of the training group in the next training round, based on the number of consecutive rounds in which the gradient overflow risk occurs when there is no gradient overflow risk or gradient underflow risk in the next round.

[0196] In an exemplary embodiment, the third prediction unit is specifically configured to: when the number of consecutive training rounds without gradient overflow risk or gradient underflow risk reaches a preset number, amplify the scaling factor of the current round of the training group according to a third value to obtain the scaling factor of the training group in the next round of training; wherein the third value is greater than 1 and less than the second value, and the consecutive preset number of training rounds include the current round of training.

[0197] In one exemplary embodiment, the model training apparatus further includes a parameter tuning module, wherein:

[0198] The parameter adjustment module is used to determine the ratio of the current gradient information of a network layer to the current scaling factor of the training group to which the network layer belongs before predicting the scaling factor of the next training round for each training group in the next training round based on the current gradient information, the current scaling factor and the precision format of each training group; and adjust the model parameters of the network layer according to the ratio.

[0199] In an exemplary embodiment, the target model is an object detection model, and different training groups include an embedding layer training group, a convolutional layer training group, and a fully connected layer training group; the model training device also includes a network layer identification module, a training group partitioning module, and a factor configuration module.

[0200] The network layer identification module is used to identify the embedding layers, convolutional layers, and fully connected layers in the target model based on its structure.

[0201] The training group partitioning module is used to partition embedded layers with the same accuracy format into the same embedded layer training group; to partition convolutional layers with the same accuracy format into the same convolutional layer training group; and to partition fully connected layers with the same accuracy format into the same fully connected layer training group.

[0202] The factor configuration module is used to configure independent initial scaling factors for the embedding layer training group, the convolutional layer training group, and the fully connected layer training group, respectively.

[0203] Each module in the aforementioned model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0204] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: when the current round of training of the target model has ended, determining the current round gradient information of each network layer in the target model based on the current round scaling factor of different training groups in the target model during the current round of training; wherein the different training groups are network layers of the same type and precision format divided according to the structure of the target model, and the different training groups are configured with independent initial scaling factors; and predicting the next round scaling factor of each training group in the next round of training based on the current round gradient information, as well as the current round scaling factor and precision format of each training group.

[0205] In one embodiment, when the processor executes the computer program, it further performs the following steps: when the next round of training of the target model is completed, it determines the next round gradient information of each network layer in the target model according to the next round scaling factor of each training group, and repeats the above operation until the target model meets the model convergence condition, and determines that the target model training is completed.

[0206] In one embodiment, when the processor executes the computer program, it further performs the following steps: scaling the loss value of the target model in the current round of training according to the scaling factor of each training group to obtain the scaling loss value corresponding to each training group; and determining the current round gradient information of each network layer in the target model according to the scaling loss value and the parameter information of each network layer in each training group.

[0207] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the training loss value of the target model based on the prediction result of the target model for the training samples and the labels of the training samples in the current round of training; determining the intermediate loss value corresponding to each training group based on the training loss value, the current round scaling factor of the training group, and the preset weights; and determining the loss value of the current round of training based on the sum of the intermediate loss values ​​corresponding to different training groups.

[0208] In one embodiment, when the processor executes the computer program, it further performs the following steps: for any training group, predicting the gradient overflow risk of the next round of the training group based on the numerical range corresponding to the current round scaling factor and precision format of the training group, and the current round gradient information of each network layer in the training group; adjusting the current round scaling factor of the training group based on the gradient overflow risk of the next round of the training group to obtain the next round scaling factor of the training group in the next round of training.

[0209] In one embodiment, the gradient overflow risk in the next round includes the gradient overflow risk and the gradient underflow risk in the next round. When the processor executes the computer program, it further implements the following steps: determining the mean gradient and maximum absolute gradient value corresponding to the training group based on the current round gradient information of each network layer in the training group; predicting the upper bound of the future gradient corresponding to the training group based on the product of the maximum absolute gradient value and the scaling factor of the current round of the training group, and predicting the gradient overflow risk in the next round of the training group based on the upper bound of the future gradient and the upper bound of the numerical range corresponding to the precision format of the training group; predicting the lower bound of the future gradient corresponding to the training group based on the product of the mean gradient and the scaling factor of the current round of the training group, and predicting the gradient underflow risk in the next round of the training group based on the lower bound of the future gradient and the lower bound of the numerical range.

[0210] In one embodiment, the current round gradient information of each network layer is represented by the current round gradient matrix; when the processor executes the computer program, it also implements the following steps: determining the average value of each element in the current round gradient matrix of each network layer in the training group as the average gradient value of the training group; and selecting the maximum value among the absolute values ​​of each element in the current round gradient matrix of each network layer in the training group as the maximum absolute value of the gradient of the training group.

[0211] In one embodiment, when the processor executes the computer program, it further implements the following steps: if the upper limit value of the future gradient is greater than a preset upper limit threshold, it determines that the gradient overflow risk representation of the training group in the next round has a risk of gradient overflow in the next round; wherein, the preset upper limit threshold is determined by the product of the upper limit value of the numerical range corresponding to the precision format of the training group and a first coefficient, and the first coefficient is greater than 0 and less than 1.

[0212] In one embodiment, when the processor executes the computer program, it further implements the following steps: if the future gradient lower bound value is less than a preset lower bound threshold, it determines that the gradient overflow risk characterization of the training group in the next round indicates that there is a risk of gradient underflow in the next round; wherein, the preset lower bound threshold is determined based on the product of the lower bound value of the numerical range and a second coefficient, and the second coefficient is greater than 1.

[0213] In one embodiment, when the processor executes the computer program, it further implements the following steps: If the gradient overflow risk representation for the next round indicates a gradient overflow risk in the next round, the scaling factor of the current round of the training group is reduced according to a first value to obtain the scaling factor of the training group in the next round of training; wherein the first value is greater than 0 and less than 1; If the gradient overflow risk representation for the next round indicates a gradient underflow risk in the next round, the scaling factor of the current round of the training group is increased according to a second value to obtain the scaling factor of the training group in the next round of training; wherein the second value is greater than 1; If the gradient overflow risk representation for the next round indicates neither a gradient overflow risk nor a gradient underflow risk in the next round, the scaling factor of the training group in the next round of training is predicted based on the consecutive rounds in which the gradient overflow risk does not exist or does not exist.

[0214] In one embodiment, when the processor executes the computer program, it further implements the following steps: when the number of consecutive training rounds without the risk of gradient overflow or gradient underflow in the next round reaches a preset number, the scaling factor of the current round of the training group is amplified according to a third value to obtain the scaling factor of the next round of the training group in the next round of training; wherein the third value is greater than 1 and less than the second value, and the consecutive preset number of training rounds include the current round of training.

[0215] In one embodiment, when the processor executes the computer program, it further performs the following steps: before predicting the scaling factor of each training group in the next round of training based on the current round gradient information, as well as the current round scaling factor and precision format of each training group, for any network layer in the target model, determining the ratio of the current round gradient information of the network layer to the current round scaling factor of the training group to which the network layer belongs; and adjusting the model parameters of the network layer according to the ratio.

[0216] In one embodiment, the target model is an object detection model, and different training groups include an embedding layer training group, a convolutional layer training group, and a fully connected layer training group. When the processor executes the computer program, it also implements the following steps: based on the structure of the target model, identifying the embedding layer, convolutional layer, and fully connected layer in the target model; grouping the identified embedding layers with the same accuracy format into the same embedding layer training group; grouping the identified convolutional layers with the same accuracy format into the same convolutional layer training group; grouping the identified fully connected layers with the same accuracy format into the same fully connected layer training group; and configuring independent initial scaling factors for the embedding layer training group, the convolutional layer training group, and the fully connected layer training group, respectively.

[0217] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: when the current round of training of the target model has ended, determining the current round gradient information of each network layer in the target model based on the current round scaling factors of different training groups in the target model during the current round of training; wherein the different training groups are network layers of the same type and precision format divided according to the structure of the target model, and each training group is configured with an independent initial scaling factor; and predicting the next round scaling factor of each training group in the next round of training based on the current round gradient information, and the current round scaling factor and precision format of each training group.

[0218] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the next round of training of the target model is completed, determine the next round gradient information of each network layer in the target model according to the next round scaling factor of each training group, and repeat the above operation until the target model meets the model convergence condition, and determine that the target model training is completed.

[0219] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: scaling the loss value of the target model in the current round of training according to the scaling factor of each training group to obtain the scaling loss value corresponding to each training group; and determining the current round gradient information of each network layer in the target model according to the scaling loss value and the parameter information of each network layer in each training group.

[0220] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the training loss value of the target model based on the prediction result of the target model for the training samples and the labels of the training samples in the current round of training; determining the intermediate loss value corresponding to each training group based on the training loss value, the current round scaling factor of the training group, and the preset weights; and determining the loss value of the current round of training based on the sum of the intermediate loss values ​​corresponding to different training groups.

[0221] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for any training group, predicting the gradient overflow risk of the next round of the training group based on the numerical range corresponding to the current round scaling factor and precision format of the training group, and the current round gradient information of each network layer in the training group; adjusting the current round scaling factor of the training group based on the gradient overflow risk of the next round of the training group to obtain the next round scaling factor of the training group in the next round of training.

[0222] In one embodiment, the gradient overflow risk in the next round includes the gradient overflow risk and the gradient underflow risk in the next round. When the computer program is executed by the processor, it further implements the following steps: determining the mean gradient and maximum absolute gradient of the training group based on the current round gradient information of each network layer in the training group; predicting the upper limit of the future gradient of the training group based on the product of the maximum absolute gradient and the scaling factor of the current round of the training group, and predicting the gradient overflow risk of the training group in the next round based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group; predicting the lower limit of the future gradient of the training group based on the product of the mean gradient and the scaling factor of the current round of the training group, and predicting the gradient underflow risk of the training group in the next round based on the lower limit of the future gradient and the lower limit of the numerical range.

[0223] In one embodiment, the current round gradient information of each network layer is represented by the current round gradient matrix; when the computer program is executed by the processor, it also performs the following steps: determining the average value of each element in the current round gradient matrix of each network layer in the training group as the average gradient value of the training group; and selecting the maximum value among the absolute values ​​of each element in the current round gradient matrix of each network layer in the training group as the maximum absolute value of the gradient of the training group.

[0224] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if the upper limit value of the future gradient is greater than a preset upper limit threshold, it determines that the gradient overflow risk representation of the training group in the next round exists in the next round; wherein, the preset upper limit threshold is determined by the product of the upper limit value of the numerical range corresponding to the precision format of the training group and a first coefficient, and the first coefficient is greater than 0 and less than 1.

[0225] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if the future gradient lower bound value is less than a preset lower bound threshold, it determines that the gradient overflow risk characterization of the training group in the next round indicates that there is a risk of gradient underflow in the next round; wherein, the preset lower bound threshold is determined based on the product of the lower bound value of the numerical range and a second coefficient, and the second coefficient is greater than 1.

[0226] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: If the gradient overflow risk representation for the next round indicates a gradient overflow risk in the next round, the scaling factor of the current round of the training group is reduced according to a first value to obtain the scaling factor of the training group in the next round of training; wherein the first value is greater than 0 and less than 1; If the gradient overflow risk representation for the next round indicates a gradient underflow risk in the next round, the scaling factor of the current round of the training group is increased according to a second value to obtain the scaling factor of the training group in the next round of training; wherein the second value is greater than 1; If the gradient overflow risk representation for the next round indicates neither a gradient overflow risk nor a gradient underflow risk in the next round, the scaling factor of the training group in the next round of training is predicted based on the consecutive rounds in which the gradient overflow risk does not exist.

[0227] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: when the number of consecutive training rounds without the risk of gradient overflow or gradient underflow in the next round reaches a preset number, the scaling factor of the current round of the training group is amplified according to a third value to obtain the scaling factor of the next round of the training group in the next round of training; wherein the third value is greater than 1 and less than the second value, and the consecutive preset number of training rounds include the current round of training.

[0228] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: before predicting the scaling factor of each training group in the next round of training based on the current round gradient information, as well as the current round scaling factor and precision format of each training group, for any network layer in the target model, determining the ratio of the current round gradient information of the network layer to the current round scaling factor of the training group to which the network layer belongs; and adjusting the model parameters of the network layer according to the ratio.

[0229] In one embodiment, the target model is an object detection model, and different training groups include an embedding layer training group, a convolutional layer training group, and a fully connected layer training group. When the computer program is executed by the processor, it further implements the following steps: based on the structure of the target model, identifying the embedding layer, convolutional layer, and fully connected layer in the target model; grouping the identified embedding layers with the same accuracy format into the same embedding layer training group; grouping the identified convolutional layers with the same accuracy format into the same convolutional layer training group; grouping the identified fully connected layers with the same accuracy format into the same fully connected layer training group; and configuring independent initial scaling factors for the embedding layer training group, the convolutional layer training group, and the fully connected layer training group, respectively.

[0230] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: upon completion of the current training iteration of the target model, determining the current-round gradient information of each network layer in the target model based on the current-round scaling factors of different training groups in the target model during the current training iteration; wherein the different training groups are network layers of the same type and precision format divided according to the structure of the target model, and each training group is configured with an independent initial scaling factor; and predicting the next-round scaling factor of each training group in the next training iteration based on the current-round gradient information, and the current-round scaling factor and precision format of each training group.

[0231] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the next round of training of the target model is completed, determine the next round gradient information of each network layer in the target model according to the next round scaling factor of each training group, and repeat the above operation until the target model meets the model convergence condition, and determine that the target model training is completed.

[0232] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: scaling the loss value of the target model in the current round of training according to the scaling factor of each training group to obtain the scaling loss value corresponding to each training group; and determining the current round gradient information of each network layer in the target model according to the scaling loss value and the parameter information of each network layer in each training group.

[0233] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the training loss value of the target model based on the prediction result of the target model for the training samples and the labels of the training samples in the current round of training; determining the intermediate loss value corresponding to each training group based on the training loss value, the current round scaling factor of the training group, and the preset weights; and determining the loss value of the current round of training based on the sum of the intermediate loss values ​​corresponding to different training groups.

[0234] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for any training group, predicting the gradient overflow risk of the next round of the training group based on the numerical range corresponding to the current round scaling factor and precision format of the training group, and the current round gradient information of each network layer in the training group; adjusting the current round scaling factor of the training group based on the gradient overflow risk of the next round of the training group to obtain the next round scaling factor of the training group in the next round of training.

[0235] In one embodiment, the gradient overflow risk in the next round includes the gradient overflow risk and the gradient underflow risk in the next round. When the computer program is executed by the processor, it further implements the following steps: determining the mean gradient and maximum absolute gradient of the training group based on the current round gradient information of each network layer in the training group; predicting the upper limit of the future gradient of the training group based on the product of the maximum absolute gradient and the scaling factor of the current round of the training group, and predicting the gradient overflow risk of the training group in the next round based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group; predicting the lower limit of the future gradient of the training group based on the product of the mean gradient and the scaling factor of the current round of the training group, and predicting the gradient underflow risk of the training group in the next round based on the lower limit of the future gradient and the lower limit of the numerical range.

[0236] In one embodiment, the current round gradient information of each network layer is represented by the current round gradient matrix; when the computer program is executed by the processor, it also performs the following steps: determining the average value of each element in the current round gradient matrix of each network layer in the training group as the average gradient value of the training group; and selecting the maximum value among the absolute values ​​of each element in the current round gradient matrix of each network layer in the training group as the maximum absolute value of the gradient of the training group.

[0237] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if the upper limit value of the future gradient is greater than a preset upper limit threshold, it determines that the gradient overflow risk representation of the training group in the next round exists in the next round; wherein, the preset upper limit threshold is determined by the product of the upper limit value of the numerical range corresponding to the precision format of the training group and a first coefficient, and the first coefficient is greater than 0 and less than 1.

[0238] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if the future gradient lower bound value is less than a preset lower bound threshold, it determines that the gradient overflow risk characterization of the training group in the next round indicates that there is a risk of gradient underflow in the next round; wherein, the preset lower bound threshold is determined based on the product of the lower bound value of the numerical range and a second coefficient, and the second coefficient is greater than 1.

[0239] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: If the gradient overflow risk representation for the next round indicates a gradient overflow risk in the next round, the scaling factor of the current round of the training group is reduced according to a first value to obtain the scaling factor of the training group in the next round of training; wherein the first value is greater than 0 and less than 1; If the gradient overflow risk representation for the next round indicates a gradient underflow risk in the next round, the scaling factor of the current round of the training group is increased according to a second value to obtain the scaling factor of the training group in the next round of training; wherein the second value is greater than 1; If the gradient overflow risk representation for the next round indicates neither a gradient overflow risk nor a gradient underflow risk in the next round, the scaling factor of the training group in the next round of training is predicted based on the consecutive rounds in which the gradient overflow risk does not exist.

[0240] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: when the number of consecutive training rounds without the risk of gradient overflow or gradient underflow in the next round reaches a preset number, the scaling factor of the current round of the training group is amplified according to a third value to obtain the scaling factor of the next round of the training group in the next round of training; wherein the third value is greater than 1 and less than the second value, and the consecutive preset number of training rounds include the current round of training.

[0241] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: before predicting the scaling factor of each training group in the next round of training based on the current round gradient information, as well as the current round scaling factor and precision format of each training group, for any network layer in the target model, determining the ratio of the current round gradient information of the network layer to the current round scaling factor of the training group to which the network layer belongs; and adjusting the model parameters of the network layer according to the ratio.

[0242] In one embodiment, the target model is an object detection model, and different training groups include an embedding layer training group, a convolutional layer training group, and a fully connected layer training group; when the computer program is executed by a processor, it also performs the following steps:

[0243] Based on the structure of the target model, identify the embedding layer, convolutional layer, and fully connected layer in the target model; group the identified embedding layers with the same accuracy format into the same embedding layer training group; group the identified convolutional layers with the same accuracy format into the same convolutional layer training group; group the identified fully connected layers with the same accuracy format into the same fully connected layer training group; and configure independent initial scaling factors for the embedding layer training group, convolutional layer training group, and fully connected layer training group respectively.

[0244] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0245] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0246] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0247] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model training method, characterized in that, The method includes: When the current round of training of the target model ends, the current round gradient information of each network layer in the target model is determined based on the current round scaling factor of different training groups in the target model during the current round of training; wherein, the different training groups are network layers of the same type and precision format divided according to the structure of the target model, and the different training groups are configured with independent initial scaling factors; Based on the current round gradient information, as well as the current round scaling factor and precision format of each training group, predict the next round scaling factor for each training group in the next training round.

2. The method of claim 1, wherein, The method further includes: When the next round of training of the target model is completed, the next round gradient information of each network layer in the target model is determined according to the next round scaling factor of each training group, and the above operation is repeated until the target model meets the model convergence condition, and the training of the target model is determined to be completed.

3. The method of claim 1, wherein, The step of determining the current round gradient information of each network layer in the target model based on the current round scaling factor of different training groups in the target model during the current round of training includes: The loss value of the target model in the current training round is scaled according to the scaling factor of each training group to obtain the scaling loss value corresponding to each training group. Based on the scaling loss value and the parameter information of each network layer in each training group, the current round gradient information of each network layer in the target model is determined.

4. The method according to claim 3, characterized in that, The loss value for the current training round is determined in the following way: Based on the prediction results of the target model for the training samples and the labels of the training samples in the current round of training, the training loss value of the target model is determined; For each training group, the intermediate loss value corresponding to the training group is determined based on the training loss value, the current round scaling factor of the training group, and the preset weight. The loss value for the current training round is determined based on the sum of the intermediate loss values ​​corresponding to different training groups.

5. The method according to claim 1, characterized in that, The step of predicting the scaling factor of each training group in the next training round based on the current round gradient information, and the current round scaling factor and precision format of each training group, includes: For any training group, based on the numerical range corresponding to the scaling factor and precision format of the current round of the training group, and the gradient information of each network layer in the current round of the training group, predict the gradient overflow risk of the next round of the training group. Based on the gradient overflow risk of the training group in the next round, the scaling factor of the current round of the training group is adjusted to obtain the scaling factor of the training group in the next round of training.

6. The method according to claim 5, characterized in that, The gradient overflow risk in the next round includes the gradient overflow risk and the gradient underflow risk in the next round; The step of predicting the gradient overflow risk of the next round of the training group based on the numerical range corresponding to the scaling factor and precision format of the current round of the training group, and the gradient information of each network layer in the current round of the training group, includes: Based on the current round gradient information of each network layer in the training group, determine the mean gradient and the maximum absolute value of the gradient corresponding to the training group; Based on the product of the maximum absolute value of the gradient and the scaling factor of the current round of the training group, predict the upper limit of the future gradient of the training group, and based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group, predict the gradient overflow risk of the next round of the training group. Based on the product of the gradient mean and the scaling factor of the current round of the training group, the future gradient lower bound value corresponding to the training group is predicted, and based on the future gradient lower bound value and the lower bound value of the numerical range, the gradient underflow risk of the training group in the next round is predicted.

7. The method according to claim 6, characterized in that, The gradient information of each network layer in the current round is represented by the gradient matrix of the current round; The step of determining the mean gradient and maximum absolute gradient of the training group based on the current round gradient information of each network layer in the training group includes: The average value of each element in the current round gradient matrix of each network layer in the training group is determined as the average gradient of the training group; and, The maximum absolute value of each element in the current round gradient matrix of each network layer in the training group is selected as the maximum absolute value of the gradient corresponding to the training group.

8. The method according to claim 6, characterized in that, The step of predicting the gradient overflow risk of the training group in the next round based on the upper limit of the future gradient and the upper limit of the numerical range corresponding to the precision format of the training group includes: If the upper limit of the future gradient is greater than the preset upper limit threshold, it is determined that the gradient overflow risk of the training group in the next round indicates that there is a risk of gradient overflow in the next round; wherein, the preset upper limit threshold is determined by multiplying the upper limit of the numerical range corresponding to the precision format of the training group with a first coefficient, wherein the first coefficient is greater than 0 and less than 1.

9. The method according to claim 6, characterized in that, The step of predicting the gradient underflow risk of the training group in the next round based on the future gradient lower bound and the lower bound of the numerical range includes: If the future gradient lower bound is less than a preset lower bound threshold, it is determined that the gradient overflow risk of the training group in the next round indicates that there is a risk of gradient underflow in the next round; wherein, the preset lower bound threshold is determined based on the product of the lower bound of the numerical range and a second coefficient, and the second coefficient is greater than 1.

10. The method according to any one of claims 5-9, characterized in that, The step of adjusting the scaling factor of the current round of the training group based on the gradient overflow risk in the next round, to obtain the scaling factor of the training group in the next round of training, includes: If there is a risk of gradient overflow in the next round, the scaling factor of the current round of the training group is reduced according to the first value to obtain the scaling factor of the training group in the next round of training; wherein, the first value is greater than 0 and less than 1. If there is a risk of gradient underflow in the next round, the scaling factor of the current round of the training group is amplified according to the second value to obtain the scaling factor of the training group in the next round of training; wherein, the second value is greater than 1; If there is no gradient overflow risk or gradient underflow risk in the next round, the scaling factor of the training group in the next round of training is predicted based on the number of consecutive rounds in which there is no gradient overflow risk or gradient underflow risk.

11. The method according to claim 10, characterized in that, The step of, where there is no gradient overflow risk or gradient underflow risk in the next round, includes the following consecutive training rounds based on the gradient overflow risk in the next round when there is no gradient overflow risk or gradient underflow risk: If a preset number of consecutive training rounds are completed without the risk of gradient overflow or gradient underflow in the next round, the scaling factor of the current round of the training group is amplified according to the third value to obtain the scaling factor of the training group in the next round of training; wherein the third value is greater than 1 and less than the second value, and the preset number of consecutive training rounds include the current round of training.

12. The method according to any one of claims 1-9, characterized in that, Before predicting the scaling factor of each training group in the next training round based on the current round gradient information, and the current round scaling factor and precision format of each training group, the method further includes: For any network layer in the target model, determine the ratio of the current round gradient information of the network layer to the current round scaling factor of the training group to which the network layer belongs; Adjust the model parameters of the network layer according to the ratio.

13. The method according to any one of claims 1-9, characterized in that, The target model is an object detection model, and the different training groups include an embedding layer training group, a convolutional layer training group, and a fully connected layer training group; the method further includes: Based on the structure of the target model, identify the embedding layer, convolutional layer, and fully connected layer in the target model; Embedding layers with the same accuracy format are grouped into the same embedding layer training group; Convolutional layers with the same accuracy format are grouped into the same convolutional layer training group; All fully connected layers with the same accuracy format are grouped into the same fully connected layer training group. Each of the embedding layer training group, the convolutional layer training group, and the fully connected layer training group is configured with an independent initial scaling factor.

14. A model training device, characterized in that, The device includes: The gradient determination module is used to determine the gradient information of each network layer in the target model in the current round of training based on the scaling factor of the different training groups in the target model in the current round of training when the current round of training of the target model has ended; wherein, the different training groups are network layers of the same type and precision format divided according to the structure of the target model, and the different training groups are configured with independent initial scaling factors. The factor prediction module is used to predict the scaling factor of each training group in the next training round based on the gradient information of the current round, as well as the scaling factor and precision format of each training group in the current round.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images