Model training method and related devices

Abstract

This application discloses a model training method. The method is applicable to a dynamic computational graph scenario and also applicable to a static computational graph scenario. The method includes: obtaining training data; using the training data as the input of the model, and in the training process of the model, calculating parameters by using a first accuracy range to obtain a calculated value; and when the calculated value of the parameters overflows the first accuracy range, recalculating the parameters by using a second accuracy range, and performing iterative training on the model one or more times by using the recalculated parameters, where the second accuracy range includes the first accuracy range, or the second accuracy range partially overlaps with the first accuracy range. When the calculated value of the parameters overflows the accuracy range, the accuracy range used in the training process of the model is adjusted in real time, so that the problem of training stagnation caused by overflow in low-precision training can be effectively solved. In addition, the initialization solution means is customized without relying on manual experience, and the training accuracy can be automatically adjusted layer by layer in real time.

Description

[Technical Field]

[0001] This application relates to the field of artificial intelligence, particularly to model training methods and related devices. [Background technology]

[0002] Artificial intelligence (AI) is a theory, method, technology, and application system that simulates and extends human intelligence by using digital computers, or a machine controlled by a digital computer to perceive its environment, acquire knowledge, and achieve optimal results by using knowledge. A neural network is a manually established, dynamic system that uses a directed graph as its topological structure. Neural networks are information processing systems that process information by using continuous or discontinuous inputs as status responses, and aim to simulate the structure and function of the human brain. After decades of development, artificial neural networks have achieved widespread success and are widely used in many fields such as pattern recognition, automatic control, signal processing, decision support, artificial intelligence, and scientific computing. In particular, in many fields such as image processing, speech and video processing, and natural language processing, artificial neural networks are in a stage of rapid development and play an indispensable role.

[0003] Currently, in the parameter storage or calculation processes of most neural network training, the precision of the data format used is primarily determined through human experience. For example, the setter might decide, based on experience, whether to use 16-bit half-precision floating-point numbers (FP16) or 32-bit single-precision floating-point numbers (FP32) for each network layer.

[0004] However, setting the precision applicable to the network layer structure based on human experience relies too heavily on the expertise of the person setting it. [Overview of the project]

[0005] This invention provides a model training method and related device for adjusting the accuracy range used in the model training process in real time when the calculated parameter values ​​overflow the accuracy range, thereby effectively resolving the training stagnation problem caused by overflow in low-accuracy training.

[0006] A first aspect of the embodiments of the present invention provides a model training method. The method is applicable to both dynamic computation graph scenarios and static computation graph scenarios. A dynamic computation graph scenario can be understood as updating the computation graph after the network structure of each layer of the model has been computed. A static computation graph scenario can be understood as updating the computation graph after the network structure of all layers of the model has been computed. The main difference is the opportunity to update the computation graph. The computation of the computation graph is applicable to the method provided in the embodiments of the present invention. The method may be performed by a training device or by a component of the training device (e.g., a processor, chip, or chip system). The method comprises the steps of: acquiring training data; using the training data as input to a model and, in the model training process, computing parameters by using a first precision range and acquiring computed values; and, if the computed values ​​overflow the first precision range, recalculating the parameters by using a second precision range and performing iterative training on the model one or more times by using the recalculated parameters, wherein the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range.

[0007] In this embodiment of the present application, when the calculated value of a parameter overflows a first precision range during the model training process, the parameter is recalculated using a second precision range. That is, the precision range is automatically adjusted in real time using overflow information of the calculated parameter value, which can reduce the memory occupied by model training and improve model training efficiency. In this way, problems such as training stagnation or training failure caused by the parameter overflowing the first precision range are reduced. In addition, in this embodiment of the present application, compared to prior art methods that need to be used to determine whether the type of network layer of the model uses high-precision floating-point numbers or low-precision floating-point numbers, the precision range to which a parameter can be applied can be adjusted in real time using parameter overflow information, reducing the overflow problem caused by low-precision floating-point calculations.

[0008] Optionally, in a possible implementation of the first embodiment, the model includes multiple network structures, and the step of recalculating parameters by using a second precision range includes recalculating parameters starting from the network structure of the first layer of the model by using the second precision range.

[0009] In this possible implementation, if the calculated values ​​of the current layer's network structure overflow, a new precision range may be selected to perform a recalculation starting from the first layer's network structure of the model, thereby mitigating the problem of computational errors caused by the overflow of calculated values.

[0010] Optionally, in a possible implementation of the first embodiment, the model includes multiple network structures, and the step of recalculating parameters by using a second precision range includes recalculating parameters starting from the current network structure where the calculated values ​​overflow by using the second precision range.

[0011] In this possible implementation, if the calculated values ​​of the current layer's network structure overflow, a new precision range may be selected to perform a recalculation on the current layer's network structure, thereby mitigating the problem of computational errors caused by the overflow of calculated values.

[0012] Optionally, in a possible implementation of the first embodiment, the parameter relates to the loss function of the model, or the parameter relates to the calculation of the model in the forward propagation process, or the parameter relates to the calculation of the model in the backpropagation process.

[0013] Optionally, in a possible implementation of the first embodiment, the model includes multiple network structures, and the parameters include one or more of the following: intermediate features or the value of the model's loss function computed by the multiple network structures in the forward propagation process, where the intermediate features are output features of any one of the multiple network structures; and gradients computed by the multiple network structures in the backpropagation process, where the gradients include the gradients of the intermediate features and / or the weight gradients of the model.

[0014] In this possible implementation, the parameters may be parameters that need to be computed in the forward or backpropagation process of the model in the training process, or parameters output by individual layers of the model, or parameters obtained after computation of all layers of the entire model has been completed, or similar. This improves the applicable scenarios of the method in the training process. In other words, the method provided in this embodiment of the application may be used to perform precision tuning for all computations in the training process of a model.

[0015] Optionally, in a possible implementation of the first embodiment, when the parameter includes a gradient, the calculated value is obtained by dividing the gradient by a scaling factor, which is used to reduce the probability that the gradient will overflow. The method further comprises: updating the scaling factor by using a first factor, where the updated scaling factor is used to replace the unupdated scaling factor for the next iteration of training the model, and the first factor is a positive number less than 1.

[0016] In this possible implementation, if the parameters calculated during the backpropagation process (i.e., the values ​​obtained by dividing the gradient by the scaling factor) overflow, a different precision range is used to recalculate the weight gradient, the scaling factor is updated, and the overflow of calculated values ​​in subsequent iterative training is reduced.

[0017] Optionally, in a possible implementation of the first embodiment, the minimum value of the scaling coefficient is a preset threshold greater than or equal to 1, which is used to reduce the probability that the gradient will overflow.

[0018] In this possible implementation, a lower bound is set for the scaling factor value during the inverse calculation process, reducing the risk of subsequent parameter precision underflow.

[0019] Optionally, in a possible implementation of the first embodiment, when the parameters include intermediate features computed in the forward propagation process, the above step of recalculating the parameters by using a second precision range includes: computing intermediate features of an overflow layer using the second precision range, or computing intermediate features layer by layer, starting from the network structure of the first layer, where the overflow layer is a network structure in which the computed values ​​of the intermediate features overflow the first precision range.

[0020] In this possible implementation, when the intermediate feature overflows the first accuracy range, the second accuracy range can be used to perform recomputation for the overflow layer or to perform recomputation starting from the network structure of the first layer. That is, the accuracy ranges of some layers can be modified, or the accuracy ranges of all layers can be modified. This makes the solution flexible.

[0021] Optionally, in a possible implementation of the first aspect, when the parameter includes the value of the loss function of the model, the above step of recomputing the parameter by using the second accuracy range includes calculating the value of the loss function by using the second accuracy range or performing calculations layer by layer starting from the network structure of the first layer until the value of the loss function is obtained.

[0022] In this possible implementation, when the value of the loss function overflows the first accuracy range, the second accuracy range can be used to recompute the value of the loss function or to perform calculations layer by layer starting from the network structure of the first layer until the value of the loss function is obtained. That is, the accuracy ranges of some layers can be modified, or the accuracy ranges of all layers can be modified. This makes the solution flexible.

[0023] Optionally, in a possible implementation of the first aspect, the above step of training the model by using the recomputed parameter includes: in the Nth iteration of the model training process, obtaining the number of overflows of a plurality of network structures in the model based on the first accuracy range, where N is a positive integer greater than or equal to 1; and when the number of overflows is greater than or equal to the second threshold, determining that the initial accuracy range in the next iterative training process is changed from the first accuracy range to the second accuracy range and clearing the number of overflows to zero.

[0024] In this possible implementation, the initial accuracy range is adjusted by recording the number of overflows. When the first accuracy range affects the training of the model, the initial accuracy range is adjusted from the first accuracy range to the second accuracy range to ensure the accuracy of subsequent model training.

[0025] Optionally, in a possible implementation of the first aspect, the number of overflows includes: the number of overflows of a plurality of network structures in the forward propagation process and / or the number of overflows of a plurality of network structures in the backpropagation process.

[0026] In this possible implementation, the determination condition for adjusting the accuracy range (i.e., the determination of the number of overflows) can be the number of overflows in the entire training process or the number of overflows in forward or backward propagation. This improves the applicable range of the method.

[0027] Optionally, in a possible implementation of the first aspect, the parameter relates to the loss function. The loss function can vary according to different training methods of the model. In supervised learning, the loss function is used to represent the difference between the output of the model and the label to which the training data belongs. In unsupervised training, the loss function can be a user-defined function. For example, when the task of the model is a classification task, the loss function is used to represent the difference between the output and the input of the model (or the clustering result or the like). Alternatively, in unsupervised training, it is understood that the output of the model can be fed back to the input of the model. For example, the label is the training data (i.e., the output obtained by the model is sent to another network and the training data is fed back). It can be understood that the loss function is not limited in the present embodiment of the present application. The loss function can also be understood as the optimization target function of the model, specifically, it can be set based on actual requirements.

[0028] A second aspect of the embodiments of the present invention provides a training device. The training device is applicable to both dynamic and static computation graph scenarios. The training device comprises: an acquisition unit configured to acquire training data; and a computation unit configured to take the training data as input to a model and, in the model training process, to calculate parameters and acquire calculated values ​​by using a first precision range. The computation unit is further configured to: recalculate the parameters by using a second precision range if the calculated values ​​overflow the first precision range, and to perform iterative training on the model one or more times by using the recalculated parameters, wherein the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range.

[0029] Optionally, in a possible implementation of the second embodiment, the model includes multiple network structures, and the computational unit is configured to recalculate the parameters starting from the first layer network structure of the model, specifically by using a second precision range.

[0030] Optionally, in a possible implementation of the second embodiment, the computation unit is configured to recalculate the parameters starting from the current network structure where the computation values ​​overflow, by specifically using a second precision range.

[0031] Optionally, in a possible implementation of the second embodiment, the model includes multiple network structures, and the parameters include one or more of the following: intermediate features or the value of the model's loss function computed by the multiple network structures in the forward propagation process, where the intermediate features are output features of any one of the multiple network structures; and gradients computed by the multiple network structures in the backpropagation process, where the gradients include the gradients of the intermediate features and / or the weight gradients of the model.

[0032] Optionally, in a possible implementation of the second embodiment, when the parameters include a gradient calculated in the backpropagation process, the calculated value is the value obtained by dividing the gradient by a scaling factor, which is used to reduce the probability that the gradient will overflow. The computational unit further updates the scaling factor by using a first factor, where the updated scaling factor is used to replace the unupdated scaling factor for the next iteration of training the model, the first factor being a positive number less than 1, and the minimum value of the scaling factor being a preset threshold greater than or equal to 1.

[0033] Optionally, in a possible implementation of the second embodiment, when the parameters include intermediate features computed in the forward propagation process, the computation unit specifically computes the intermediate features of the overflow layer using a second precision range, or computes intermediate features layer by layer starting from the network structure of the first layer, where the overflow layer is configured to be a network structure in which the computed values ​​of intermediate features in multiple network structures overflow the first precision range.

[0034] Optionally, in a possible implementation of the second embodiment, when the parameters include the value of the model's loss function, the computation unit is configured to specifically compute the value of the loss function using a second precision range, or to start from the first layer network structure and perform computation layer by layer until the value of the loss function is obtained.

[0035] Optionally, in a possible implementation of the second embodiment, the computation unit is configured such that, in the Nth iteration of the model training process, it obtains the number of overflows of multiple network structures in the model based on a first precision range, where N is a positive integer greater than or equal to 1. Specifically, the computation unit is configured to determine that if the number of overflows is greater than or equal to a second threshold, the initial precision range for the next iterative training process will change from the first precision range to the second precision range, and to erase the number of overflows to zero.

[0036] Optionally, in a possible implementation of the second embodiment, the number of overflows includes: the number of overflows of multiple network structures in the forward propagation process, and / or the number of overflows of multiple network structures in the backpropagation process.

[0037] A third aspect of the present invention provides a training device including a processor, the processor being coupled to memory, and the memory being configured to store a program or instruction. When the program or instruction is executed by the processor, the training device can implement a method in the first aspect or any one of the possible implementations of the first aspect.

[0038] A fourth aspect of the present application provides a computer-readable medium that stores computer programs or instructions. When the computer programs or instructions are executed on a computer, the computer can perform a method in the first aspect or any one of possible implementations thereof.

[0039] A fifth aspect of the present application provides a computer program product. When the computer program product is executed on a computer, the computer is capable of performing a method in either the first aspect or any one of the possible implementations of the first aspect.

[0040] For technical effects resulting from the second, third, fourth, fifth embodiments, or any conceivable implementations thereof, please refer to the first embodiment or the technical effects resulting from different conceivable implementations of the first embodiment. Details are not described herein.

[0041] From the above technical solutions, it can be seen that the present invention has the following features: When the calculated value of a parameter overflows a first precision range during the model training process, the parameter is recalculated using a second precision range. That is, the precision range is automatically adjusted in real time using overflow information of the calculated value of the parameter, and as a result, the memory occupied by model training may be reduced, and the model training efficiency may be improved. In this way, problems such as training stagnation or training failure caused by the parameter overflowing the first precision range are reduced. In addition, in embodiments of the present invention, compared to methods in the prior art that need to be used to determine whether the type of network layer of the model uses high precision floating-point numbers or low precision floating-point numbers, the precision range to which the parameter can be applied can be adjusted in real time using parameter overflow information, and the overflow problem caused by low precision floating-point calculations is reduced. [Brief explanation of the drawing]

[0042] [Figure 1] This is a diagram of the structure of the system architecture according to the embodiment of the present invention. [Figure 2] This is a diagram of the hardware structure of the chip according to an embodiment of the present invention. [Figure 3] This is a schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 4] This is an exemplary diagram of the structure of the model according to the embodiment of the present invention. [Figure 5A] This is another schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 5B]This is another schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 6A] This is another schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 6B] This is another schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 7A] This is another schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 7B] This is another schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 7C] This is another schematic flowchart of the model training method according to the embodiment of the present invention. [Figure 8] This is a diagram of the structure of a training device according to an embodiment of the present invention. [Figure 9] This is another diagram of the structure of the training device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0043] This invention provides a model training method and related device for adjusting the accuracy range used in the model training process in real time when the calculated value overflows the accuracy range, thereby effectively resolving the problem of training stagnation or training failure caused by overflow in low-accuracy training. In addition, the requirements for the network mixed-accuracy initialization solution are low, the initialization solution can be customized without relying on human experience, and the training accuracy can be automatically adjusted layer by layer in real time.

[0044] To facilitate understanding, the terms and concepts primarily relevant to the embodiments of this application are first explained below. 1. Neural Networks A neural network may contain neural units. A neural unit is X s And it may be an arithmetic unit that uses the intercept b as input, and the output of the arithmetic unit may be as follows:

number

[0045] Here, s = 1, 2, ..., or n, where n is a natural number greater than 1, and W s X s θ is the weight of the neural unit, where b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear properties into the neural network and converts the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input to the next layer. The activation function can be a ReLU function. A neural network is a network formed by connecting many single neural units together. Specifically, the output of one neural unit can be the input to another neural unit. The input of each neural unit is connected to the local receptive field of the previous layer, and features of the local receptive field can be extracted. The local receptive field can be a region containing multiple neural units.

[0046] The operation of each layer of a neural network can be described using the formula y = a(Wx + b). From a physical standpoint, the operation of each layer of a neural network can be understood as completing a transformation from the input space to the output space (i.e., from the row space to the column space of a matrix) by performing five operations on the input space (a set of input vectors). The five operations are as follows: 1. Dimensionality increase / dimension reduction; 2. Scaling up / scaling down; 3. Rotation; 4. Translation; and 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 is performed by +b, and operation 5 is performed by a(). The word "space" is used here for descriptive purposes because the objects being classified are not single things but types of things. Space is the entire set of individual things of this type. W is the weight vector, and each value in the vector represents the weight value of one neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above. In other words, the weights W in each layer control how the space is transformed. The goal of training a neural network is to eventually obtain a weight matrix (a weight matrix formed by the vectors W in multiple layers) in all layers of the trained neural network. Thus, the neural network training process is essentially a method of learning the control of spatial transformation, more specifically, the learning of the weight matrix.

[0047] Neural networks, also known as Artificial Neural Networks (ANNs), are dynamic systems that use manually established, directed graphs as their topological structure. Neural networks are information processing systems that aim to simulate the structure and function of the human brain by processing information using continuous or discontinuous inputs as status responses. After decades of development, artificial neural networks have achieved widespread success and are widely used in many fields, including pattern recognition, automated control, signal processing, decision support, artificial intelligence, and scientific computing. Generally, a network consists of an input layer, a hidden layer, and an output layer.

[0048] 2. Loss function In the process of training a neural network, the output of the neural network is expected to be as close as possible to the truly expected predicted value. Therefore, the current network's predicted value can be compared to the truly expected target value, and then the weight matrix of each layer of the neural network is updated based on the difference between the two values ​​(of course, there is usually an initialization process before the update is performed for the first time, specifically, the parameters for each layer of the neural network are pre-configured). For example, if the network's predicted value is large, the weight matrix is ​​adjusted to make the predicted value smaller, and this adjustment continues until the neural network can obtain the truly expected target value through prediction. Therefore, it is necessary to pre-define "how to obtain the difference between the predicted value and the target value through comparison." This is the loss function or objective function. The loss function and objective function are important equations that measure the difference between the predicted value and the target value. The loss function is used as an example. A high output value (loss) of the loss function indicates a large difference. Therefore, training a neural network is the process of minimizing the loss as much as possible.

[0049] 3.Forward Propagation Forward propagation in a neural network is the computational process from the input layer to the hidden layer and then to the output layer. Starting from the input layer, the output (activation value) of the previous layer is used as the input to the next layer based on the network's topological structure. The output of each layer is computed layer by layer until the final output layer. This process is called forward propagation of the network.

[0050] 4. Backward Propagation Backpropagation in neural networks, short for "error backpropagation," is a common method for training artificial neural networks in combination with optimization methods (such as gradient descent). This method is used to calculate the gradient of the loss function for all weights in the network. The gradient is then fed back into the optimization method to update the weights and minimize the loss function.

[0051] 5. Calculation graph Calculation graphs typically use arrows to indicate the order of calculations. For example, using the function y=5(a+bc), the value of bc is calculated first and stored in the variable i. Next, a+i is calculated and stored in the variable j. Then, 5*j is calculated, and the result of y is obtained.

[0052] 6. Accuracy Range The precision range is the range of precision for a data type used by a computer. The precision range can be a specific precision or a dynamic range of precision. For illustrative purposes, the example below uses floating-point (FP) numbers as the common data type in neural networks. In practical applications, the data type could alternatively be an integer (int), such as int8 or int16.

[0053] Floating-point numbers (FPs) are primarily used to represent decimal numbers and typically consist of three parts: sign, exponent, and mantissa. The sign can be a single bit representing positive or negative, while the exponent and mantissa can be multiple bits. Generally, the mantissa represents precision, and the exponent is used to represent the dynamic range (referred to as the precision range in the embodiments of this application) that precision can achieve. When floating-point numbers represent decimal numbers, decimal numbers in the decimal system cannot be accurately converted to binary, and decimal numbers are truncated when stored in a computer with fixed bits. Therefore, precision loss can occur when floating-point numbers represent decimal numbers.

[0054] Floating-point numbers generally can include three formats, specifically, half-precision floating-point numbers, single-precision floating-point numbers, and double-precision floating-point numbers, which are described below.

[0055] Half-precision floating-point numbers are binary data types used by computers, occupy 16 bits (i.e., 2 bytes) in computer memory, and can also be abbreviated as FP16. The absolute value range of the values that can be represented by half-precision floating-point numbers is approximately [6×10 -8 ,65504]. The precision of FP16 is 2 -10 .

[0056] Single-precision floating-point numbers are binary data types used by computers, occupy 32 bits (i.e., 4 bytes) in computer memory, and can also be abbreviated as FP32. The absolute value range of the values that can be represented by single-precision floating-point numbers is approximately [1.4×10 -45 ,1.7×10 38 . The precision of FP32 is 2 -23 .

[0057] Double-precision floating-point numbers are binary data types used by computers, occupy 64 bits (i.e., 8 bytes) in computer memory, and can also be abbreviated as FP64. Double-precision floating-point numbers can represent 15 or 16 significant digits in decimal. The absolute value range of the values that can be represented by double-precision floating-point numbers is approximately [2.23×10 -308 ,1.80×10 38 . The P64 precision is 2 -52 .

[0058] To present the above three types of floating-point numbers with different precisions more intuitively, the structures of the three types of floating-point numbers are shown in Table 1. Table 1 [Table 1]

[0059] In the 16 bits occupied by FP16, the sign occupies 1 bit, the exponent occupies 5 bits, and the mantissa occupies 10 bits; in the 32 bits occupied by FP32, the sign occupies 1 bit, the exponent occupies 8 bits, and the mantissa occupies 23 bits; and in the 64 bits occupied by FP64, the sign occupies 1 bit, the exponent occupies 11 bits, and the mantissa occupies 52 bits.

[0060] In practical applications, it may be understood that floating-point number formats, storage formats occupying more bits, and similar extensions may be used to represent higher-precision floating-point numbers. For example, there is a floating-point number occupying 128 bits (which may be abbreviated as FP128). This specification is not limited to this.

[0061] 7. Overflow In embodiments of this application, overflow includes overflow and underflow. Overflow means that the absolute value of the calculated value is excessively large and exceeds the maximum value that can be represented by the precision range. Underflow means that the absolute value of the calculated value is excessively small and is smaller than the closest positive value to zero that can be represented by the precision range.

[0062] Overflows may include storage overflows and computational overflows. Embodiments of this application primarily apply to computational overflow scenarios.

[0063] For example, if the first precision range is 1 to 5, the values ​​of two parameters are 3, and it is assumed that the two parameters are within the first precision range when stored. However, parameter calculations can overflow the first precision range. For example, the two parameters above are added together (3 + 3 = 6), and 6 is greater than the maximum value of 5 that can be represented by the first precision range; that is, the calculated value of the parameters overflows the first precision range. It should be understood that this example does not limit the boundary values ​​of the precision range.

[0064] Specifically, FP16 is used as an example to illustrate overflow conditions. Overflow includes, if the calculated value is a positive number, being greater than the largest positive number that can be represented by FP16; or, if the calculated value is a negative number, being less than the smallest negative number that can be represented by FP16. Underflow includes, if the calculated value is a positive number, being less than the smallest positive number that can be represented by FP16; or, if the calculated value is a negative number, being greater than the largest negative number that can be represented by FP16.

[0065] 8.Mixed Precision (MP) Currently, most models are trained using 32-bit single-precision floating-point numbers (FP32). In mixed-precision training methods, model training is performed using half-precision or even lower precision and single precision, thereby reducing the memory required for model training. In addition, since lower-precision arithmetic is faster than single-precision arithmetic, hardware efficiency is further improved.

[0066] From the above, it is clear that the key point of mixed precision is a policy to ensure accuracy and improve training efficiency by defining which parts of the network are trained using high precision and which parts are trained using low precision. In other words, the key point of mixed precision is how to specifically combine single precision and high precision for training.

[0067] Currently, mixed precision training methods mainly include the following two approaches.

[0068] In the first method, computational precision is specified based on the type of layer in the neural network. Some types of layers use high precision for computation, while others use low precision.

[0069] In the second method, precision is dynamically selected by determining whether the quantization error exceeds a threshold. The quantization error can be measured at different points in the network or measured over time as training is performed. For example, the quantization error can be calculated by comparing the training results to a baseline value. The baseline value can be determined by using various methods, such as training the same network using full-precision floating-point values, iteratively calculating a subset using high precision, and analyzing or sampling data statistics for the relevant calculations.

[0070] However, in the first method, layers of the same type have different requirements for computational accuracy in different networks, or in different training phases of the same network. Therefore, it is clear that specifying the accuracy applicable to computation based on type is not sufficiently flexible or wise.

[0071] In the second method, the adjustment of training accuracy depends on the accuracy baseline value. The baseline value must be obtained by iteratively calculating it using high accuracy or by training the same network using full-precision floating-point values. As a result, the solution is not yet fully automated and the amount of computation required for network training can increase significantly due to the construction of the baseline value. This is the opposite of the objective of using low-precision training to reduce computation and speed up training.

[0072] In view of this, embodiments of the present invention provide a model training method and related device that adjust the accuracy range used in the model training process in real time when the parameters of the computed values ​​in the neural network overflow the accuracy range, thereby effectively resolving the training stagnation caused by overflow in low-accuracy training in the prior art. In addition, the requirements for the network mixed-accuracy initialization solution are low, the initialization solution can be customized without relying on human experience (for example, human experience is not required to perform accuracy adjustments in different layers of each network), and the training accuracy can be automatically adjusted layer by layer in real time during the training process based on whether or not an overflow occurs.

[0073] Referring to the attached drawings, the model training method and related devices provided in the embodiments of the present application will be described in detail below.

[0074] First, we will describe the system architecture provided in the embodiments of this application.

[0075] Refer to Figure 1. Embodiments of the present invention provide a system architecture 100. As shown in the system architecture 100, a data acquisition device 160 is configured to collect training data. The training data in this embodiment of the application may include one or more of images, audio, text, and the like. The training data is stored in a database 130, and the training device 120 acquires a target model / rule 101 through training based on the training data maintained in the database 130. The target model / rule 101 may be used to implement computer vision tasks (e.g., classification, segmentation, detection, and image generation). Specifically, the target model / rule 101 in this embodiment of the application may be a neural network or the like. Note that in actual application, the training data maintained in the database 130 is not necessarily collected by the data acquisition device 160 and may be received from another device. In addition, note that the training device 120 does not necessarily train the target model / rule 101 entirely on the training data maintained in the database 130, and may acquire training data from the cloud or another location to perform model training. The above description should not be construed as a limitation of this embodiment.

[0076] The target model / rule 101 acquired through training with the training device 120 may be applied to different systems or devices, for example, to the execution device 110 shown in Figure 1. The execution device 110 may be a terminal, such as a mobile phone terminal, tablet computer, notebook computer, augmented reality (AR) device / virtual reality (VR) device, or in-vehicle terminal. Of course, the execution device 110 may alternatively be a server, cloud, or similar. In Figure 1, the I / O interface 112 is configured for the execution device 110 and is configured to exchange data with an external device. A user may input data into the I / O interface 112 by using a client device 140. The input data corresponds to training data. The input data in this embodiment of the application may also include one or more of images, audio, text, and the like. In addition, the input data may be entered by the user, uploaded by the user by using a capture device, or, of course, originate from a database or the like. This specification does not particularly limit this.

[0077] The preprocessing module 113 is configured to perform preprocessing (e.g., splitting, selection, and transformation) based on the input data received by the I / O interface 112. For example, the input data is split, and multiple data blocks (patches) are obtained.

[0078] In related processing steps in which the execution device 110 preprocesses input data, or in which the calculation module 111 of the execution device 110 performs calculations, the execution device 110 may call data, codes and similar items from the data storage system 150 and implement the corresponding processing, or may store data, instructions and similar items obtained through the corresponding processing in the data storage system 150.

[0079] Finally, the I / O interface 112 returns the processing results (e.g., classification results, segmentation results, or detection results) to the client device 140, providing the processing results to the user.

[0080] It should be noted that the training device 120 can generate corresponding target models / rules 101 for different goals or tasks based on different training data. These corresponding target models / rules 101 can be used to achieve the goals or complete the tasks and provide the user with the desired results.

[0081] In the case shown in Figure 1, the user may manually provide input data. Input data can be manually provided using a screen provided by the I / O interface 112. In another case, the client device 140 may automatically send input data to the I / O interface 112. If the client device 140 needs user authorization to automatically send input data, the user may set the corresponding permission on the client device 140. The user may view the results output by the execution device 110 on the client device 140. Specifically, the results may be presented in a specific manner, such as a display, sound, action, or the like. Alternatively, the client device 140 may be used as a data acquisition terminal, collecting input data input to the I / O interface 112 and output values ​​output from the I / O interface 112 as new sample data, and storing the new sample data in the database 130. Of course, the client device 140 may, alternatively, not perform the collection. Instead, the I / O interface 112 directly stores the input data and output values ​​shown in the diagram, which are input to the I / O interface 112 and output values ​​that are output from the I / O interface 112, as new sample data in the database 130.

[0082] It should be noted that Figure 1 is merely a diagram of a system architecture according to an embodiment of the present invention. The spatial relationships between the devices, components, modules, and similar shown in the figure do not constitute a limitation. For example, in Figure 1, the data storage system 150 is external memory to the execution device 110. In another case, the data storage system 150 may be located on the execution device 110 instead.

[0083] The hardware structure of the chip provided in the embodiments of this application will be described below.

[0084] Figure 2 shows the hardware structure of a chip according to an embodiment of the present invention. The chip includes a neural network processing unit 20. The chip may be located in the execution device 110 shown in Figure 1 and configured to complete the computational operations of the computing module 111. Alternatively, the chip may be located in the training device 120 shown in Figure 1 and configured to complete the training operations of the training device 120 and output a target model / rule 101.

[0085] The neural network processing unit 20 may be any processor suitable for large-scale exclusive OR operations, such as a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). An NPU is used as an example. The neural network processing unit 20 functions as a coprocessor and is mounted on a host central processing unit (CPU) (host CPU). The host CPU assigns tasks. The core of the NPU is the arithmetic circuit 203, and the controller 204 controls the arithmetic circuit 203 to extract data from memory (weight memory or input memory) and perform operations.

[0086] In some implementations, the arithmetic circuit 203 includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 203 is a two-dimensional systolic array. Alternatively, the arithmetic circuit 203 may be a one-dimensional systolic array, or another electronic circuit capable of performing arithmetic operations such as multiplication and addition. In some implementations, the arithmetic circuit 203 is a general-purpose matrix processor.

[0087] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 203 fetches data corresponding to matrix B from the weight memory 202 and buffers the data in each PE in the arithmetic circuit. The arithmetic circuit fetches data for matrix A from the input memory 201, performs matrix operations using matrix B, and stores the obtained partial result or the obtained final result of the matrix in the accumulator 208.

[0088] The vector computation unit 207 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operations, logarithmic operations, or value comparisons. For example, the vector computation unit 207 may be configured to perform network computations such as pooling, batch normalization, or local response normalization in non-convolutional / non-FC layers of a neural network.

[0089] In some implementations, the vector computation unit 207 may store the processed output vector in the integrated memory 206. For example, the vector computation unit 207 may apply a nonlinear function to the output of the arithmetic circuit 203, for example, a vector of cumulative values, to generate activation values. In some implementations, the vector computation unit 207 generates normalized values, combined values, or both. In some implementations, the processed output vector may be used as an activation input to the arithmetic circuit 203, for example, in later layers of a neural network.

[0090] The integrated memory 206 is configured to store input data and output data.

[0091] Regarding weight data, the direct memory access controller (DMAC) 205 transfers the input data from the external memory to the input memory 201 and / or the integrated memory 206, stores the weight data from the external memory in the weight memory 202, and stores the data from the integrated memory 206 in the external memory.

[0092] The bus interface unit (BIU) 210 is configured to implement interaction between the host CPU, DMAC, and instruction fetch buffer 209 via the bus.

[0093] The instruction fetch buffer 209, which is connected to the controller 204, is configured to store instructions used by the controller 204.

[0094] The controller 204 is configured to control the operation process of the arithmetic accelerator by calling instructions buffered in the instruction fetch buffer 209.

[0095] Generally, the integrated memory 206, input memory 201, weight memory 202, and instruction fetch buffer 209 are on-chip memories. External memory is memory located outside the NPU. External memory may be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0096] The model training method and data processing method in the embodiment of this application will be described in detail below with reference to the attached drawings.

[0097] First, the application scenarios for the model training method provided in embodiments of the present application will be described. The method is applicable to both dynamic computation graph scenarios and static computation graph scenarios. A dynamic computation graph scenario can be understood as updating the computation graph after the network structure of each layer of the model has been computed. A static computation graph scenario can be understood as updating the computation graph after the network structure of all layers of the model has been computed. The main difference is the opportunity to update the computation graph. The computation of the computation graph is applicable to the model training method (or computation accuracy tuning method) provided in embodiments of the present application.

[0098] The model training method in an embodiment of the present invention will be described in detail below with reference to Figure 3. The method may be performed by a training device or by a component of the training device (e.g., a processor, chip, or chip system). The training device may be a cloud device or a terminal device. For example, the training device may be a computer or server or other device having robust computing capabilities to perform the model training method, or it may be a system including a cloud device and terminal devices. For example, the training method may be performed by the training device 120 in Figure 1 and the neural network processing unit 20 in Figure 2.

[0099] Optionally, the model training method may be processed by the CPU, or by both the CPU and the GPU; or the GPU may not be used, and another processor suitable for neural network computation may be used. This is not limited to the present invention.

[0100] Figure 3 is a schematic flowchart of a model training method according to an embodiment of the present invention. The method may include steps 301 to 303. Steps 301 to 303 will be described in detail below.

[0101] Step 301: Obtain training data.

[0102] In this embodiment of the present application, the training device may acquire training data in multiple ways. The training data may be transmitted by and received from another device (e.g., a server or service device), selected from a database, captured by a user, or acquired in other ways. This specification does not limit this.

[0103] The training data in this embodiment of the present application may include one or more of images, audio, text, and the like. Specifically, the training data relates to the scenario in which the model is applied. For example, when the function of the model is speech recognition, the specific form of the training data may be audio data or the like. In another example, when the function of the model is image classification, the specific form of the training data may be image data or the like. In yet another example, when the function of the model is speech prediction, the specific form of the training data may be text data or the like. It should be understood that the above multiple cases are merely examples and do not necessarily have a one-to-one correspondence. For example, for speech recognition, the specific form of the training data may be image data, text data, or the like (for example, when the model is applied to a scenario in the field of education where images are displayed and audio is played, the function of the model is to recognize the audio corresponding to the images, and the specific form of the training data may be image data). There are other scenarios in actual application. For example, when the model is applied to a video recommendation scenario, the training data may be word vectors or the like corresponding to the videos. In some application scenarios, the training data may alternatively include data from different modalities simultaneously. For example, in an autonomous driving scenario, training data may include image / video data collected by a camera, and may further include voice / text data or similar for sending commands by a user. The specific form or type of training data is not limited to the embodiments of this application.

[0104] If the model is trained using supervised learning, the training data acquired in this step can be understood as training data that retains labels. If the model is trained using unsupervised learning, the training data acquired in this step is training data that does not retain labels.

[0105] Step 302: Use the training data as input to the model, and during the model training process, calculate the parameters by using the first precision range and obtain the calculated values.

[0106] After acquiring training data, the training device uses the training data as input to the model, calculates parameters using a first precision range, and obtains the calculated values. The parameters relate to the model's loss function. In supervised learning, the loss function is used to represent the difference between the model's output and the labels to which the training data belongs. In unsupervised training, the loss function can be a user-defined function. For example, when the model's task is a classification task, the loss function is used to represent the difference between the model's output and input (or clustering results or similar). Alternatively, in unsupervised training, it is understood that the model's output is expected to be fed back into the model's input. For example, the labels are the training data (i.e., the output acquired by the model is sent to another network and returns the training data). It should be understood that the loss function is not limited to these embodiments of the application. The loss function can also be understood as the model's optimization target function, which can be set specifically based on actual requirements.

[0107] The parameters in this embodiment of the present application may include one or more of the following: parameters relating to the loss function of the model, parameters relating to the calculation of the model in the forward propagation process, and parameters relating to the calculation of the model in the backpropagation process.

[0108] Optionally, the model includes multiple network structures. In addition, the model in this embodiment of the present application is specifically an artificial neural network. The specific number of layers or the specific structure included in the artificial neural network may be determined based on actual requirements, but is not limited herein.

[0109] The precision ranges in this embodiment of the Application (e.g., the first precision range and the second precision range) may be the precision ranges of the data type (e.g., int or float). For the sake of ease of explanation, in this embodiment of the Application, only examples where the precision range is the FP precision range will be used for explanation. Of course, in actual applications, the precision range may alternatively be the precision range of another data type (e.g., int).

[0110] Optionally, the first precision range is the dynamic reachable range of the FP16 precision. For explanations of FP16 and precision ranges, please refer to the explanations in the related terms above. Further details are not provided herein.

[0111] In this embodiment of the present application, the precision ranges used by the network structure of all layers of the model may or may not be the same. In other words, the first precision range may be a refined precision range for each layer of the model, or it may be the same precision range used by all layers of the model. That is, later replacing the first precision range with the second precision range and recalculating the parameters may be a layer-granular precision range adjustment (i.e., only the precision range of a specific layer is adjusted, and the precision range of non-specific layers is not adjusted, and a specific layer may also be an overflow layer), or it may be a precision range adjustment for the entire model (i.e., all layers).

[0112] In a conceivable implementation, the network structure of all layers of the model uses the same precision range, in which case the first precision range is the same. Alternatively, this embodiment of the present application is understood to be applicable to training scenarios using single precision.

[0113] In another possible implementation, at least two layers of the network structure in the model use different precision ranges. For example, the model is trained by using mixed precision. In this case, the first precision range is either low precision or high precision in mixed precision. Generally, in mixed precision training, the probability of a high-precision overflow problem is relatively low by default, and the first precision range may specifically be the low precision range in mixed precision. Alternatively, it should be understood that this embodiment of the present application is applicable to training scenarios using mixed precision, that is, precision tuning is then performed only for specific layers.

[0114] In this embodiment of the present application, the parameters can take multiple forms. These forms will be described separately below.

[0115] In the first case, the parameters are parameters in the forward propagation process.

[0116] Thus, the parameters can be intermediate features or loss function values ​​computed by multiple network structures during the forward propagation process. Intermediate features can also be understood as activation values ​​obtained by multiple network structures using activation functions. Loss function values ​​can also be understood as the difference between model outputs and labels computed by all layers of the network structure after forward propagation is complete.

[0117] For example, the model shown in Figure 4 is used as an example. The model includes a three-layer network structure, which consists of an input layer (containing three neurons), a hidden layer (containing four neurons), and an output layer (containing two neurons).

number

number

number

number

[0118] Formula 1:

number

number

[0119] The activation value can be calculated for each layer using Equation 2, and finally, the model's output f(x) can be obtained based on the input X. The value of the loss function is calculated based on the difference between the model's output f(x) and the label value Y of X. The loss function can be a Mean Square Error (MSE) loss, an absolute squared error loss, a cross-entropy loss, a hinge loss, or similar, and can be specifically set based on the actual requirements. The structure of the loss function is not limited.

[0120] It should be understood that the model structure shown in Figure 4 is merely an example and does not constitute a limitation to the model referred to in this embodiment of the application. In addition, Equations 1 and 2 are merely examples of representations in the forward propagation process. In actual application, the forward propagation process may have alternative representations, which are not particularly limited herein.

[0121] In the example above, the parameters may include the values ​​of intermediate features and / or the loss function. a calculated at each layer l This can be understood as an intermediate feature, and the difference between f(x) and Y is the value of the loss function.

[0122] In the second case, the parameters are the parameters in the backpropagation process.

[0123] Thus, the parameters can be gradients calculated by multiple network structures in the backpropagation process. These gradients include one or more of the following: gradients of intermediate features, gradients of weights in the model, gradients of losses, and similar ones.

[0124] The backpropagation process can be understood as a process of continuously adjusting weights by using the values ​​of the loss function.

[0125] For example, in the example above, the backpropagation process minimizes the loss function by using the optimizer and a pre-set learning rate to find the optimal parameters (e.g., weights).

number

[0126] It should be understood that the two cases of parameters described above are merely examples. In actual application, parameters may have alternative or other cases, which are not specifically limited herein.

[0127] Step 303: If the calculated values ​​overflow the first precision range, recalculate the parameters using the second precision range, and then perform iterative training on the model one or more times using the recalculated parameters.

[0128] If, in step 302, the calculated values ​​obtained by the training device calculating the parameters using the first precision range overflow the first precision range, the parameters are recalculated using the second precision range, and the model is repeatedly trained using the recalculated parameters one or more times. The second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range.

[0129] Compared to conventional techniques where, if accuracy overflow occurs in an iteration, the training data used in the overflow is discarded, leading to training stagnation, the method provided in this embodiment of the present application allows training to continue by adjusting the accuracy range in a timely manner, thereby potentially improving model training efficiency.

[0130] In this embodiment of the present application, the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range.

[0131] For example, the second precision range includes the first precision range. An example is used where the first precision range is FP16 and the second precision range is FP32. The first precision range may be (-65504, 65504) and the second precision range is [-1.7 × 10 38 ,1.7×10 38 It could be.

[0132] For example, the second precision range partially overlaps with the first precision range. An example is used where the first precision range is FP16 and the second precision range is int16. The first precision range is (-65504, 65504), and the second precision range can be an integer from -32768 to 32767.

[0133] It should be understood that the example does not limit the boundary values ​​of the precision range.

[0134] Optionally, the first and second precision ranges can be concepts of relative high and low ranges. If the first precision range is the high-precision range (hereinafter referred to as high precision), then the second precision range is the low-precision range (hereinafter referred to as low precision). If the first precision range is low precision, then the second precision range is high precision. In general, overflow problems arise in low precision.

[0135] In this embodiment of the present application, only examples in which the first precision range is low precision and the second precision range is high precision are used for illustrative purposes. Of course, in actual applications, adjustments between high and low precision, or from low precision to high precision, can also be implemented by using the methods provided in this embodiment of the present application.

[0136] Furthermore, recalculating parameters by using a second precision range may include multiple cases, where the parameters may be recalculated starting from the network structure of the first layer of the model by using the second precision range. Alternatively, the parameters may be recalculated starting from the current network structure where the calculated values ​​overflow by using the second precision range. This specification does not limit this.

[0137] In a possible implementation, when the parameters include intermediate features computed in the forward propagation process, the above step of recalculating the parameters by using a second precision range specifically includes: computing intermediate features of the overflow layer using the second precision range, or computing intermediate features layer by layer, starting from the network structure of the first layer, where the overflow layer is a network structure in which the computed values ​​of intermediate features in multiple network structures overflow the first precision range.

[0138] In another possible implementation, when the parameters include the value of the model's loss function, the above step of recalculating the parameters by using a second precision range specifically includes calculating the value of the loss function using the second precision range, or starting from the network structure of the first layer and performing calculations layer by layer until the value of the loss function is obtained.

[0139] For example, the model includes five layers of network structure. If the calculated values ​​obtained by performing parameter calculations for the fourth layer using the first precision range overflow the first precision range, the second precision range may be used to perform recalculations from the first to the fourth layer. Alternatively, the second precision range may be used to recalculate the parameters for the fourth layer.

[0140] Optionally, when the parameter includes a gradient, the calculated value is obtained by dividing the gradient by a scaling factor, which is used to reduce the probability that the gradient will overflow. The method further comprises: a step of updating the scaling factor by using a first coefficient, where the updated scaling factor is used to replace the unupdated scaling factor for the next iteration of training the model, and the first coefficient is a positive number less than 1.

[0141] For more information on overflow, please refer to the explanations in the related terms above. Further details are not provided herein. Overflow of the first precision range in a calculated value can also be understood as the inability to accurately represent the calculated value of a parameter using the first precision range. This can result in subsequent rounding errors or overflow errors caused by the narrower precision range.

[0142] If the calculated parameter values ​​overflow the first precision range, the parameters are recalculated using a second precision range different from the first, and the model is subjected to iterative training one or more times using the updated parameters. For specific training processes, please refer to the explanations in the forward and backpropagation processes. For example, the loss function in forward propagation is calculated using parameters calculated using the second precision range. To achieve the goal that the value of the loss function is less than a threshold, the model is subjected to iterative training one or more times, and the trained model is obtained.

[0143] Note that, as described in step 302, recalculating using the second precision range can be a precision range adjustment for the layer granularity or a precision range adjustment for the entire model (i.e., all layers). That is, if the calculation precision of all layers is initialized to the same precision range, the recalculation in this step is for all layers (or is understood as a precision adjustment for all layers). If the calculation precision of all layers is initialized to different precision ranges (i.e., mixed precision calculation), the recalculation in this step is for a specific layer (or is understood as a precision adjustment for a specific layer). The calculation precision of a specific layer overflows the first precision range.

[0144] In addition, steps 301 to 303 in this embodiment may be performed one or more times, and after updates have been performed one or more times during the model training process, steps 301 to 303 may be performed once. Alternatively, steps 301 to 303 may be performed again when a predetermined cycle or predetermined number of times has been met.

[0145] In this embodiment of the present application, when the calculated value of a parameter overflows a first precision range during the model training process, the parameter is recalculated using a second precision range. That is, the precision range is automatically adjusted in real time using overflow information of the calculated parameter value, and as a result, the memory occupied by model training may be reduced, and the model training efficiency may be improved. In this way, problems such as training stagnation caused by parameter calculations overflowing the first precision range are reduced. In addition, in this embodiment of the present application, compared to prior art methods that need to be used to determine whether the type of network layer of the model uses high precision floating-point numbers or low precision floating-point numbers, the precision range to which the parameter can be applied can be adjusted in real time using parameter overflow information, and the overflow problem caused by low precision floating-point calculations is reduced.

[0146] Optionally, in the embodiment shown in Figure 3, if the calculated parameter values ​​do not overflow the first precision range, the value of the loss function obtained in the forward propagation process is multiplied by the scaling factor, and the gradient calculation in the backpropagation process is performed using the first precision range. If the value obtained by dividing the gradient by the scaling factor overflows the first precision range, the scaling factor is updated using the first factor, and recalculation is performed in the forward and backpropagation processes using the second precision range. The first factor is a positive number less than 1, and the updated scaling factor is used to replace the unupdated scaling factor and perform the next iteration of training the model.

[0147] To reduce the risk of subsequent parameter precision underflow, the lower bound of the scaling factor value may be limited. For example, the minimum value of the scaling factor may be a preset threshold greater than or equal to 1.

[0148] Optionally, an initial precision range (e.g., a first precision range) may be set for the model in the first iteration process, and the initial precision range for each iteration may be adjusted based on the number of iterations and the number of overflows in order to improve the efficiency of the model in subsequent iterations.

[0149] Specifically, in the Nth iteration of the model training process, the number of overflows of multiple network structures in the model based on the first precision range is obtained, where N is a positive integer greater than or equal to 1. If the number of overflows is greater than or equal to the second threshold, it is determined that the initial precision range for the next iterative training process will be changed from the first precision range to the second precision range, and the number of overflows is wiped clean and becomes zero. The number of overflows includes the number of overflows of multiple network structures in the forward propagation process and / or the number of overflows of multiple network structures in the backpropagation process.

[0150] In addition, the parameters in the embodiment shown in Figure 3 can take multiple forms. In a possible implementation, if the parameter is an intermediate feature and the calculation of the intermediate feature overflows the first precision range, the second precision range may be used to recalculate the intermediate feature, and the second precision range may be further used to perform subsequent loss calculations and / or weight gradient calculations. In other words, the calculation precision adjustment may be a precision range adjustment for an overflow parameter, and further may be a calculation precision range adjustment for another parameter in a later training process. This specification does not particularly limit this. In another possible implementation, if the parameter is a loss and the calculation of the loss overflows the first precision range, the second precision range may be used to recalculate the loss (including recalculating the loss by using the second precision range, or recalculating from the first layer until the loss is obtained, i.e., the recalculation range may include the current calculation or include multiple calculations prior to the current overflow calculation), and the second precision range may be further used to perform subsequent weight gradient calculations. In other words, when a parameter calculated in an intermediate computation overflows the precision range during the training process, the computational precision adjustment may be a precision range adjustment for the overflow parameter, a precision range adjustment for a parameter preceding the overflow parameter that does not overflow during the training process, and a computational precision range adjustment for another parameter in a later training process. This specification is not limited to this.

[0151] Figures 5A and 5B show another model training method according to an embodiment of the present invention. The implementer of the method is the same as the implementer of the embodiment shown in Figure 3, and its details are not described here. The method may include steps 501 to 511. Steps 501 to 511 are described in detail below.

[0152] Step 501: Obtain training data.

[0153] For step 501, please refer to the description of step 301 in the embodiment shown in Figure 3. Further details are not described herein.

[0154] Step 502: Determine if the number of overflows (num) is greater than or equal to the second threshold (N). If the number of overflows is greater than or equal to the second threshold, step P508 It is executed. If the number of overflows is less than the second threshold, step P503 It will be executed.

[0155] In this embodiment of the present application, only examples in which the overflow count includes the number of times the calculated values ​​of each layer overflow the first precision range in the forward and inverse calculation processes of the model are used for illustrative purposes. In practical applications, the overflow count may be understood to be the number of times the intermediate features of each layer overflow the first precision range in the forward calculation of the model, or the number of times the gradient overflows the first precision range in the inverse calculation of the model. That is, the overflow count may be a count in the overall forward and inverse processes, or a separate count in the forward or inverse process. This specification does not particularly limit this.

[0156] In this embodiment of the present application, forward calculation is the calculation of parameters (e.g., intermediate features or loss) in the forward propagation process of the model. Inverse calculation is the calculation of parameters (e.g., gradients) in the backpropagation process of the model.

[0157] The training device determines whether the number of overflows is greater than or equal to a second threshold (N), where N is an integer greater than or equal to 0. If the number of overflows is greater than or equal to the second threshold, step P508 It is executed. If the number of overflows is less than the second threshold, step P503 It will be executed.

[0158] First, it can be understood that the overflow count num is set to 0.

[0159] Step 503: Perform a forward calculation using the first precision range and obtain the loss.

[0160] The number of overflows in step 502 is the second threshold. A small place In that case, the execution of this step is triggered.

[0161] For the calculation of the loss in step 503, please refer to the description of step 302 in the embodiment shown in Figure 3. Further details are not described herein.

[0162] Step 504: Determine if the loss will overflow. If the loss will overflow, step 509 is performed. If the loss will not overflow, the loss is multiplied by the scaling factor and step 505 is performed.

[0163] After acquiring the loss, the training device determines whether the loss overflows. If the loss overflows, step 509 is performed. If the loss does not overflow, the loss is multiplied by the scaling factor and step 505 is performed.

[0164] Step 505: Perform the inverse calculation using the first precision range to obtain the weight gradient.

[0165] If the loss in step 504 does not overflow the first precision range, the loss is multiplied by the scaling factor, and the execution of this step is triggered.

[0166] Step 505 is performed by multiplying by a scaling factor to prevent underflow of the calculated value.

[0167] For an explanation of the calculation of the weight gradient in step 505, please refer to the explanation of step 302 in the embodiment shown in Figure 3. Further details are not described herein.

[0168] Step 506: Determine whether the value obtained by dividing the weight gradient by the scaling factor overflows. If the value obtained by dividing the weight gradient by the scaling factor overflows, step 510 is performed. If the value obtained by dividing the weight gradient by the scaling factor does not overflow, step 507 is performed.

[0169] After obtaining the weight gradient, the training device determines whether the value obtained by dividing the weight gradient by the scaling factor overflows the first precision range. If the value obtained by dividing the weight gradient by the scaling factor overflows, step 510 is performed. If the value obtained by dividing the weight gradient by the scaling factor does not overflow, step 507 is performed.

[0170] Step 507: Update the weights.

[0171] If the value obtained by dividing the weight gradient by the scaling factor in step 506 does not overflow the first precision range, and / or after step 508, the execution of this step is triggered.

[0172] Alternatively, if the value obtained by dividing the weight gradient by the scaling factor does not overflow the first precision range, the value is understood to be used to perform iterative updates to the weights.

[0173] Step 508: Perform forward and reverse recalculations using the second precision range.

[0174] The number of overflows in step 502 is the second threshold. Larger or equal to In addition, and / or after step 509, the execution of this step is triggered.

[0175] For step 508, please refer to the description of step 303 in the embodiment shown in Figure 3. Further details are not described herein.

[0176] Step 509: Increase the number of overflows by 1 (i.e., num+1).

[0177] This step is triggered if the loss in step 504 overflows the first precision range, and / or if the value obtained by dividing the weight gradient by the scaling factor overflows the first precision range.

[0178] Alternatively, the number of overflows is recorded, and in subsequent iterations, it is understood that a decision is made whether to adjust the initial precision range in each iteration based on a comparison between the number of overflows and a second threshold. For example, if the number of overflows is greater than the second threshold, the initial precision range is adjusted from the first precision range to the second precision range. Naturally, in addition to the number of overflows, the number of iterations may also be considered for adjusting the initial precision range. For example, the number of iterations reaches 1000, and the number of overflows is greater than 800 (i.e., the second threshold is 800). In this case, setting the initial precision range to the first precision range indicates that it has affected model training. To ensure the accuracy of subsequent model training, the initial precision range is adjusted to the second precision range, thereby reducing problems such as training stagnation caused by parameters overflowing the first precision range.

[0179] Step 510: Update the scaling factor by using the first coefficient.

[0180] If the value obtained by dividing the weight gradient by the scaling factor in step 506 overflows the first precision range, this step is triggered. If the value obtained by dividing the weight gradient by the scaling factor overflows the first precision range, the scaling factor is updated using the first factor, which is a positive number less than 1.

[0181] Alternatively, if the value obtained by dividing the weight gradient by the scaling factor overflows the first precision range, the scaling factor is understood to be multiplied by a positive number less than 1 for adjustment.

[0182] Step 511: Determine if the scaling factor is greater than or equal to a preset threshold.

[0183] After determining that the value obtained by dividing the weight gradient by the scaling factor overflows the first precision range, the training device may determine whether the scaling factor is smaller than a preset threshold when updating the scaling factor using the first factor. If the scaling factor is smaller than the preset threshold, the scaling factor is adjusted to the preset threshold. If the scaling factor is greater than or equal to the preset threshold, the scaling factor is not modified and step 509 is performed.

[0184] In this step, the lower limit of the scaling coefficient value is restricted, which can reduce the risk of subsequent parameter underflows and improve the stability of model training.

[0185] Setting a lower bound on the adjusted scaling factor can be understood as being for the next iteration. After the scaling factor has been adjusted, step 509 is performed.

[0186] In addition, steps 501 to 511 in this embodiment may be executed multiple times, and after updates have been performed one or more times during the model training process, steps 501 to 511 may be executed once. Alternatively, steps 501 to 511 may be executed again when a predetermined cycle or predetermined number of times has been met.

[0187] In this embodiment of the present application, on the one hand, the precision range is automatically adjusted in real time by using overflow information generated in the forward and / or inverse calculation processes, which can reduce the memory occupied by model training and improve model training efficiency. In this way, problems such as training stagnation caused by parameter calculations overflowing the first precision range are reduced. In addition, in this embodiment of the present application, the precision range to which parameters can be applied can be adjusted in real time by using parameter overflow information, compared to prior art methods that need to be used to determine whether the type of network layer of the model uses high-precision floating-point numbers or low-precision floating-point numbers, which reduces overflow problems caused by low-precision floating-point calculations. On the other hand, the initial precision range can be adjusted, and when the first precision range affects model training, the initial precision range is adjusted from the first precision range to the second precision range to ensure the accuracy of subsequent model training. On the other hand, in the inverse calculation process, a lower limit is set for the scaling coefficient value to reduce the risk of subsequent parameter precision underflow.

[0188] In the embodiments shown in Figures 5A and 5B, it can be understood that there are multiple means of processing after a forward calculation overflows. For example, if a forward calculation overflows, a second precision range may be used to perform the forward calculation again, and the first precision range may be used to perform the inverse calculation. For example, if a forward calculation overflows, the second precision range may be used to perform the forward calculation again and then the subsequent inverse calculation. That is, precision adjustment may be an adjustment of the overflow calculation only, or it may be a precision adjustment of the entire calculation. This specification does not particularly limit this.

[0189] Figures 6A and 6B show another model training method according to an embodiment of the present invention. The implementer of the method is the same as the implementer of the embodiment shown in Figure 3, and its details are not described here. The method may include steps 601 to 615. Steps 601 to 615 are described in detail below.

[0190] Step 601: Obtain training data.

[0191] Step 602: Determine whether the number of overflows (num) is greater than or equal to the second threshold (N). If the number of overflows is greater than or equal to the second threshold, step P612 It is executed. If the number of overflows is less than the second threshold, step P603 It will be executed.

[0192] For steps 601 and 602, please refer to the description of steps 501 and 502 in the embodiments shown in Figures 5A and 5B. Further details are not described herein.

[0193] Step 603: Perform forward calculations using the first precision range to obtain intermediate features.

[0194] The number of overflows in step 602 is the second threshold. A small place In that case, the execution of this step is triggered.

[0195] Step 604: Determine whether the intermediate features overflow. If the intermediate features overflow, step 605 is executed. If the intermediate features do not overflow, step 607 is executed.

[0196] After acquiring the intermediate features, the training device determines whether the calculated values ​​of the intermediate features overflow. If the calculated values ​​of the intermediate features overflow, step 605 is performed. If the calculated values ​​of the intermediate features do not overflow, step 607 is performed.

[0197] Step 605: Increase the number of overflows by 1 (i.e., num+1).

[0198] This step is triggered if the intermediate features in step 604, the loss in step 608, and / or the value obtained by dividing the weight gradient by the scaling factor overflow the first precision range.

[0199] Step 606: Calculate intermediate features of the overflow layer using the second precision range, or start from the first layer and calculate intermediate features layer by layer.

[0200] This step can be understood as follows: If an intermediate feature overflows the first precision range, the second precision range may be used to perform calculations for the overflow layer, or for all layers.

[0201] Step 607: Perform a forward calculation using the first precision range and obtain the loss.

[0202] If the intermediate features in step 604 do not overflow, the execution of this step is triggered.

[0203] Step 608: Determine if the loss will overflow. If the loss will overflow, step 613 is performed. If the loss will not overflow, the loss is multiplied by the scaling factor and step 609 is performed.

[0204] Step 609: Perform the inverse calculation using the first precision range to obtain the weight gradient.

[0205] Step 610: Determine whether the value obtained by dividing the weight gradient by the scaling factor overflows. If the value obtained by dividing the weight gradient by the scaling factor overflows, step 614 is performed. If the value obtained by dividing the weight gradient by the scaling factor does not overflow, step 611 is performed.

[0206] Step 611: Update the weights.

[0207] Step 612: Perform forward and reverse recalculations using the second precision range.

[0208] Step 613: Increase the number of overflows by 1 (i.e., num+1).

[0209] Step 614: Update the scaling coefficient by using the first coefficient.

[0210] Step 615: Determine if the scaling factor is greater than or equal to a preset threshold.

[0211] For steps 607 to 615, please refer to the description of steps 503 to 511 in the embodiments shown in Figures 5A and 5B. Further details are not described herein.

[0212] In this embodiment of the present application, when the calculation of intermediate features overflows a first precision range, a second precision range may be used to calculate intermediate features of the overflow layer, or to calculate intermediate features layer by layer, starting from the network structure of the first layer. In other words, the adjustment of the precision range may be an adjustment of a specific layer, or an adjustment of all layers in the entire network structure. On the other hand, the precision range is automatically adjusted in real time by using overflow information generated in the forward and / or inverse calculation processes, which may reduce the memory occupied by model training and improve model training efficiency. In this way, problems such as training stagnation caused by parameter calculations overflowing the first precision range are reduced. In addition, in this embodiment of the present application, the precision range to which parameters can be applied may be adjusted in real time by using parameter overflow information, compared to methods in the prior art that need to be used to determine whether the type of network layer of the model uses high-precision floating-point numbers or low-precision floating-point numbers, and the overflow problem caused by low-precision floating-point calculations is reduced. On the other hand, the initial precision range can be adjusted, and when the first precision range affects model training, the initial precision range is adjusted from the first precision range to the second precision range to ensure accuracy in subsequent model training. On the other hand, in the inverse calculation process, a lower limit is set for the scaling coefficient value to reduce the risk of subsequent parameter precision underflow.

[0213] Figures 7A, 7B, and 7C illustrate another model training method according to an embodiment of the present invention. The implementer of the method is the same as that of the embodiment shown in Figure 3, and its details are not described here. The method may include steps 701 to 717. Steps 701 to 717 are described in detail below.

[0214] Step 701: Obtain training data.

[0215] Step 702: Determine whether the number of overflows (num) is greater than or equal to the second threshold (N). If the number of overflows is greater than or equal to the second threshold, step P712 It is executed. If the number of overflows is less than the second threshold, step P703 It will be executed.

[0216] Step 703: Perform forward calculations using the first precision range to obtain intermediate features.

[0217] Step 704: Determine whether the intermediate features overflow. If the intermediate features overflow, step 705 is executed. If the intermediate features do not overflow, step 707 is executed.

[0218] Step 705: Increase the number of overflows by 1 (i.e., num+1).

[0219] Step 706: Calculate intermediate features of the overflow layer using the second precision range, or start from the first layer and calculate intermediate features layer by layer.

[0220] Step 707: Perform a forward calculation using the first precision range and obtain the loss.

[0221] Step 708: Determine if the loss will overflow. If the loss will overflow, step 716 is performed. If the loss will not overflow, the loss is multiplied by the scaling factor and step 709 is performed.

[0222] Step 709: Perform the inverse calculation using the first precision range to obtain the weight gradient.

[0223] Step 710: Determine whether the value obtained by dividing the weight gradient by the scaling factor overflows. If the value obtained by dividing the weight gradient by the scaling factor overflows, step 714 is performed. If the value obtained by dividing the weight gradient by the scaling factor does not overflow, step 711 is performed.

[0224] Step 711: Update the weights.

[0225] Step 712: Perform forward and reverse recalculations using the second precision range.

[0226] Step 713: Increase the number of overflows by 1 (i.e., num+1).

[0227] Step 714: Update the scaling factor by using the first coefficient.

[0228] Step 715: Determine that the scaling factor is greater than or equal to a preset threshold.

[0229] For steps 701 to 715, please refer to the description of steps 601 to 615 in the embodiments shown in Figures 6A and 6B. Further details are not described herein.

[0230] Step 716: Increase the number of overflows by 1 (i.e., num+1).

[0231] Step 716 is the same as step 705. In other words, the number of overflows is the cumulative number of times during the model training process that the intermediate features, loss, and weight gradients overflow the first accuracy range.

[0232] Step 717: Recalculate the loss using the second precision range, or start from the first layer and perform calculations layer by layer until a loss is obtained. The loss is multiplied by the scaling factor and step 709 is performed.

[0233] This step can be understood as follows: If the loss overflows the first precision range, the second precision range may be used to perform a final recalculation of the loss for precision range adjustment, or to perform calculations for all layers of the model for precision range adjustment.

[0234] In this embodiment of the present application, when the loss calculation overflows the first precision range, the second precision range may be used to recalculate the loss or to perform calculations layer by layer, starting from the first layer of the network structure, until the loss is obtained. In other words, the adjustment of the precision range may be an adjustment of a specific layer or an adjustment of all layers in the entire network structure. On the other hand, the precision range is automatically adjusted in real time by using overflow information generated in the forward and / or inverse calculation processes, which may reduce the memory occupied by model training and improve model training efficiency. Thus, problems such as training stagnation caused by parameter calculations overflowing the first precision range are reduced. In addition, in this embodiment of the present application, the precision range to which parameters are applicable may be adjusted in real time by using parameter overflow information, compared to methods in the prior art that need to be used to determine whether the type of network layer of the model uses high-precision floating-point numbers or low-precision floating-point numbers, which reduces the overflow problem caused by low-precision floating-point calculations. On the other hand, the initial precision range can be adjusted, and when the first precision range affects model training, the initial precision range is adjusted from the first precision range to the second precision range to ensure accuracy in subsequent model training. On the other hand, in the inverse calculation process, a lower limit is set for the scaling coefficient value to reduce the risk of subsequent parameter precision underflow.

[0235] The above describes the model training method in the embodiment of the present invention. Below, the training device in the embodiment of the present invention will be described. Please refer to Figure 8. The embodiment of the training device in the embodiment of the present invention comprises: an acquisition unit 801 configured to acquire training data; and a calculation unit 802 configured to use the training data as input to the model and to calculate parameters and acquire calculated values ​​by using a first precision range in the model training process.

[0236] The computing unit 802 is further configured such that, if the calculated values ​​overflow the first precision range, it recalculates the parameters by using the second precision range, and then performs iterative training on the model one or more times using the recalculated parameters, where the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range.

[0237] Optionally, the model includes multiple network structures, and the computing unit 802 is configured to recalculate the parameters starting from the first layer network structure of the model, specifically by using a second precision range.

[0238] Optionally, the computing unit 802 is configured to recalculate the parameters starting from the current network structure where the calculated values ​​overflow, specifically by using a second precision range.

[0239] Optionally, the model includes multiple network structures, and the parameters include one or more of the following: intermediate features or the value of the model's loss function computed by the multiple network structures in the forward propagation process, where the intermediate features are output features of any one of the multiple network structures; and gradients computed by the multiple network structures in the backpropagation process, where the gradients include the gradients of the intermediate features and / or the weight gradients of the model.

[0240] When the parameter optionally includes a gradient that is calculated in the backpropagation process, the calculated value is obtained by dividing the gradient by a scaling factor, which is used to reduce the probability that the gradient will overflow. The computational unit 802 further updates the scaling factor by using a first coefficient, where the updated scaling factor is used to replace the unupdated scaling factor for the next iteration of training the model, the first coefficient is a positive number less than 1, and the minimum value of the scaling factor is a preset threshold greater than or equal to 1.

[0241] Optionally, when the parameters include intermediate features calculated in the forward propagation process, the computation unit 802 specifically calculates the intermediate features of the overflow layer using a second precision range, or calculates intermediate features layer by layer, starting from the network structure of the first layer, where the overflow layer is a network structure that is among multiple network structures and whose calculated intermediate features overflow the first precision range.

[0242] Optionally, when the parameters include the value of the model's loss function, the computation unit 802 is configured to calculate the value of the loss function using a second precision range, or to perform calculations layer by layer, starting from the first layer network structure, until the value of the loss function is obtained.

[0243] Optionally, the computing unit 802 is configured to obtain the number of overflows of multiple network structures in the model based on a first precision range during the Nth iteration of the model training process, where N is a positive integer greater than or equal to 1. Specifically, the computing unit 802 is configured to determine that if the number of overflows is greater than or equal to a second threshold, the initial precision range for the next iterative training process will change from the first precision range to the second precision range, and to erase the number of overflows to zero.

[0244] Optionally, the number of overflows includes the number of overflows of multiple network structures in the forward propagation process and / or the number of overflows of multiple network structures in the backpropagation process.

[0245] In this embodiment, the calculations performed by the units in the training device are the same as those described in the embodiments shown in Figures 1 to 7A, 7B, and 7C. Further details are not described herein.

[0246] In this embodiment, when the calculated value of a parameter overflows the first precision range during the model training process, the calculation unit To802 is The parameters are recalculated by using a second precision range. That is, the precision range is automatically adjusted in real time using overflow information of the calculated parameter values, which can reduce the memory occupied by model training and improve model training efficiency. In this way, problems such as training stagnation caused by parameter calculations overflowing the first precision range are reduced. In addition, in this embodiment of the present application, compared to prior art methods that need to be used to determine whether the type of network layer of the model uses high-precision floating-point numbers or low-precision floating-point numbers, the precision range to which the parameters can be applied can be adjusted in real time using parameter overflow information, and the overflow problem caused by low-precision floating-point calculations is reduced.

[0247] Figure 9 is a diagram of the structure of another training device according to the present invention. The training device may include a processor 901, a memory 902, and a communication port 903. The processor 901, the memory 902, and the communication port 903 are interconnected by the use of wiring. The memory 902 stores program instructions and data.

[0248] Memory 902 stores program instructions and data corresponding to the steps performed by the training device in the corresponding implementations shown in Figures 1 to 7A, 7B, and 7C.

[0249] The processor 901 is configured to perform steps performed by the training device in any one of the embodiments shown in Figures 1 to 7A, 7B, and 7C.

[0250] The communication port 903 may be configured to receive and transmit data, and to perform the acquisition and reception steps in any one of the embodiments shown in Figures 1 to 7A, 7B, and 7C.

[0251] In implementation, the training device may include more or fewer components than those shown in Figure 9. This is merely an example for illustrative purposes and is not limited to the present invention.

[0252] Those skilled in the art will clearly understand that, for the sake of convenience and simplicity, the detailed operating processes of the above systems, apparatuses and units are to be described by referring to the corresponding processes in embodiments of the above methods, and such details are not described herein.

[0253] It should be understood that, in some embodiments provided herein, the systems, apparatus, and methods disclosed may be implemented in other ways. For example, the embodiments of the apparatus described are merely examples. For example, the division into units is merely a logical functional division, and other division methods may be possible in actual implementation. For example, multiple units or components may be combined or integrated into another system, and some features may be ignored or not performed. In addition, the mutual coupling, direct coupling, or communication connection shown or discussed may be implemented by using some interfaces. Indirect coupling or communication connection between apparatus or units may be implemented in an electrical, mechanical, or other form.

[0254] A unit described as a separate part may or may not be physically separate. Also, a part shown as a unit may or may not be a physical unit, may be located in one position, or may be distributed among multiple network units. Some or all of these units may be selected based on the actual requirements for realizing the object of the solution means of the embodiment.

[0255] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may physically exist alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0256] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such understanding, although the technical solution means in the present application is essential, the part that contributes to the prior art, or all or part of the technical solution means, may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for instructing a computer device (which may be a personal computer, a server, or a network device) to execute all or some of the steps of the method described in the embodiments of the present application. The above storage medium includes any medium such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk that can store program codes. 。 [Other possible items] [Item 1] A model training method, The stage of acquiring training data; A step of using training data as input to a model and calculating parameters and obtaining calculated values ​​by using a first precision range during the training process of the model; and If the calculated value overflows the first precision range, the parameters are recalculated using the second precision range, and the model is repeatedly trained using the recalculated parameters, wherein the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range. A method for providing this. [Item 2] The aforementioned model includes multiple network structures, and the step of recalculating the parameters by using a second precision range is: The step of recalculating the parameters starting from the network structure of the first layer of the model by using the second accuracy range. The method described in item 1, including the method described in item 1. [Item 3] The aforementioned model includes multiple network structures, and the step of recalculating the parameters by using a second precision range is: The step of recalculating the parameters starting from the current network structure in which the calculated values ​​overflow by using the second precision range. The method described in item 1, including the method described in item 1. [Item 4] The aforementioned model includes multiple network structures, and the parameters are as follows: In the forward propagation process, the intermediate features calculated by the plurality of network structures or the value of the loss function of the model, where the intermediate features are the output features of any one of the plurality of network structures; and In the backpropagation process, the gradient calculated by the plurality of network structures, wherein the gradient includes the gradient of the intermediate features and / or the weight gradient of the model. The method described in any one of items 1 to 3, including one or more of the following. [Item 5] When the parameter includes the gradient calculated in the backpropagation process, the calculated value is obtained by dividing the gradient by a scaling factor, the scaling factor is used to reduce the probability that the gradient overflows, and the method further: A step of updating the scaling coefficient by using a first coefficient, wherein the updated scaling coefficient is used to replace the unupdated scaling coefficient and perform the next iterative training of the model, the first coefficient being a positive number less than 1, and the minimum value of the scaling coefficient being a preset threshold greater than or equal to 1. The method described in item 4, comprising: [Item 6] When the parameter includes the intermediate feature calculated in the forward propagation process, the step of recalculating the parameter by using a second precision range is: A step of calculating the intermediate features of the overflow layer using the second precision range, or calculating the intermediate features layer by layer starting from the network structure of the first layer, where the overflow layer is a network structure in which the calculated value of the intermediate features in the plurality of network structures overflows the first precision range. The method described in item 4, including the method described in item 4. [Item 7] When the parameter includes the value of the loss function of the model, the step of recalculating the parameter by using a second precision range is: A step of calculating the value of the loss function using the second precision range, or performing calculations layer by layer, starting from the network structure of the first layer, until the value of the loss function is obtained. The method described in item 4, including the method described in item 4. [Item 8] The step of training the model using the recalculated parameters is: In the Nth iteration of the training process of the model, a step is taken to obtain the number of overflows of the plurality of network structures in the model based on the first precision range, where N is a positive integer greater than or equal to 1; and If the number of overflows is greater than or equal to the second threshold, it is determined that the initial accuracy range in the next iterative training process will change from the first accuracy range to the second accuracy range, and the number of overflows is reset to zero. The method described in any one of items 1 through 7, including the method described in item 1 through 7. [Item 9] The method according to item 8, wherein the number of overflows includes the number of overflows of the plurality of network structures in the forward propagation process and / or the number of overflows of the plurality of network structures in the backpropagation process. [Item 10] Acquisition units configured to acquire training data; and A computing unit configured to use the aforementioned training data as input to a model and to calculate parameters and obtain calculated values ​​in the training process of the model by using a first precision range, wherein the computing unit is further configured to: recalculate the parameters by using a second precision range if the calculated values ​​overflow the first precision range, and to perform iterative training on the model one or more times by using the recalculated parameters, wherein the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range. A training device equipped with the following features. [Item 11] The training device according to item 10, wherein the model comprises multiple network structures, and the computing unit is specifically configured to recalculate the parameters starting from the first layer network structure of the model by using the second precision range. [Item 12] The training device according to item 10, wherein the calculation unit is specifically configured to recalculate the parameters starting from the current network structure in which the calculated values ​​overflow by using the second precision range. [Item 13] The aforementioned model includes multiple network structures, and the parameters are as follows: In the forward propagation process, the intermediate features calculated by the plurality of network structures or the value of the loss function of the model, where the intermediate features are the output features of any one of the plurality of network structures; and In the backpropagation process, the gradient calculated by the plurality of network structures, wherein the gradient includes the gradient of the intermediate features and / or the weight gradient of the model. A training device as described in any one of items 10 to 12, including one or more of the following. [Item 14] When the parameter includes the gradient calculated in the backpropagation process, the calculated value is a value obtained by dividing the gradient by a scaling factor, the scaling factor is used to reduce the probability that the gradient overflows, and the calculation unit further updates the scaling factor by using a first factor, where the updated scaling factor is used to replace the unupdated scaling factor and perform the next iterative training of the model, the first factor is a positive number less than 1, and the minimum value of the scaling factor is a preset threshold greater than or equal to 1, as described in item 13. [Item 15] When the parameters include the intermediate features calculated in the forward propagation process, the calculation unit specifically calculates the intermediate features of the overflow layer using the second precision range, or calculates the intermediate features layer by layer, starting from the network structure of the first layer, wherein the overflow layer is a network structure in which the calculated values ​​of the intermediate features in the plurality of network structures overflow the first precision range, as described in item 13. [Item 16] The training device according to item 13, wherein, when the parameter includes the value of the loss function of the model, the computing unit is configured to specifically compute the value of the loss function using the second precision range, or to perform calculations layer by layer, starting from the network structure of the first layer, until the value of the loss function is obtained. [Item 17] The computing unit is configured such that, in the Nth iteration of the training process of the model, it obtains the number of overflows of the plurality of network structures in the model based on the first precision range, where N is a positive integer greater than or equal to 1; and Specifically, the calculation unit is configured to determine that if the number of overflows is greater than or equal to a second threshold, the initial accuracy range in the next iterative training process will be changed from the first accuracy range to the second accuracy range, and to erase the number of overflows to zero. A training device as described in any one of items 10 through 16. [Item 18] The training device according to item 17, wherein the number of overflows includes the number of overflows of the plurality of network structures in the forward propagation process and / or the number of overflows of the plurality of network structures in the backpropagation process. [Item 19] A training device comprising a processor, wherein the processor is coupled to a memory, the memory is configured to store a program or instruction, and when the program or instruction is executed by the processor, the training device is capable of performing the method described in any one of items 1 to 9. [Item 20] A computer storage medium comprising computer instructions, wherein when the computer instructions are executed on a training end device, the training device is capable of performing the method described in any one of items 1 to 9. [Item 21] A computer program product in which, when executed on a computer, the computer is capable of performing the actions described in any one of items 1 to 9.

Claims

1. A model training method performed by a computer, The stage of acquiring training data; A step of using training data as input to a model and calculating parameters and obtaining calculated values ​​by using a first precision range during the training process of the model; and If the calculated value overflows the first precision range, the parameters are recalculated using a second precision range within the same iterative training process, and the model is trained iteratively one or more times using the recalculated parameters, wherein the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range. A method for providing this.

2. The method according to claim 1, wherein the adjustment from the first accuracy range to the second accuracy range within the same iterative training process is performed automatically in real time.

3. The model includes multiple network structures, and the step of recalculating the parameters by using a second precision range is: The step of recalculating the parameters starting from the network structure of the first layer of the model by using the second accuracy range. The method according to claim 1, including the method described in claim 1.

4. The model includes multiple network structures, and the step of recalculating the parameters by using a second precision range is: The step of recalculating the parameters starting from the current network structure where the calculated values ​​have overflowed by using the second precision range. The method according to claim 1, including the method described in claim 1.

5. The aforementioned model includes multiple network structures, and the parameters are as follows: Intermediate features calculated by the plurality of network structures in the forward propagation process or the value of the loss function of the model, where the intermediate features are the output features of any one of the plurality of network structures; and In the backpropagation process, the gradient calculated by the plurality of network structures, wherein the gradient includes the gradient of the intermediate features and / or the weight gradient of the model. The method according to any one of claims 1 to 4, comprising one or more of the above.

6. When the parameter includes the gradient calculated in the backpropagation process, the calculated value is obtained by dividing the gradient by a scaling factor, the scaling factor is used to reduce the probability that the gradient overflows, and the method further: A step of updating the scaling coefficient by using a first coefficient, wherein the updated scaling coefficient is used to replace the unupdated scaling coefficient and perform the next iterative training of the model, the first coefficient being a positive number less than 1, and the minimum value of the scaling coefficient being a preset threshold greater than or equal to 1. The method according to claim 5, comprising:

7. When the parameters include the intermediate features calculated in the forward propagation process, the step of recalculating the parameters by using a second precision range is: A step of calculating the intermediate features of the overflow layer using the second precision range, or calculating the intermediate features layer by layer starting from the network structure of the first layer of the model, where the overflow layer is a network structure in which the calculated value of the intermediate features in the plurality of network structures overflows the first precision range. The method according to claim 5, including the method described in claim 5.

8. When the parameter includes the value of the loss function of the model, the step of recalculating the parameter by using a second precision range is: A step of calculating the value of the loss function using the second precision range, or performing calculations layer by layer, starting from the network structure of the first layer of the model, until the value of the loss function is obtained. The method according to claim 5, including the method described in claim 5.

9. The step of training the model by using the recalculated parameters is: In the Nth iteration of the training process of the model, a step is taken to obtain the number of overflows of the multiple network structures in the model based on the first precision range, where N is a positive integer greater than or equal to 1; and If the number of overflows is greater than or equal to the second threshold, it is determined that the initial accuracy range in the next iterative training process should be changed from the first accuracy range to the second accuracy range, and the number of overflows is reset to zero. The method according to any one of claims 1 to 4, including the method described in any one of claims 1 to 4.

10. The method according to claim 9, wherein the number of overflows includes the number of overflows of the plurality of network structures in the forward propagation process and / or the number of overflows of the plurality of network structures in the backpropagation process.

11. Acquisition units configured to acquire training data; and A computing unit configured to use the aforementioned training data as input to a model, and to calculate parameters and obtain calculated values ​​by using a first precision range during the training process of the model, wherein the computing unit is further configured to recalculate the parameters by using a second precision range within the same iterative training process if the calculated values ​​overflow the first precision range, and to perform iterative training on the model one or more times by using the recalculated parameters, wherein the second precision range includes the first precision range, or the second precision range partially overlaps with the first precision range. A training device equipped with the following features.

12. The training device according to claim 11, wherein the model comprises a plurality of network structures, and the computing unit is specifically configured to recalculate the parameters starting from the first layer network structure of the model by using the second precision range.

13. The training device according to claim 11, wherein the calculation unit is specifically configured to recalculate the parameters starting from the current network structure in which the calculated values ​​have overflowed by using the second precision range.

14. The aforementioned model includes multiple network structures, and the parameters are as follows: Intermediate features calculated by the plurality of network structures in the forward propagation process or the value of the loss function of the model, where the intermediate features are the output features of any one of the plurality of network structures; and In the backpropagation process, the gradient calculated by the plurality of network structures, wherein the gradient includes the gradient of the intermediate features and / or the weight gradient of the model. A training device according to any one of claims 11 to 13, comprising one or more of the above.

15. When the parameter includes the gradient calculated in the backpropagation process, the calculated value is a value obtained by dividing the gradient by a scaling factor, the scaling factor is used to reduce the probability that the gradient overflows, and the calculation unit further updates the scaling factor by using a first coefficient, where the updated scaling factor is used to replace the unupdated scaling factor and perform the next iterative training of the model, the first coefficient is a positive number less than 1, and the minimum value of the scaling factor is a preset threshold greater than or equal to 1, the training device according to claim 14.

16. When the parameters include the intermediate features calculated in the forward propagation process, the calculation unit specifically calculates the intermediate features of the overflow layer using the second precision range, or calculates the intermediate features layer by layer, starting from the network structure of the first layer of the model, wherein the overflow layer is a network structure in which the calculated values ​​of the intermediate features in the plurality of network structures overflow the first precision range, the training device according to claim 14.

Images