Model training method and device and storage medium

Abstract

The invention provides a model training method and device and a storage medium. The model training method comprises the steps that training equipment firstly obtains a to-be-trained target model, wherein the target model comprises a target operator at least supporting a first execution mode; and then, the training equipment performs iterative training on the target model, and adopts a first execution mode to execute the target operator in the training process. In the mode, the training device analyzes the applicability of each candidate parameter by acquiring the current precision parameter corresponding to the target operator and a plurality of candidate precision parameters and combining the input data of the target operator, so as to judge whether the target precision parameter which enables the training effect of the target model to be better than that of the current precision parameter exists or not. And if yes, the training device dynamically adjusts the expression precision of the input data based on the target precision parameter to generate first target data adaptive to the current training stage, and executes a target operator based on the data. And the calculation precision of the target operator is adaptively and dynamically adjusted.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a model training method, device, and storage medium. Background Art

[0002] In the field of deep learning model training, mixed-precision training technology has become a mainstream solution to improve training efficiency and reduce computing resource consumption. Mixed-precision training allows different operators (OPs) in the model to be trained using different precision quantization strategies during training. By combining high-precision (such as FP32) and low-precision (such as BF16, FP16, FP8, FP4, etc.) quantization strategies, computational precision is minimized and model training speed is increased without affecting model convergence. The precision parameters configured in mixed-precision training directly affect the model's convergence, convergence speed, and post-training model performance.

[0003] Current mixed-precision training solutions typically rely on manual experience to statically configure the precision parameters of each operator in the model. For example, workers can manually specify the quantization granularity and floating-point format of different operators based on the model structure and training objectives. However, in actual applications, precision parameters configured based on manual experience often fail to achieve optimal results, resulting in reduced model training efficiency or unsatisfactory model convergence performance. Finding the optimal precision parameters requires multiple attempts using different precision parameters, further reducing the effectiveness of model training.

[0004] The content of the background technology section is merely information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the Invention

[0005] This specification provides a model training method, device, and storage medium that can adaptively adjust the calculation accuracy of the target operator when training the target model to ensure the training efficiency and model performance of the model.

[0006] In order to achieve the above objectives, the embodiments of this specification adopt the following technical solutions:

[0007] In a first aspect, the present specification provides a model training method, which is applied to a training device, and the method includes: obtaining a target model to be trained, the target model including a target operator, and the execution mode of the target operator including at least a first execution mode; and iteratively training the target model, and executing the target operator in the first execution mode during the training process, wherein the execution process of the target operator in the first execution mode includes: obtaining input data of the target operator, and obtaining a current precision parameter and multiple candidate precision parameters corresponding to the target operator, based on the input data, determining whether there is a target precision parameter among the multiple candidate precision parameters, wherein the training effect of the training device on the target model under the target precision parameter is better than the training effect of the target model under the current precision parameter, and in the case that the target precision parameter exists, adjusting the expression accuracy of the training device for the input data based on the target precision parameter to obtain first target data, and executing the target operator based on the first target data.

[0008] In some embodiments, the target accuracy parameter satisfies at least one of the following: the degree of influence of the target accuracy parameter on the convergence performance of the target model is lower than the degree of influence of the current accuracy parameter on the convergence performance of the target model; or, the execution time of the target operator of the training device under the target accuracy parameter is less than the execution time of the target operator under the current accuracy parameter.

[0009] In some embodiments, based on the input data, determining whether there is a target accuracy parameter among the multiple candidate accuracy parameters includes: predicting a first degree of influence of the current accuracy parameter on the convergence performance of the target model based on the input data; and in a case where the first degree of influence is greater than a preset degree of influence, determining whether there is a target accuracy parameter that meets a first condition among the multiple candidate accuracy parameters, the first condition including: the degree of influence on the convergence performance of the target model is less than or equal to the preset degree of influence.

[0010] In some embodiments, determining whether there is a target precision parameter that meets the first condition among the multiple candidate precision parameters includes: determining a first set among the multiple candidate precision parameters, the calculation precision corresponding to each candidate precision parameter in the first set being greater than the calculation precision corresponding to the current precision parameter; and determining whether there is a target precision parameter that meets the first condition in the first set.

[0011] In some embodiments, determining whether there is a target accuracy parameter that meets the first condition in the first set includes: traversing each candidate accuracy parameter in the first set in order of calculation accuracy from low to high, and for the i-th candidate accuracy parameter in the traversal: based on the input data, predicting the second degree of influence of the i-th candidate accuracy parameter on the convergence performance of the target model; if the second degree of influence is less than or equal to the preset degree of influence, stopping the traversal and determining the i-th candidate accuracy parameter as the target accuracy parameter; if the second degree of influence is greater than the preset degree of influence, continuing to traverse the first set.

[0012] In some embodiments, the method further includes: when the first degree of influence is less than or equal to a preset degree of influence, determining whether there is a target accuracy parameter that meets a second condition among the multiple candidate accuracy parameters, wherein the second condition includes: the execution time of the target operator by the training device under the target accuracy parameter is less than the execution time of the target operator under the current accuracy parameter.

[0013] In some embodiments, determining whether there is a target precision parameter that meets the second condition among the multiple candidate precision parameters includes: determining a second set among the multiple candidate precision parameters, the calculation precision corresponding to each candidate precision parameter in the second set being less than the calculation precision corresponding to the current precision parameter; and determining whether there is a target precision parameter that meets the second condition in the second set.

[0014] In some embodiments, determining whether there is a target accuracy parameter that meets the second condition in the second set includes: traversing each candidate accuracy parameter in the second set in order of calculation accuracy from high to low, and for the jth candidate accuracy parameter in the traversal: based on the input data, predicting the third degree of influence of the jth candidate accuracy parameter on the convergence performance of the target model; if the third degree of influence is greater than the preset degree of influence, stopping the traversal, and determining the j-1th candidate accuracy parameter as the target accuracy parameter based on the value of j, or determining that the target accuracy parameter does not exist in the second set; if the third degree of influence is less than or equal to the preset degree of influence, continuing to traverse the second set.

[0015] In some embodiments, the current precision parameter includes: at least one of a first floating-point format and a first quantization granularity, and the predicting of the first degree of influence of the current precision parameter on the convergence performance of the target model based on the input data includes: determining a first quantization range corresponding to the input data based on the first floating-point format; dividing the input data into multiple first sub-data based on the first quantization granularity, and quantizing the multiple first sub-data respectively based on the first quantization range to obtain quantization results corresponding to the multiple first sub-data; obtaining quantization data corresponding to the input data based on the quantization results corresponding to the multiple first sub-data; and predicting the first degree of influence of the current precision parameter on the convergence performance of the target model based on the quantization data.

[0016] In some embodiments, predicting the first degree of influence of the current precision parameter on the convergence performance of the target model based on the quantization data includes: determining the first degree of influence of the current precision parameter on the convergence performance of the target model based on an overflow situation of the quantization data or at least one of the quantization errors of the quantization data.

[0017] In some embodiments, the target accuracy parameters include: at least one of a second floating-point format and a second quantization granularity, and adjusting the expression accuracy of the input data of the training device based on the target accuracy parameters to obtain the first target data includes: determining the second quantization range corresponding to the input data based on the second floating-point format; dividing the input data into multiple second sub-data based on the second quantization granularity, and quantizing the multiple second sub-data respectively based on the second quantization range to obtain quantization results corresponding to the multiple second sub-data; and obtaining the first target data based on the quantization results corresponding to the multiple second sub-data.

[0018] In some embodiments, the method further includes: in the absence of the target accuracy parameter, adjusting the expression accuracy of the input data by the training device based on the current accuracy parameter to obtain second target data, and executing the target operator based on the second target data.

[0019] In some embodiments, the execution mode of the target operator also includes a second execution mode, the training process includes multiple iterative rounds, and the use of the first execution mode to execute the target operator during the training process includes: in multiple rounds of N in the multiple iterative rounds, the first execution mode is used to execute the target operator, N is an integer greater than 1, and in other rounds in the multiple iterative rounds, the second execution mode is used to execute the target operator; the execution process of the target operator in the second execution mode includes: and obtaining the input data of the target operator, and obtaining the current precision parameter corresponding to the target operator, adjusting the expression accuracy of the input data of the training device based on the current precision parameter to obtain second target data, and executing the target operator based on the second target data.

[0020] In some embodiments, in rounds that are multiples of N in the multiple iterative rounds, the method further includes: recording the model parameters of the target model in the current round; in the N*kth round, in the presence of the target accuracy parameters, adjusting the expression accuracy of the training device for the input data based on the target accuracy parameters to obtain first target data, including: in the presence of the target accuracy parameters, updating the model parameters of the target model to the model parameters recorded in the N*(k-1)th round, where k is an integer greater than 1, and adjusting the expression accuracy of the training device for the input data based on the updated model parameters and the target accuracy parameters to obtain the first target data.

[0021] In a second aspect, this specification also provides a training device, comprising: at least one storage medium storing at least one instruction set for training a target model; and at least one processor communicatively connected to the at least one storage medium, wherein when the training device is running, the at least one processor reads the at least one instruction set and implements a method as described in any one of the first aspects according to the instructions of the at least one instruction set.

[0022] In a third aspect, this specification also provides a computer-readable non-volatile storage medium, wherein the computer-readable non-volatile storage medium stores at least one instruction set, and when the at least one instruction set is executed by at least one processor, it implements the method as described in any one of the first aspects.

[0023] Other features of the model training method, device, and storage medium provided in this specification are partially listed in the following description. The creative aspects of the model training method, device, and storage medium provided in this specification can be fully explained by practicing or using the methods, devices, and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic diagram of an application scenario of a model training method provided according to an embodiment of this specification is shown;

[0026] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0027] Figure 3 A flow chart of a model training method provided according to an embodiment of this specification is shown;

[0028] Figure 4 A schematic diagram illustrating executing a target operator in a first execution mode according to an embodiment of this specification is shown; and

[0029] Figure 5 A flowchart for determining target accuracy parameters according to an embodiment of this specification is shown. DETAILED DESCRIPTION

[0030] The following description provides specific application scenarios and requirements for this specification, with the goal of enabling those skilled in the art to make and use the contents of this specification. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but is intended to be accorded the broadest scope consistent with the claims.

[0031] The terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. For example, as used herein, the singular forms "a," "an," and "the" may also include the plural forms unless the context clearly indicates otherwise. When used in this specification, the terms "comprise," "include," and / or "contain" are intended to refer to the presence of the associated integers, steps, operations, elements, and / or components, but do not preclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups or the addition of other features, integers, steps, operations, elements, components, and / or groups in the system / method.

[0032] These and other features of this specification, as well as the operation and function of the associated elements of the structure, and the economical assembly and manufacture of the components, can be significantly improved with consideration of the following description. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be expressly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0033] The flowcharts used in this specification illustrate operations implemented by systems according to some embodiments of the present specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. Rather, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0034] For the convenience of description, the terms that will appear in the following text of this specification are first explained.

[0035] Term 1: Operator. An operator (OP) is also called a computational operation. A model consists of multiple modules, and the computation of a module involves at least one operator. For example, a linear module may include a matrix multiplication operator, or a matrix multiplication operator and a matrix addition operator. An operator can be calculated with different precisions.

[0036] The following is an introduction to the application scenarios of this manual.

[0037] The technical solutions provided in this specification are applicable to scenarios where mixed-precision training is performed on models. For example, when using a training device to perform mixed-precision training on a deep learning model, the technical solutions provided in this specification can be used. The deep learning model can be a natural language processing model, a computer vision processing model, etc. This specification does not limit the type of model.

[0038] Currently, when using training equipment to perform mixed-precision training on deep learning models, staff must manually configure the quantization granularity and floating-point format of different operators. This means statically configuring the precision parameters of each operator in the model. However, deep learning models have different precision requirements at different training stages. Static configuration of precision parameters cannot adapt to different training stages, potentially reducing model training efficiency or resulting in suboptimal performance.

[0039] In this regard, this specification provides a model training method that can be executed by a training device. First, the training device can obtain a target model to be trained, the target model includes a target operator, and the execution mode of the target operator includes at least a first execution mode. Then, the training device can iteratively train the target model and execute the target operator in the first execution mode during the training process, wherein, when the target operator is in the first execution mode, the training device can obtain the current precision parameter and multiple candidate precision parameters corresponding to the target operator, and determine whether the target precision parameter exists in the multiple candidate precision parameters based on the input data of the target operator, and the training effect of the training device on the target model under the target precision parameter is better than the training effect of the target model under the current precision parameter. If it exists, the training device can adjust the expression accuracy of the training device for the input data based on the target precision parameter to obtain the first target data, and execute the target operator based on the first target data.

[0040] In the solution provided herein, a training device can, in a first execution mode of a target operator, determine whether a target precision parameter exists among multiple candidate precision parameters based on the input data of the target operator. When the target precision parameter exists, the training device can adjust the accuracy of the training device's expression of the input data based on the target precision parameter to obtain first target data, and execute the target operator based on the first target data. This achieves adaptive dynamic configuration of the computational precision of the target operator, ensuring that the computational precision of the target operator matches the precision requirements of the current training phase, thereby ensuring the training efficiency of the model and the performance of the trained model.

[0041] It should be noted that the above scenario of mixed-precision training of deep learning models is only one of the multiple usage scenarios provided in this specification. The model training method provided in this specification can be applied not only to the scenario of mixed-precision training of deep learning models, but also to all scenarios involving mixed precision, such as mixed-precision training of multimodal models, molecular dynamics simulation using mixed precision, fluid mechanics modeling using mixed precision, etc. Those skilled in the art should understand that when the model training method provided in this specification is applied to other scenarios, its implementation method and technical effects are similar.

[0042] Figure 1 FIG2 shows a schematic diagram of an application scenario of a model training method provided according to an embodiment of this specification. Figure 1 As shown, the application scenario 100 may include a training device 11, which includes a target model, an execution engine, a precision parameter library, and a precision parameter selector. The target model includes a target operator, and the execution engine may call the precision parameter library and the precision parameter selector and interact with the target operator to determine and configure target precision parameters for the target operator.

[0043] The training device 11 can be a device or device cluster that provides various model training functions. For example, the training device 11 can be a personal computer, a supercomputer, a server, a server cluster, a cloud server, etc. The training device 11 can execute the model training method. In this case, the training device 11 can store data or instructions for executing the model training method described in this specification and can execute or be used to execute the data or instructions. In some embodiments, the training device 11 can include a hardware device with data information processing capabilities and the necessary programs required to drive the hardware device to work.

[0044] It should be understood that Figure 1 The number of training devices 11 in FIG. 1 is merely illustrative. Any number of training devices 11 may be provided according to implementation requirements.

[0045] Figure 2 The hardware structure diagram of a computing device provided according to an embodiment of this specification is shown. The computing device can be used as Figure 1 The training device 11 in the embodiment executes the model training method described in this specification.

[0046] like Figure 2 As shown, computing device 200 may include at least one storage medium 230 and at least one processor 220. In some embodiments, computing device 200 may further include communication port 250 and internal communication bus 210. Computing device 200 may also include I / O component 260.

[0047] The internal communication bus 210 can connect various system components, such as the storage medium 230 , the processor 220 , the communication port 250 , and the I / O component 260 .

[0048] I / O components 260 support input / output between computing device 200 and other components.

[0049] The communication port 250 is used for data communication between the computing device 200 and the outside world. For example, the communication port 250 can be used for data communication between the computing device 200 and a network. The communication port 250 can be a wired communication port or a wireless communication port.

[0050] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 235. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set may include computer program code, which may include a program, routine, object, component, data structure, procedure, module, etc.

[0051] At least one processor 220 may be communicatively connected to at least one storage medium 230. When the computing device 200 is running, the at least one processor 220 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the model training method provided in this specification. The processor 220 may execute the steps included in the model training method. The processor 220 may be in the form of one or more processors. In some embodiments, the processor 220 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0052] For illustrative purposes only, the computing device 200 shown in the accompanying drawings only has one processor 220. However, it should be noted that the computing device 200 described herein may also include multiple processors. Therefore, the operations and / or method steps disclosed herein may be performed by a single processor or jointly by multiple processors. For example, if the processor 220 of the computing device 200 is described herein as performing steps A and B, it should be understood that steps A and B may also be performed jointly or separately by two different processors 220 (e.g., a first processor performing step A and a second processor performing step B, or a first and a second processor performing steps A and B together).

[0053] Figure 3 FIG2 is a flow chart of a model training method provided according to an embodiment of the present specification. As before, the training device 11 can execute the model training method provided in the present specification.

[0054] like Figure 3 As shown, the model training method may include:

[0055] S310: Obtain a target model to be trained, where the target model includes a target operator, and the execution mode of the target operator includes at least a first execution mode.

[0056] In some embodiments, the target model may include multiple submodules, and each submodule may include at least one operator. For example, taking the target model as a deep learning model as an example, the submodule may include a convolution layer, a fully connected layer, a normalization layer, etc. Among them, commonly used operators include convolution operators, normalization operators, matrix multiplication operators, matrix addition operators, etc. It should be noted that any of the operators in the above examples can be used as the target operator, and the execution process of the model training method provided in this specification is similar for different types of operators.

[0057] When training a target model, the lower the data precision of the target operator, the faster the execution speed, which improves the model training efficiency. However, if the data precision of the target operator is too low, it can affect the model's convergence performance. For example, the model may not converge, converge more slowly, or converge less efficiently.

[0058] To this end, the execution modes of the target operator referred to in this specification should include at least a first execution mode, which is a mode for dynamically adjusting the calculation accuracy of the target operator. In the first execution mode, the training device can dynamically adjust the accuracy parameters corresponding to the target operator to achieve better model training results.

[0059] S320: Iteratively train the target model, and execute the target operator in the first execution mode during the training process, wherein the execution process of the target operator in the first execution mode includes S321-S323.

[0060] The process of iteratively training the target model by the training device may include multiple iterations. In some embodiments, the training device may execute the target operator using the first execution mode in each iteration. In some embodiments, the training device may execute the target operator using the first execution mode in some iterations.

[0061] Figure 4 FIG2 shows a schematic diagram of executing a target operator in a first execution mode according to an embodiment of this specification. Figure 4 Each step S321-S323 involved in the first execution mode is described in detail.

[0062] S321: Obtain input data of the target operator, and obtain the current precision parameter and multiple candidate precision parameters corresponding to the target operator. Figure 4,In a round of iterative training of the target model, when the training device detects that the target operator needs to be executed, it can obtain the input data and current precision parameters of the target operator, and obtain multiple candidate precision parameters from the precision parameter library based on the type of the target operator.

[0063] In some embodiments, the input data of the target operator may be one or more tensors, and the precision parameter of the target operator may include at least one of a floating-point format and a quantization granularity of the target operator.

[0064] As an example, the precision parameter may include a floating point format and a quantization granularity corresponding to the floating point format. Among them, commonly used floating point (Floating Point, fp) number formats may include: single-precision floating point (fp32), half-precision floating point (fp16), brain floating point 16-bit (bf16), 8-bit floating point (fp8) and 4-bit floating point (fp4). Among them, fp32 can be used in the initial stage of model training because it can ensure higher calculation accuracy and avoid the loss of accuracy causing the convergence performance of the model to be damaged. Bf16 or fp16 can be used in the model inference stage and some scenarios where the accuracy requirements are not very high. Using bf16 or fp16 can increase the calculation speed and ensure the convergence performance of the model. FP8 can be used in scenarios where memory usage needs to be further reduced and computations need to be accelerated, such as inference tasks on edge devices and large-scale data centers. FP8 provides higher computational speeds, but in some cases may affect model convergence performance. FP4 can be used for model compression and in scenarios with extremely limited resources, such as inference tasks on ultra-low-power edge devices. FP4 provides higher computational speeds and lower energy consumption, but may significantly impact model convergence performance.

[0065] The quantization granularity is the unit used to quantize the input data. From large to small, the quantization granularity can include tensor, block, and tile.

[0066] For example, assuming that the input data is a tensor and the quantization granularity is the tensor granularity, the training device can calculate a uniform scaling factor based on the entire input tensor, and quantize the input tensor based on this scaling factor to obtain quantized data.

[0067] When the quantization granularity is block granularity, the training device can split the input tensor into multiple 128*128 blocks and calculate the corresponding scaling factor for each block. During quantization, each block is quantized using its corresponding scaling factor. The collection of quantization results after multiple blocks are quantized is the quantized data.

[0068] When the quantization granularity is tile granularity, the training device can divide the input tensor into multiple 1*128 tiles (or 128*1 tiles) and calculate the corresponding scaling factor for each tile. During quantization, each tile is quantized using its corresponding scaling factor. The set of quantization results after multiple tiles are quantized is the quantized data.

[0069] For example, when the floating-point format is "fp4" or "fp8," you can adjust the computational accuracy by configuring the quantization granularity. A larger quantization granularity reduces computational complexity and speeds up calculations, but also results in a greater loss of accuracy. This means that the resulting data from quantizing the input data will have a greater impact on the convergence performance of the target model. When the floating-point format is "bf16," "fp16," or "fp32," these formats can represent a sufficiently large range of values, eliminating the need to calculate a scaling factor during the quantization process. In this case, the training device can default to the "tensor granularity" quantization granularity for "bf16," "fp16," and "fp32." In the examples below, even when the precision parameter "bf16" or "fp32" is not specified, the default quantization granularity is "tensor granularity."

[0070] In some embodiments, a plurality of candidate precision parameters corresponding to each operator may be pre-stored in the precision parameter library. For example, assuming that the target operator is a matrix multiplication operator, the training device may obtain a plurality of candidate precision parameters corresponding to the matrix multiplication operator in the precision parameter library. For example, if the current precision parameter corresponding to the matrix multiplication operator is "fp8, tensor granularity", the plurality of candidate precision parameters corresponding to the matrix multiplication operator may include: "fp4, tile granularity", "fp4, block granularity", "fp4, tensor granularity", "fp8, block granularity", "fp8, tile granularity", "bf16", "fp32", etc.

[0071] S322: Based on the input data, determine whether there is a target accuracy parameter among the multiple candidate accuracy parameters, wherein the training effect of the training device on the target model under the target accuracy parameter is better than the training effect of the target model under the current accuracy parameter.

[0072] In this specification, the training effect of the target model may include the effect of at least one dimension of convergence performance and training speed.

[0073] In some embodiments, the training effect of the training device on the target model under the target accuracy parameters is better than the training effect of the target model under the current accuracy parameters, which may include at least one of the following (1) or (2).

[0074] (1) The influence of the target accuracy parameter on the convergence performance of the target model is lower than the influence of the current accuracy parameter on the convergence performance of the target model. That is, in terms of convergence performance, the training effect corresponding to the target accuracy parameter is better than the training effect corresponding to the current accuracy parameter.

[0075] (2) The execution time of the target operator by the training device under the target precision parameters is less than the execution time of the target operator under the current precision parameters. That is, in terms of training speed / training time, the training effect corresponding to the target precision parameters is better than the training effect corresponding to the current precision parameters.

[0076] In some embodiments, the training device adjusts the expression accuracy of the input data based on the target accuracy parameter, and the obtained first target data can make the convergence performance of the model better, or can make the training speed of the model faster when the convergence performance of the model meets the requirements.

[0077] As an example, when the training device determines that the current precision parameter corresponding to the target operator has too great an impact on the convergence performance, the precision parameter can be adjusted higher to ensure that the convergence performance is not affected; when the training device determines that the current precision parameter corresponding to the target operator has a sufficient impact on the convergence performance, the precision parameter can be lowered to increase the training speed of the model while ensuring that the convergence performance is not affected.

[0078] Figure 5 A flowchart for determining target accuracy parameters according to an embodiment of this specification is shown.

[0079] In some embodiments, reference Figure 4 and Figure 5 The training device can determine whether a target precision parameter exists based on the input data of the target operator, the current precision parameter corresponding to the target operator, and multiple candidate precision parameters corresponding to the target operator through the precision parameter selector.

[0080] In some embodiments, the training device can predict the first degree of influence of the current precision parameter on the convergence performance of the target model based on the input data through the precision parameter selector. As an example, the current precision parameter includes: a first floating-point format and a first quantization granularity. The training device can determine the first quantization range corresponding to the input data based on the first floating-point format. Then, the training device divides the input data into multiple first sub-data based on the first quantization granularity, and quantizes the multiple first sub-data based on the first quantization range to obtain quantization results corresponding to the multiple first sub-data. Then, the training device obtains quantization data corresponding to the input data based on the quantization results corresponding to the multiple first sub-data. Finally, the training device can predict the first degree of influence of the current precision parameter on the convergence performance of the target model based on the quantization data.

[0081] In some embodiments, the degree of influence of the precision parameter on the convergence performance of the target model can be determined based on at least one of the overflow of the quantized data or the error between the quantized data and the input data to determine the first degree of influence of the current precision parameter on the convergence performance of the target model.

[0082] Quantization is the operation of mapping data from a continuous range of values to a finite number of discrete values. Quantization data overflow occurs when the original data before quantization exceeds the range of values that can be represented after quantization (either too large or too small), resulting in an inaccurate quantization (mapping to the minimum or maximum discrete values) and an incorrect quantization result. The overflow of quantization data can be measured as the proportion of the overflowed quantization data to the total quantization data.

[0083] The quantization error of quantized data refers to the difference between the original data before quantization and the data obtained after quantization. The quantization error includes absolute error and relative error. The absolute error refers to the difference between the data before and after quantization, and the relative error refers to the ratio of the absolute error to the original data before quantization.

[0084] As an example, assume that the input data is a tensor X = [2.0 × 10 -3 , 1.0×10 -10 , 5.0×10 -9 , 3.0×10 -8 ]; The current precision parameter is "fp8, tensor granularity".

[0085] The format of fp8 is 1-bit sign, 4-bit exponent, and 3-bit mantissa, which is E4M3, where the exponent bias is 7. As an example, E4M3 can be written as "S.0000.000", where "S" represents the sign bit, ".0000" represents the exponent bit, and ".000" represents the mantissa. In this case, the maximum value of E4M3 is "S.1111.1112", which is 1.75*2. 8 =448, the minimum value of E4M3 is "S.0000.0012" = 2 -9 .

[0086] When the training device performs low-precision quantization on tensor X based on the current precision parameter, it can determine the quantization range corresponding to tensor X as [0, 448] based on the first floating-point format (fp8). That is, the training device can quantize tensor X to the range [0, 448]. Then, because the first quantization granularity is the "tensor granularity", the training device does not divide tensor X. That is, tensor X itself is the first sub-data, and the quantization result of tensor X by the training device is the quantized data corresponding to the input data.

[0087] As an example, the quantization process of the tensor X by the training device may include: determining a scaling factor (scale), scaling data, and equivalent quantization to fp8 E4M3.

[0088] The training device can determine the scaling factor based on the maximum value of the tensor X and the maximum value of the quantization range. For example, the maximum value of the tensor X is 2.0×10 -3 , the maximum value of the quantization range is 448, so the scaling factor scale = 448 / 2.0×10 -3 , which equals 224,000.

[0089] Then, the training device can scale the data in tensor X according to the scaling factor, that is, multiply each data in tensor X by the scaling factor to obtain the scaled data X scaled =[448.0,2.24×10 -5 , 1.12×10 -3 , 6.72×10 -3 ].

[0090] Finally, the training device can use the scaled data X scaled Equivalent quantization to fp8 E4M3. As an example, since the minimum value of E4M3 is 2 -9 (approximately equal to 1.953×10 -3 ), so it is less than 1.953×10 -3 The value will be equivalent to 0. scaled Chinese data "2.24×10 -5” and “1.12×10 -3 "are all less than 1.953×10 -3 (i.e. underflow), so they are all equivalent to 0. 6.72×10 -3 Since the mantissa is too long and needs to be truncated, it is equivalent to 0.0067 (6.7×10 -3 ) is equivalent to the quantitative data X quantized =[448.0, 0, 0, 0.0067]. Equivalent quantized data X quantized That is the quantitative data corresponding to the input data.

[0091] As an example, the training device can be based on the equivalent quantized data X quantized Determine the first degree of influence of the current precision parameter on the convergence performance of the target model. For example, when the training device uses the overflow of quantized data to characterize the degree of influence of the current precision parameter on the convergence performance of the target model, the proportion of the overflowed data to the quantized data can be used as a convergence indicator to characterize the degree of influence of the current precision parameter on the convergence performance of the target model. Referring to the above example, the number of overflowed data is 2, accounting for 50% (2 / 4=0.5) of the total amount of quantized data (4). The overflow of quantized data is: the proportion of overflowed data to quantized data is 50%.

[0092] Alternatively, when the training device uses the quantization error of the quantized data to characterize the influence of the current precision parameter on the convergence performance of the target model, the relative quantization error of the quantized data can be used as a convergence indicator to characterize the influence of the current precision parameter on the convergence performance of the target model. For example, referring to the above example, X quantized =[448.0, 0, 0, 0.0067], dequantize the equivalent quantized data and get X dequant =[2.0×10 -3 ,0,0,3.0×10 -8 ].

[0093] X dequant Compared to tensor X, "2.0×10 -3 ” and “3.0×10 -8 The relative quantization error of " is 0% (no error), "1.0×10 -10 ” and “5.0×10 -9 The relative quantization error of " is 100% (that is, the value is equivalent to 0), then the quantization error of the quantized data can be the average value of the quantization errors of each data, that is, 25%.

[0094] In some embodiments, when the first influence degree is greater than a preset influence degree, the training device may determine whether there is a target accuracy parameter among multiple candidate accuracy parameters that satisfies a first condition. The first condition may be that the influence degree on the convergence performance of the target model is less than or equal to the preset influence degree.

[0095] In some embodiments, the preset impact level can be a pre-set threshold that does not affect the model convergence speed. If the first impact level is greater than the preset impact level, it means that the current precision parameter is too low, which will have a significant impact on the convergence performance of the target model. In this case, you can try to increase the precision parameter, that is, you need to determine a target precision parameter that meets the first condition from multiple candidate precision parameters to achieve better convergence performance of the target model.

[0096] As an example, when the training device uses the overflow of quantized data to characterize the degree of influence of the current precision parameter on the convergence performance of the target model, the preset degree of influence can be a preset threshold value of the proportion of overflow data to quantized data. For example, the preset threshold value of the proportion of overflow data to quantized data can be set to 0.1%, that is, during the quantization process, the proportion of overflow data to quantized data should be less than one thousandth. Among them, the preset threshold value of the proportion of overflow data to quantized data can also be set to 0.2%, 1%, 2%, etc., and its specific value can be determined according to the actual situation during application, and this manual does not impose any restrictions on this.

[0097] Alternatively, in other embodiments, when the training device uses the quantization error of the quantized data to characterize the degree of influence of the current precision parameter on the convergence performance of the target model, the preset degree of influence can be a preset quantization error threshold. For example, the preset quantization error threshold can be set to 1%, wherein the preset quantization error threshold can also be set to 0.5%, 2%, 3%, etc., and its specific value can be determined according to the actual situation during application, and this manual does not impose any restrictions on this.

[0098] In some embodiments, the training device can determine a first set from multiple candidate precision parameters, where the calculation accuracy corresponding to each candidate precision parameter in the first set is greater than the calculation accuracy corresponding to the current precision parameter; and determine whether there is a target precision parameter in the first set that meets the first condition.

[0099] As an example, the computational precision of the candidate precision parameters that meet the first condition must be greater than the current precision parameter. Therefore, the training device can determine M candidate precision parameters whose computational precision is greater than the current precision parameter from the multiple candidate precision parameters corresponding to the target operator in the precision parameter library to obtain a first set, where M is an integer greater than or equal to 1. It should be noted that the precision parameters include floating-point format and quantization granularity. For two precision parameters, when the floating-point format is the same, the one with smaller quantization granularity has greater computational precision; when the floating-point formats are different, the one with more floating-point digits has greater computational precision. For example, for the two precision parameters of "fp8, tensor granularity" and "fp8, block granularity", their floating-point formats are the same, both "fp8", and "tensor granularity" is greater than "block granularity". Then, of the two precision parameters, the computational precision of "fp8, tensor granularity" is less than that of "fp8, block granularity". For "fp8, block granularity" and "bf16", "bf16" is a 16-bit floating point number and "fp8" is an 8-bit floating point number. Therefore, of these two precision parameters, the calculation accuracy of "bf16" is greater than that of "fp8, block granularity".

[0100] In some embodiments, the training device can traverse each candidate precision parameter in the first set in order of calculation accuracy from low to high, and for the i-th candidate precision parameter in the traversal: based on the input data, predict the second degree of influence of the i-th candidate precision parameter on the convergence performance of the target model; if the second degree of influence is less than or equal to the preset degree of influence, stop the traversal and determine the i-th candidate precision parameter as the target precision parameter; if the second degree of influence is greater than the preset degree of influence, continue to traverse the first set.

[0101] In some embodiments, the training device may first arrange the candidate precision parameters in the first set from low to high according to the calculation precision. After the arrangement, the calculation precision between two adjacent candidate precision parameters differs by one level.

[0102] Then, the training device can traverse from low calculation accuracy to high calculation accuracy, and the calculation accuracy of the candidate precision parameters in the latter round of traversal can be one level higher than the calculation accuracy of the candidate precision parameters in the previous round of traversal. During the traversal process, the training device can make predictions based on the input data and the traversed candidate precision parameters (i.e., the i-th candidate precision parameters) to obtain the second degree of influence of the i-th candidate precision parameter on the convergence performance of the target model. Among them, the method for predicting the second degree of influence is similar to the method for obtaining the first degree of influence based on the input data and the current precision parameters in the above example, and will not be repeated here.

[0103] In this embodiment, candidate precision parameters are traversed in ascending order of computational accuracy, with low-precision candidate precision parameters prioritized. Combined with the second degree of influence of the predicted candidate precision parameters on the convergence performance of the target model, the lowest candidate precision parameters that meet the preset influence are selected. If the current precision parameters of the target operator do not meet the convergence requirements, a set of precision parameters that meet the convergence requirements and execute the fastest can be automatically selected, enabling dynamic adjustment of the target model's precision parameters and improving both the training effect and efficiency of the target model.

[0104] In some embodiments, when the first degree of influence is less than or equal to a preset degree of influence, it is determined whether there is a target accuracy parameter that meets the second condition among multiple candidate accuracy parameters, wherein the second condition includes: the execution time of the target operator of the training device under the target accuracy parameter is less than the execution time of the target operator under the current accuracy parameter.

[0105] In some embodiments, if the first impact level is less than the preset impact level, it indicates that the current precision parameter meets the convergence requirements of the target model and will not significantly affect the convergence performance of the target model. In this case, you can try to lower the precision parameter, that is, determine a target precision parameter that meets the second condition from multiple candidate precision parameters, so as to improve the execution speed of the target model without significantly affecting the convergence performance.

[0106] In some embodiments, the training device can determine a second set from multiple candidate precision parameters, where the calculation accuracy corresponding to each candidate precision parameter in the second set is less than the calculation accuracy corresponding to the current precision parameter; and determine whether there is a target precision parameter in the second set that meets the second condition.

[0107] It should be noted that the calculation accuracy corresponding to the precision parameter is negatively correlated with the corresponding execution time, that is, the greater the calculation accuracy of the target precision parameter, the longer the execution time of the target operator based on the target precision parameter will be. Therefore, the calculation accuracy of the candidate precision parameters that meet the second condition must be less than the current precision parameter. Therefore, the training device can determine N candidate precision parameters with calculation accuracy less than the current precision parameter from the multiple candidate precision parameters corresponding to the target operator in the precision parameter library to obtain the second set, where N is an integer greater than or equal to 1. Among them, the method of determining the calculation accuracy is similar to the method of determining the first set, and will not be repeated here.

[0108] In some embodiments, the training device can traverse each candidate precision parameter in the second set in order of calculation accuracy from high to low, and for the jth candidate precision parameter in the traversal: based on the input data, predict the third degree of influence of the jth candidate precision parameter on the convergence performance of the target model; if the third degree of influence is greater than the preset degree of influence, stop the traversal, and determine the j-1th candidate precision parameter as the target precision parameter based on the value of j, or determine that the target precision parameter does not exist in the second set; and if the third degree of influence is less than or equal to the preset degree of influence, continue to traverse the second set.

[0109] In some embodiments, the training device may first arrange the candidate precision parameters in the second set from high to low according to the calculation accuracy. Then, the training device may traverse from high calculation accuracy to low calculation accuracy, and the calculation accuracy of the candidate precision parameters in the latter round of traversal may be one level lower than the calculation accuracy of the candidate precision parameters in the previous round of traversal. During the traversal process, the training device may make a prediction based on the input data and the traversed candidate precision parameters (i.e., the j-th candidate precision parameter) to obtain the third degree of influence of the j-th candidate precision parameter on the convergence performance of the target model. Among them, the method for predicting the third degree of influence is similar to the method for obtaining the first degree of influence based on the input data and the current precision parameter in the above example, and will not be elaborated here.

[0110] When j = 1, it means that the candidate precision parameter with one calculation accuracy level lower than the current precision parameter can no longer meet the convergence requirements of the target model, that is, there is no target precision parameter in the second set. Similarly, when j > 1, it indicates that the j-1th candidate precision parameter is the target precision parameter.

[0111] In this embodiment, by traversing the candidate precision parameters in descending order of computational accuracy, prioritizing the high-precision candidate precision parameters, and combining the third degree of influence of the predicted candidate precision parameters on the convergence performance of the target model, the lowest candidate precision parameters that meet the preset degree of influence can be screened out. When the current precision parameters of the target operator meet the convergence requirements, it is possible to continue to determine whether precision parameters with lower computational accuracy meet the convergence requirements. Ultimately, a set of precision parameters that meet the convergence requirements and have the fastest execution speed is obtained, thereby achieving dynamic adjustment of the target model precision parameters and improving the training effect and efficiency of the target model.

[0112] S323: In the case where the target accuracy parameter exists, adjust the expression accuracy of the training device for the input data based on the target accuracy parameter to obtain first target data, and execute the target operator based on the first target data.

[0113] In some embodiments, when target precision parameters are present, the target precision parameters include: a second floating-point format and a second quantization granularity. The training device may determine a second quantization range corresponding to the input data based on the second floating-point format; divide the input data into a plurality of second sub-data based on the second quantization granularity, and quantize the plurality of second sub-data based on the second quantization range to obtain quantization results corresponding to the plurality of second sub-data; and obtain the first target data based on the quantization results corresponding to the plurality of second sub-data.

[0114] The method by which the training device quantizes according to the target precision parameter to obtain the first target data is similar to the method by which the training device quantizes according to the current precision parameter to obtain quantized data corresponding to the input data. Here, the target precision parameter is "fp8, block granularity" as an example for explanation.

[0115] As an example, assume that the input data is a tensor X = [2.0 × 10 -3 , 1.0×10 -10 , 5.0×10 -9 , 3.0×10 -8 The target precision parameter is "fp8, block granularity", where the format of fp8 is 4-bit exponent and 3-bit mantissa, i.e. E4M3, with an exponent bias of 7.

[0116] When the training device performs low-precision quantization on the tensor X based on the target precision parameter, it can determine the quantization range corresponding to the tensor X as [0, 448] based on the second floating-point format (fp8), that is, the training device can quantize the tensor X to the range of [0, 448]. Then, since the second quantization granularity is "block granularity", the training device can divide the tensor X into two blocks, each block corresponding to a second sub-data. Among them, block1: [2.0×10 -3 , 3.0×10 -8 ], block2: [1.0×10 -10 , 5.0×10 -9 The training device quantizes each block in the same way as the "tensor" granularity quantization in the above example. Block1 is scaled to [448.0, 6.72×10 -3], the equivalent quantization is [448.0, 0.0067] (the quantization result corresponding to one second sub-data); block2 is [8.96, 448.0] after scaling, and the equivalent quantization is [8.96, 448.0] (the quantization result corresponding to another second sub-data). When quantization is performed at block granularity, the number of overflowed data is 0, accounting for 0% of the total quantized data (4) (0 / 4=0). That is, the overflow of quantized data is: the proportion of overflowed data to quantized data is 0%.

[0117] After obtaining the quantization results corresponding to the plurality of second sub-data, the training device can store them in a structured format, retaining the quantization results corresponding to each block. The data set stored in the structured format is the first target data. When executing the target operator based on the first target data, the training device can locate the required block based on the index file stored in the structured format (recording the mapping relationship between each block and the input data), and execute the target operator based on the quantization results therein.

[0118] In some embodiments, reference Figure 5 If the target precision parameter does not exist, the current precision parameter of the target operator is the optimal precision parameter and does not need to be adjusted. Therefore, in this case, the training device can adjust the accuracy of the input data expressed by the training device based on the current precision parameter to obtain second target data, and execute the target operator based on the second target data.

[0119] Here, in combination with the above examples, the application process of the model training method provided in this specification is explained. It is assumed that in the target model, the current precision parameter of the matrix multiplication operator (target operator) is "fp8 block granularity", and multiple candidate precision parameters include "fp4 tile granularity", "fp8 tensor granularity", "fp8 tile granularity", "bf16", and "fp32". The training device determines the degree of influence of the precision parameter on the convergence performance of the target model based on the overflow of the quantized data, assuming that the preset influence degree is 0.1% for the overflow of the quantized data. During the training process of the target model, for the matrix multiplication operator in the first execution mode, the training device can obtain the above-mentioned multiple candidate precision parameters by calling the precision parameter library through the execution engine.

[0120] In some embodiments, the training device determines, through the precision parameter selector, that the overflow of the quantized data at the "fp8 block granularity" is 2%, that is, the first degree of influence of the current precision parameter on the convergence performance of the target model is greater than the preset degree of influence.

[0121] In this case, the training device can determine the first set (i.e., "fp8 tile granularity", "bf16", "fp32") whose precision is greater than the current precision parameter from multiple candidate precision parameters, and then traverse the first set from low to high according to the calculation precision. The training device first traverses the candidate precision parameters of "fp8 tile granularity". The training device determines through the precision parameter selector that the overflow of quantized data under "fp8 tile granularity" is less than or equal to 0.1%, and then determines that the candidate precision parameter of "fp8tile granularity" is the target precision parameter. If the training device determines through the precision parameter selector that the overflow of quantized data under "bf16" is still greater than 0.1%, it can continue to traverse the candidate precision parameters of the next precision level until the target precision parameter is determined.

[0122] In some embodiments, the training device determines, through a precision parameter selector, that the overflow of the quantized data at the "fp8 block granularity" is less than 0.1%, that is, the first degree of influence of the current precision parameter on the convergence performance of the target model is less than a preset degree of influence.

[0123] In this case, the training device can identify a second set of candidate precision parameters (i.e., "fp4 tile granularity" and "fp8 tensor granularity") with lower precision than the current precision parameter from multiple candidate precision parameters, and then traverse the second set from high to low according to computational precision. The training device first traverses the candidate precision parameters for "fp8 tensor granularity." If the training device determines, through the precision parameter selector, that the overflow of quantized data at "fp8 tensor granularity" is greater than 0.1%, the current precision parameter can be determined as the target precision parameter, i.e., the precision parameter of the target operator is not adjusted. If the training device determines, through the precision parameter selector, that the overflow of quantized data at "fp8 tensor granularity" is less than or equal to 0.1%, the training device can continue to traverse the candidate precision parameters for the next precision level, "fp4 tile granularity." If the training device determines, through the precision parameter selector, that the overflow of quantized data at "fp4 tile granularity" is greater than 0.1%, the candidate precision parameter for "fp8 tensor granularity" can be determined as the target precision parameter.

[0124] The above describes the execution process of the target operator using the first execution mode. In some embodiments, the target operator may also include a second execution mode, in which the computational precision of the target operator is not adjusted. The execution process of the target operator in the second execution mode includes: the training device obtains input data of the target operator and obtains a current precision parameter corresponding to the target operator; adjusts the training device's representation precision of the input data based on the current precision parameter to obtain second target data; and executes the target operator based on the second target data.

[0125] It should be noted that, in the absence of target precision parameters and in the second execution mode of the target operator, the training device may not adjust the precision parameters of the target operator, but directly execute the target operator based on the current precision parameters corresponding to the target operator.

[0126] Among them, the way in which the training device adjusts the expression accuracy of the input data of the training device based on the current accuracy parameter to obtain the second target data, and executes the target operator based on the second target data is the same as the way in which the training device adjusts the expression accuracy of the input data of the training device based on the target accuracy parameter to obtain the first target data, and executes the target operator based on the first target data, and will not be repeated here.

[0127] When the execution mode of the target operator includes a first execution mode and a second execution mode. During the iterative training process, the training device may execute the target operator in the first execution mode in some rounds and in the second execution mode in other rounds. For example, in rounds that are multiples of N, the training device executes the target operator in the first execution mode; in other rounds, the training device executes the target operator in the second execution mode. This can achieve timely adjustment of the operator calculation accuracy during the model training process, and avoid adding excessive additional overhead to the training device.

[0128] In some embodiments, in multiple rounds of N in a plurality of iteration rounds, the training device may record the model parameters of the target model in the current round. In the N*kth round, the training device, in the presence of a target accuracy parameter, updates the model parameters of the target model to the model parameters recorded in the N*(k-1)th round, where k is an integer greater than 1, and adjusts the expression accuracy of the input data of the training device based on the updated model parameters and the target accuracy parameter to obtain the first target data.

[0129] In some embodiments, when training the target model, the training device actually updates the model parameters of the target model. Assuming N=100, after the N*kth round, the training device updates the current precision parameters of at least one target operator through the model training method provided in this specification, indicating that in the N*(k-1)th round and the N*kth round, inappropriate precision parameters occurred. Among them, when the first influence degree is greater than the preset influence degree, it means that in the iterations of the N*k-1th round and the N*kth round, there are rounds that do not meet the convergence requirements of the target model, and the iterations of these rounds may affect the training effect of the target model. Therefore, when the target precision parameters exist, the training device can roll back the model parameters of the target model to the last recorded point. For example, when the target precision parameters exist in the N*kth round, the training device can roll back the model parameters of the target model to the model parameters recorded in the N*(k-1)th round, and re-train the target model to ensure the training effect of the target model.

[0130] In this embodiment, when the precision parameters corresponding to the target operator are updated because the precision parameters do not meet the convergence requirements in the N*kth round, the model parameters of the target model are rolled back to the N*(k-1)th round parameters, eliminating the interference of invalid training rounds on the model training effect. The precision deviation can be corrected without interrupting training, avoiding the risk of divergence caused by error accumulation, thereby ensuring the reliability of model convergence and improving the training effect of the model.

[0131] In summary, in the model training method and device provided in this specification, the training device first obtains the target model to be trained, and the target model includes a target operator that supports at least the first execution mode. Then, the training device iteratively trains the target model and executes the target operator in the first execution mode during the training process. In this mode, the training device obtains the current precision parameter and multiple candidate precision parameters corresponding to the target operator, and analyzes the applicability of each candidate parameter in combination with the input data of the target operator to determine whether there is a target precision parameter that makes the training effect of the target model better than the current precision parameter. If so, the training device dynamically adjusts the expression accuracy of the input data based on the target precision parameter to generate the first target data adapted to the current training stage, and executes the target operator based on the data. Through the above mechanism, this scheme realizes the adaptive dynamic adjustment of the calculation accuracy of the target operator, so that the calculation accuracy of the target operator matches the accuracy requirements of the current training stage, ensuring the training efficiency of the model and the performance of the trained model.

[0132] On the other hand, this specification provides a computer-readable non-transitory storage medium storing at least one instruction set for model training. When the at least one instruction set is executed by a processor, the at least one instruction set instructs the processor to implement the steps of the model training method described in this specification. In some possible implementations, various aspects of this specification can also be implemented in the form of a program product, which includes program code. When the program product is run on a computing device 200, the program code is used to cause the computing device 200 to perform the steps of the model training method described in this specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on the computing device 200. However, the program product of this specification is not limited to this. In this specification, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing. Program code for performing the operations described herein may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may execute entirely on the computing device 200, partially on the computing device 200, as a stand-alone software package, partially on the computing device 200 and partially on a remote computing device, or entirely on the remote computing device.

[0133] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0134] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented by way of example only and may not be limiting. Although not expressly stated herein, those skilled in the art will understand that this specification encompasses various reasonable changes, improvements, and modifications to the embodiments. Such changes, improvements, and modifications are intended to be suggested by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0135] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, “one embodiment,” “an embodiment,” and / or “some embodiments” mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is emphasized and should be understood that two or more references to “an embodiment,” “one embodiment,” or “an alternative embodiment” in various parts of this specification do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be appropriately combined in one or more embodiments of this specification.

[0136] It should be understood that in the foregoing descriptions of the embodiments of this specification, to facilitate understanding of a feature and to simplify this specification, various features are combined in a single embodiment, figure, or description thereof. However, this does not necessarily mean that these features are combined. When reading this specification, a person skilled in the art may label some of the devices as separate embodiments. In other words, the embodiments of this specification can also be understood as the integration of multiple sub-embodiments. The content of each sub-embodiment is also valid even when it includes fewer than all the features of a single previously disclosed embodiment.

[0137] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, articles, and the like, cited herein, except to the extent that it is inconsistent or conflicting with this document or that it has a limiting effect on the broadest scope of the claims, is hereby incorporated by reference for all purposes now or hereafter connected with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall control.

[0138] Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can adopt alternative configurations based on the embodiments in this specification to implement the application in this specification. Therefore, the embodiments of this specification are not limited to the embodiments precisely described in the application.

Claims

1. A model training method, applied to a training device, comprising: Obtaining a target model to be trained, the target model including a target operator, the execution mode of the target operator including at least a first execution mode; as well as Iteratively train the target model, and execute the target operator in the first execution mode during the training process, wherein the execution process of the target operator in the first execution mode includes: Obtain input data of the target operator, and obtain a current precision parameter and multiple candidate precision parameters corresponding to the target operator, Based on the input data, determining whether there is a target accuracy parameter among the multiple candidate accuracy parameters, wherein the training effect of the training device on the target model under the target accuracy parameter is better than the training effect of the target model under the current accuracy parameter, and In the presence of the target accuracy parameter, the expression accuracy of the input data by the training device is adjusted based on the target accuracy parameter to obtain first target data, and the target operator is executed based on the first target data.

2. The method according to claim 1, wherein The target accuracy parameter satisfies at least one of the following: The degree of influence of the target accuracy parameter on the convergence performance of the target model is lower than the degree of influence of the current accuracy parameter on the convergence performance of the target model; or, The time consumed by the training device to execute the target operator under the target precision parameter is less than the time consumed to execute the target operator under the current precision parameter.

3. The method according to claim 1, wherein Determining whether a target accuracy parameter exists among the plurality of candidate accuracy parameters based on the input data includes: Predicting, based on the input data, a first degree of influence of the current precision parameter on the convergence performance of the target model; and When the first influence degree is greater than the preset influence degree, it is determined whether there is a target accuracy parameter that meets a first condition among the multiple candidate accuracy parameters. The first condition includes: the influence degree on the convergence performance of the target model is less than or equal to the preset influence degree.

4. The method according to claim 3, wherein: Determining whether there is a target accuracy parameter that meets a first condition among the plurality of candidate accuracy parameters includes: Determining a first set from the plurality of candidate precision parameters, wherein the calculation precision corresponding to each candidate precision parameter in the first set is greater than the calculation precision corresponding to the current precision parameter; and It is determined whether there is a target accuracy parameter in the first set that meets the first condition.

5. The method according to claim 4, wherein The determining whether there is a target accuracy parameter satisfying the first condition in the first set includes: The candidate precision parameters in the first set are traversed in order of calculation precision from low to high, and for the i-th candidate precision parameter in the traversal: Predicting, based on the input data, a second degree of influence of the i-th candidate precision parameter on the convergence performance of the target model; If the second influence degree is less than or equal to the preset influence degree, stop traversing and determine the i-th candidate accuracy parameter as the target accuracy parameter; If the second impact degree is greater than the preset impact degree, continue to traverse the first set.

6. The method according to claim 3, wherein: The method further comprises: When the first influence degree is less than or equal to the preset influence degree, it is determined whether there is a target accuracy parameter that meets the second condition among the multiple candidate accuracy parameters, wherein the second condition includes: the execution time of the target operator by the training device under the target accuracy parameter is less than the execution time of the target operator under the current accuracy parameter.

7. The method according to claim 6, wherein: Determining whether there is a target accuracy parameter that meets a second condition among the plurality of candidate accuracy parameters includes: Determining a second set from the plurality of candidate precision parameters, wherein the calculation precision corresponding to each candidate precision parameter in the second set is less than the calculation precision corresponding to the current precision parameter; and It is determined whether there is a target accuracy parameter in the second set that meets the second condition.

8. The method according to claim 7, wherein: Determining whether there is a target accuracy parameter in the second set that meets the second condition includes: The candidate precision parameters in the second set are traversed in descending order of calculation precision, and for the j-th candidate precision parameter in the traversal: Predicting, based on the input data, a third degree of influence of the j-th candidate precision parameter on the convergence performance of the target model; If the third impact degree is greater than the preset impact degree, stop traversing, and determine the j-1th candidate precision parameter as the target precision parameter based on the value of j, or determine that the target precision parameter does not exist in the second set; and If the third impact degree is less than or equal to the preset impact degree, continue to traverse the second set.

9. The method according to claim 3, wherein: The current precision parameter includes at least one of a first floating-point number format and a first quantization granularity, and the predicting, based on the input data, a first degree of influence of the current precision parameter on the convergence performance of the target model includes: Determining a first quantization range corresponding to the input data based on the first floating-point number format; Dividing the input data into a plurality of first sub-data based on the first quantization granularity, and quantizing the plurality of first sub-data respectively based on the first quantization range to obtain quantization results corresponding to the plurality of first sub-data; Obtaining quantized data corresponding to the input data based on quantization results corresponding to the plurality of first sub-data; and Based on the quantized data, a first influence degree of the current precision parameter on the convergence performance of the target model is predicted.

10. The method according to claim 9, wherein: The predicting, based on the quantized data, a first degree of influence of the current precision parameter on the convergence performance of the target model includes: Based on at least one of an overflow condition of the quantized data or a quantization error of the quantized data, a first influence degree of the current precision parameter on the convergence performance of the target model is determined.

11. The method according to claim 1, wherein The target precision parameter includes at least one of a second floating point format and a second quantization granularity, and adjusting the expression precision of the input data by the training device based on the target precision parameter to obtain the first target data includes: determining, based on the second floating-point number format, a second quantization range corresponding to the input data; Dividing the input data into a plurality of second sub-data based on the second quantization granularity, and quantizing the plurality of second sub-data respectively based on the second quantization range to obtain quantization results corresponding to the plurality of second sub-data; and The first target data is obtained based on the quantization results corresponding to the plurality of second sub-data.

12. The method according to claim 1, wherein The method further comprises: In the absence of the target accuracy parameter, the expression accuracy of the input data by the training device is adjusted based on the current accuracy parameter to obtain second target data, and the target operator is executed based on the second target data.

13. The method according to claim 1, wherein The execution mode of the target operator further includes a second execution mode, the training process includes multiple iteration rounds, and executing the target operator in the first execution mode during the training process includes: In multiple rounds of N in the multiple iteration rounds, the target operator is executed in the first execution mode, where N is an integer greater than 1. In other rounds of the multiple iteration rounds, executing the target operator in the second execution mode; The execution process of the target operator in the second execution mode includes: Obtain the input data of the target operator and obtain the current precision parameter corresponding to the target operator, The expression accuracy of the input data by the training device is adjusted based on the current accuracy parameter to obtain second target data, and the target operator is executed based on the second target data.

14. The method according to claim 13, wherein In a multiple of N rounds in the plurality of iteration rounds, the method further comprises: recording model parameters of the target model in the current round; In the N*kth round, in the presence of the target accuracy parameter, adjusting the expression accuracy of the input data by the training device based on the target accuracy parameter to obtain first target data includes: In the presence of the target accuracy parameter, the model parameters of the target model are updated to the model parameters recorded in the N*(k-1)th round, where k is an integer greater than 1, and Based on the updated model parameters and the target accuracy parameters, the expression accuracy of the training device for the input data is adjusted to obtain the first target data.

15. A training device comprising: At least one storage medium storing at least one instruction set for training a target model; as well as At least one processor is communicatively connected to the at least one storage medium, wherein when the training device is running, the at least one processor reads the at least one instruction set and implements the method as described in any one of claims 1-14 according to the instructions of the at least one instruction set.

16. A computer-readable non-volatile storage medium, wherein: The computer-readable non-volatile storage medium stores at least one instruction set, and when the at least one instruction set is executed by at least one processor, the method according to any one of claims 1 to 14 is implemented.

Images