Neural network model training acceleration method and system based on dynamic precision quantization

Abstract

This invention provides a method and system for accelerating neural network model training based on dynamic precision quantization. Before the neural network model training begins, the data matrix involved in the computation is divided into logical blocks. During training, the quantization range of each block of data and the gradient value corresponding to that block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks. Based on the current number of training steps of the network, the target average quantization bit width is determined. Combining the relative quantization bit width, the target average quantization bit width, and the pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of the weight values ​​and activation values ​​of each block is dynamically determined. The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for computation, thereby accelerating the training computation of the neural network model. This invention solves the problems of slow training speed and difficulty in directly obtaining the theoretically optimal value of quantization bit width in existing neural networks.

Description

Technical Field

[0001] This invention relates to the field of neural network training technology, and in particular to a method and system for accelerating neural network model training based on dynamic precision quantization. Background Technology

[0002] In existing technologies, neural network models can provide a comprehensive overview of various aspects of work and life. However, traditional neural network model training typically requires a large data flow and is slow. While some corresponding techniques have made corresponding optimizations, such as dividing the activation value data in neural network model training into blocks and iteratively trying to apply different quantization precisions to different data blocks during training, the final result is obtained by using the precision allocation method that minimizes the mean square value of the network weight gradient. This reduces the precision of some activation values, thereby reducing the memory requirements during training, while maintaining the overall accuracy of the trained model as much as possible. Since this method uses the actual data (mean square value of weight gradient) generated during training as the optimization target, it does not require users to have prior knowledge of a specific network and can be adaptively applied to the training process of various neural networks, reducing the storage requirements for storing activation values.

[0003] However, it only quantifies the activation values ​​in the neural network training rather than the weight values, so it does not accelerate the forward and backward training processes that use weights for calculation; the weight gradient mean square value optimization objective it uses cannot be solved analytically, so it can only use an iterative search to find a better precision allocation method, and cannot directly obtain the theoretical optimal value. Summary of the Invention

[0004] This invention provides a method and system for accelerating the training of neural network models based on dynamic precision quantization, in order to solve the problems of slow training speed and difficulty in directly obtaining the theoretical optimal value of quantization bit width in existing neural networks.

[0005] This invention provides a method for accelerating neural network model training based on dynamic precision quantization, comprising:

[0006] Before training the neural network model, the data matrix involved in the operation is divided into logical blocks;

[0007] During training, the quantization range of each data block and the gradient value corresponding to that data block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks.

[0008] Based on the current number of training steps of the network, determine the current target average quantization bit width;

[0009] Combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of each block weight value and activation value data is dynamically determined.

[0010] The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training and computation of the neural network model.

[0011] According to the present invention, a method for accelerating neural network model training based on dynamic precision quantization is provided, wherein before the neural network model training begins, the data matrix involved in the computation process is divided into logically smaller blocks, specifically including:

[0012] Before the neural network model training begins, the size of the data matrix involved in each step of the operation is known in advance;

[0013] The data matrix is ​​divided into multiple logical blocks according to a certain row and column size.

[0014] According to the present invention, a method for accelerating neural network model training based on dynamic precision quantization is provided. During training, the quantization range of each data block and the gradient value corresponding to that block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks. Specifically, this includes:

[0015] After the neural network model training begins, in each batch of training, the block quantization range of the weight values ​​and activation values ​​of the current batch, as well as the gradient data of the weight values ​​and activation values, are obtained.

[0016] By combining the block quantization range of the weight and activation values ​​of the current batch, as well as the gradient data of the weight and activation values, with historical data, we can obtain the average quantization range per block and the average gradient value of each data point in the current round. Each round includes multiple batches.

[0017] After each training round, the relative sensitivity of each data block is calculated according to the set formula, and expressed as the relative quantization bit width.

[0018] According to the present invention, a method for accelerating neural network model training based on dynamic precision quantization is provided, wherein determining the target average quantization bit width based on the current number of training steps of the network specifically includes:

[0019] Users specify in advance the average quantization bit width they expect to achieve during training;

[0020] During the training process, after multiple rounds of training, the average quantization bit width is forcibly set to a specific value to ensure accuracy.

[0021] According to the present invention, a method for accelerating neural network model training based on dynamic precision quantization dynamically determines the absolute quantization bit width of each block of weight and activation value data by combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum computational bit width parameters supported by the hardware. Specifically, the method includes:

[0022] The relative quantization bit width r i Convert to absolute quantization bit width b i :

[0023]

[0024] In the formula, a and b are the minimum and maximum bit widths supported by the current computing hardware, and λ is a parameter whose initial value is set to 0. Its value is gradually adjusted using a binary search method until it reaches b. i The difference between the target quantization bit width α and the set value satisfies:

[0025]

[0026] Where ∈ is 0.05.

[0027] According to the present invention, a method for accelerating neural network model training based on dynamic precision quantization is provided, wherein the training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training calculation of the neural network model, specifically including:

[0028] Each training round includes a forward phase, a backward phase, and a gradient update phase;

[0029] In each training round, the forward phase uses quantized weight values ​​and unquantized activation values ​​for calculation; the backward phase uses the gradient of quantized weight values ​​and unquantized activation values ​​for calculation; and the gradient update phase uses the gradient of quantized activation values ​​and unquantized activation values ​​for calculation. These steps are repeated until the neural network model training is completed.

[0030] The present invention also provides a neural network model training acceleration system based on dynamic precision quantization, the system comprising:

[0031] The block segmentation module is used to divide the data matrix involved in the operation into logical blocks before the neural network model training begins.

[0032] The quantization sensitivity calculation module is used to calculate the quantization sensitivity of each block of data and the gradient value corresponding to that block of data during the training process, and to represent the sensitivity as the optimal relative quantization bit width between blocks;

[0033] The average quantization bit width target determination module is used to determine the current required average quantization bit width target based on the current number of training steps of the network.

[0034] The absolute quantization bit width determination module is used to dynamically determine the absolute quantization bit width of each block of weight value and activation value data by combining the relative quantization bit width, the average quantization bit width target, and the preset maximum and minimum calculation bit width parameters supported by the hardware.

[0035] The accelerated computing module is used to quantize the training data according to the absolute quantization bit width to obtain low-precision quantized data for computation, thereby accelerating the training computation of the neural network model.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the neural network model training acceleration method based on dynamic precision quantization as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the neural network model training acceleration method based on dynamic precision quantization as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the neural network model training acceleration method based on dynamic precision quantization as described above.

[0039] This invention provides a method and system for accelerating neural network model training based on dynamic precision quantization. It employs online data quantization sensitivity analysis, utilizing existing activation and weight gradient data and quantization range data from the training process to analytically solve for the optimal relative quantization bit width of the activation and weight data with relatively low computational complexity. Simultaneously, it utilizes an adaptive block-based dynamic precision allocation method to convert the relative quantization bit width into the absolute bit width of the data according to different training stages. This enables computational acceleration using data with low average precision throughout the network training process, while maintaining the accuracy of the final trained network. An analytical expression obtained through mathematical methods directly yields the optimal quantization bit width distribution of the data used in the neural network training process. This significantly reduces the average quantization bit width of the weights and activations involved in the calculation during training, while reserving a higher quantization bit width for the most important data in the network to maintain its accuracy, provided that the final accuracy of the trained neural network model is not affected or is minimally affected. When combined with hardware that supports variable bit width data computation, it can effectively improve the training speed of the neural network model. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts illustrating a method for accelerating neural network model training based on dynamic precision quantization provided by the present invention;

[0042] Figure 2 This is the second flowchart of a method for accelerating neural network model training based on dynamic precision quantization provided by the present invention;

[0043] Figure 3 This is the third flowchart of a method for accelerating neural network model training based on dynamic precision quantization provided by the present invention;

[0044] Figure 4 This is the fourth flowchart of a method for accelerating neural network model training based on dynamic precision quantization provided by the present invention;

[0045] Figure 5 This is the fifth flowchart of a method for accelerating neural network model training based on dynamic precision quantization provided by the present invention;

[0046] Figure 6 This is a schematic diagram of the module connections of a neural network model training acceleration system based on dynamic precision quantization provided by the present invention;

[0047] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0048] Figure label:

[0049] 110: Block module; 120: Quantization sensitivity calculation module; 130: Average quantization bit width target determination module; 140: Absolute quantization bit width determination module; 150: Accelerated calculation module; 710: Processor; 720: Communication interface; 730: Memory; 740: Communication bus. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The following is combined Figures 1-5 The present invention describes a method for accelerating neural network model training based on dynamic precision quantization, comprising:

[0052] S100. Before the neural network model training begins, the data matrix involved in the operation is divided into logical blocks;

[0053] S200. During training, the quantization range of each block of data and the gradient value corresponding to that block of data are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks.

[0054] S300. Based on the current number of training steps of the network, determine the target average quantization bit width required at the moment.

[0055] S400: Combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum calculation bit width parameters supported by the hardware, dynamically determine the absolute quantization bit width of each block weight value and activation value data;

[0056] S500: The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training calculation of the neural network model.

[0057] The neural network model training acceleration method based on dynamic precision quantization disclosed in this invention directly obtains the optimal quantization bit width distribution of the data used in the neural network training process. It can significantly reduce the average quantization bit width of the weights and activations involved in the calculation during training, while ensuring that the final accuracy of the trained neural network model is not affected or is only minimally affected. However, it retains a high quantization bit width for the most important data in the network to maintain its accuracy. When used in conjunction with hardware that supports variable bit width data calculation, it can effectively improve the training speed of the neural network model.

[0058] Before training the neural network model begins, the data matrix involved in the computation process is divided into logically smaller blocks, specifically including:

[0059] S101. Before the neural network model training begins, the size of the data matrix involved in each step of the operation is known in advance;

[0060] S102. Divide the data matrix into multiple logical blocks according to a certain row and column size.

[0061] In this invention, the data matrix is ​​illustrated by dividing it into 4×4 blocks, but this invention is applicable to various block sizes.

[0062] During training, the quantization range of each data block and the gradient value corresponding to that block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks, specifically including:

[0063] S201. After the neural network model training begins, in each batch of training, obtain the block quantization range of the weight values ​​and activation values ​​of the current batch (i.e., the maximum absolute value of the data within each block).

[0064] (Values) and gradients of weight values ​​and activation values;

[0065] S202. Combine the block quantization range of the current batch's weight values ​​and activation values, the weight value gradient (5 degrees), and the activation value gradient data with historical data to obtain the average quantization per block for the current round.

[0066] The range and the average gradient value for each data point, with each round including multiple batches;

[0067] S203. After each training round, calculate the relative sensitivity of each data block according to the set formula, which is expressed as the relative quantization bit width.

[0068] The method for calculating quantization sensitivity is as follows:

[0069]

[0070]

[0071] Among them, R i Δ is the quantization range of the i-th data block, 16 is the data size of each block (4×4), and Δ is the quantization range of the i-th data block. i,j T is the gradient value corresponding to the j-th value in the i-th data block. i It represents the number of multiplications performed on the i-th data block in each batch of calculations.

[0072] The r obtained from the above formula i This refers to the relative sensitivity corresponding to the i-th data block, also known as the optimal relative quantization bit width. It means that if r... i -r j =n, which means that in the quantization bit width allocation that has the least impact on the accuracy of the final trained model, the i-th block of data should use a quantization bit width that is n bits higher than that of the j-th block of data.

[0073] Based on the current number of training steps of the network, determine the current target average quantization bit width.

[0074] The body includes:

[0075] S301. The user specifies in advance the average quantization bit width expected to be achieved during training;

[0076] S302. During the training process, after multiple rounds of training, the average quantization bit width is forcibly set to a specific value to ensure accuracy.

[0077] In this invention, the user needs to specify the expected average quantization bit width α during training in advance. This setting is used for most of the training process; however, in the last M rounds of training, to ensure accuracy, α is forcibly set to 8. In this invention, M is set to 10.

[0078] Combining the relative quantization bit width, average quantization bit width target, and pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of each block's weight value and activation value data is dynamically determined, specifically including:

[0079] The relative quantization bit width r i Convert to absolute quantization bit width b i :

[0080]

[0081] In the formula, a and b are the minimum and maximum bit widths supported by the current computing hardware, and λ is a parameter whose initial value is set to 0. Its value is gradually adjusted using a binary search method until it reaches b. i The difference between the target quantization bit width α and the set value satisfies:

[0082]

[0083] Where ∈ is 0.05.

[0084] The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for computation, thereby accelerating the training computation of the neural network model. Specifically, this includes:

[0085] S501. Each training round includes a forward phase, a backward phase, and a gradient update phase.

[0086] S502. In each training round, the forward phase is calculated using quantized weight values ​​and unquantized activation values; the backward phase is calculated using the gradient of quantized weight values ​​and unquantized activation values; and the gradient update phase is calculated using the gradient of quantized activation values ​​and unquantized activation values. These steps are repeated until the neural network model training ends.

[0087] In this invention, in the next round of neural network training, the number of bits per block width b calculated above is used... i Quantize each weight and activation value—that is, round the mantissa of the corresponding floating-point value to the nearest integer b. i One binary digit.

[0088] This completes one round of training for the neural network. The steps of this invention continue to be executed repeatedly, starting with calculating the average gradient value, until the network model training is finished.

[0089] This invention provides a method for accelerating neural network model training based on dynamic precision quantization. It employs online data quantization sensitivity analysis, utilizing existing activation and weight gradient data and quantization range data from the training process to analytically solve for the optimal relative quantization bit width of the activation and weight data with relatively low computational complexity. Simultaneously, it utilizes an adaptive block-based dynamic precision allocation method to convert the relative quantization bit width into the absolute bit width of the data according to different training stages. This enables computational acceleration using data with low average precision throughout the network training process, while maintaining the accuracy of the final trained network. An analytical expression obtained through mathematical methods directly yields the optimal quantization bit width distribution of the data used in the neural network training process. This significantly reduces the average quantization bit width of the weights and activations involved in the calculation during training, while reserving a higher quantization bit width for the most important data in the network to maintain their accuracy, provided that the final accuracy of the trained neural network model is not affected or is minimally affected. When combined with hardware that supports variable bit width data computation, it can effectively improve the training speed of the neural network model.

[0090] refer to Figure 6 The present invention also discloses a neural network model training acceleration system based on dynamic precision quantization, the system comprising:

[0091] Blocking module 110 is used to divide the data matrix involved in the operation into logical blocks before the neural network model training begins.

[0092] The quantization sensitivity calculation module 120 is used to calculate the quantization sensitivity of each block of data and the gradient value corresponding to the block of data during the training process, and to represent the sensitivity as the optimal relative quantization bit width between blocks;

[0093] The average quantization bit width target determination module 130 is used to determine the current required average quantization bit width target based on the current number of training steps of the network.

[0094] The absolute quantization bit width determination module 140 is used to dynamically determine the absolute quantization bit width of each block weight value and activation value data by combining the relative quantization bit width, the average quantization bit width target, and the preset maximum and minimum calculation bit width parameters supported by the hardware.

[0095] The accelerated computing module 150 is used to quantize the training data according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training and computing of the neural network model.

[0096] Among them, the size of the data matrix involved in each step of the operation is known in advance in the block module 110 before the neural network model training begins;

[0097] The data matrix is ​​divided into multiple logical blocks according to a certain row and column size.

[0098] The quantization sensitivity calculation module 120, after the neural network model training begins, acquires the block quantization range of the weight values ​​and activation values ​​of the current batch and the gradient data of the weight values ​​and activation values ​​in each batch of training.

[0099] By combining the block quantization range of the weight and activation values ​​of the current batch, as well as the gradient data of the weight and activation values, with historical data, we can obtain the average quantization range per block and the average gradient value of each data point in the current round. Each round includes multiple batches.

[0100] After each training round, the relative sensitivity of each data block is calculated according to the set formula, and expressed as the relative quantization bit width.

[0101] The average quantization bit width target determination module 130 allows the user to specify the average quantization bit width expected to be achieved during training in advance.

[0102] During the training process, after multiple rounds of training, the average quantization bit width is forcibly set to a specific value to ensure accuracy.

[0103] The absolute quantization bit width determination module 140 determines the relative quantization bit width r. i Convert to absolute quantization bit width b i :

[0104]

[0105] In the formula, a and b are the minimum and maximum bit widths supported by the current computing hardware, and λ is a parameter whose initial value is set to 0. Its value is gradually adjusted using a binary search method until it reaches b. i The difference between the target quantization bit width α and the set value satisfies:

[0106]

[0107] Where ∈ is 0.05.

[0108] The accelerated computing module 150 includes a forward phase, a backward phase, and a gradient update phase in each training round.

[0109] In each training round, the forward phase uses quantized weight values ​​and unquantized activation values ​​for calculation; the backward phase uses the gradient of quantized weight values ​​and unquantized activation values ​​for calculation; and the gradient update phase uses the gradient of quantized activation values ​​and unquantized activation values ​​for calculation. These steps are repeated until the neural network model training is completed.

[0110] This invention discloses a neural network model training acceleration system based on dynamic precision quantization. It employs online data quantization sensitivity analysis and utilizes existing activation and weight gradient data and quantization range data during training to analytically solve for the optimal relative quantization bit width of activation and weight data with relatively low computational complexity. Simultaneously, it utilizes an adaptive block-based dynamic precision allocation method to convert the relative quantization bit width into the absolute bit width of the data according to different training stages. This enables computational acceleration using data with low average precision throughout the network training process while maintaining the accuracy of the final trained network. The analytical expression obtained through mathematical methods directly yields the optimal quantization bit width distribution of the data used in the neural network training process. This significantly reduces the average quantization bit width of the weights and activations involved in the calculation during training, while reserving a higher quantization bit width for the most important data in the network to maintain its accuracy, ensuring that the final accuracy of the trained neural network model is not affected or is minimally affected. When used in conjunction with hardware that supports variable bit width data computation, it can effectively improve the training speed of the neural network model.

[0111] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions from the memory 730 to execute a method for accelerating neural network model training based on dynamic precision quantization. This method includes: before the neural network model training begins, dividing the data matrix involved in the computation into logically smaller blocks;

[0112] During training, the quantization range of each data block and the gradient value corresponding to that data block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks.

[0113] Based on the current number of training steps of the network, determine the current target average quantization bit width;

[0114] Combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of each block weight value and activation value data is dynamically determined.

[0115] The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training and computation of the neural network model.

[0116] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute a neural network model training acceleration method based on dynamic precision quantization provided by the above methods, the method including: before the neural network model training begins, dividing the data matrix involved in the operation into logically small blocks;

[0118] During training, the quantization range of each data block and the gradient value corresponding to that data block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks.

[0119] Based on the current number of training steps of the network, determine the current target average quantization bit width;

[0120] Combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of each block weight value and activation value data is dynamically determined.

[0121] The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training and computation of the neural network model.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform a method for accelerating neural network model training based on dynamic precision quantization provided by the above methods, the method comprising: dividing the data matrix involved in the operation into logical blocks before the neural network model training begins;

[0123] During training, the quantization range of each data block and the gradient value corresponding to that data block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks.

[0124] Based on the current number of training steps of the network, determine the current target average quantization bit width;

[0125] Combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of each block weight value and activation value data is dynamically determined.

[0126] The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training and computation of the neural network model.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for accelerating neural network model training based on dynamic precision quantization, characterized in that, include: Before training the neural network model, the data matrix involved in the operation is divided into logical blocks; During training, the quantization range of each data block and the gradient value corresponding to that data block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks. Based on the current number of training steps of the network, determine the current target average quantization bit width; Combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of each block weight value and activation value data is dynamically determined. The training data is quantized according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training calculation of the neural network model. During training, the quantization range of each data block and the gradient value corresponding to that block are used to calculate its quantization sensitivity, and the sensitivity is represented by the optimal relative quantization bit width between blocks. Specifically, this includes: After the neural network model training begins, in each batch of training, the block quantization range of the weight values ​​and activation values ​​of the current batch, as well as the gradient data of the weight values ​​and activation values, are obtained. By combining the block quantization range of the weight and activation values ​​of the current batch, as well as the gradient data of the weight and activation values, with historical data, we can obtain the average quantization range per block and the average gradient value of each data point in the current round. Each round includes multiple batches. After each training round, the relative sensitivity of each data block is calculated according to the set formula, and expressed as the relative quantization bit width.

2. The method for accelerating neural network model training based on dynamic precision quantization according to claim 1, characterized in that, Before the neural network model training begins, the data matrix involved in the computation process is divided into logically smaller blocks, specifically including: Before the neural network model training begins, the size of the data matrix involved in each step of the operation is known in advance; The data matrix is ​​divided into multiple logical blocks according to a certain row and column size.

3. The method for accelerating neural network model training based on dynamic precision quantization according to claim 1, characterized in that, The step of determining the current target average quantization bit width based on the current number of training steps of the network specifically includes: Users specify in advance the average quantization bit width they expect to achieve during training; During the training process, after multiple rounds of training, the average quantization bit width is forcibly set to a specific value to ensure accuracy.

4. The method for accelerating neural network model training based on dynamic precision quantization according to claim 1, characterized in that, Combining the relative quantization bit width, the average quantization bit width target, and the pre-set maximum and minimum computational bit width parameters supported by the hardware, the absolute quantization bit width of each block's weight value and activation value data is dynamically determined, specifically including: relative quantization bit width Convert to absolute quantization bit width : ； In the formula, and These are the minimum and maximum bit widths supported by currently used computing hardware. A parameter is initially set to 0, and its value is gradually adjusted using a binary search method until... With the set quantization bit width target The gap between them satisfies: ； in, Take 0.

05.

5. The method for accelerating neural network model training based on dynamic precision quantization according to claim 1, characterized in that, The step of quantizing the training data according to the absolute quantization bit width to obtain low-precision quantized data for calculation, thereby accelerating the training calculation of the neural network model, specifically includes: Each training round includes a forward phase, a backward phase, and a gradient update phase; In each training round, the forward phase uses quantized weight values ​​and unquantized activation values ​​for calculation; the backward phase uses quantized weight values ​​and unquantized activation value gradients for calculation; and the gradient update phase uses quantized activation values ​​and unquantized activation value gradients for calculation. These steps are repeated until the neural network model training is completed.

6. A neural network model training acceleration system based on dynamic precision quantization, characterized in that, The system includes: The block segmentation module is used to divide the data matrix involved in the operation into logical blocks before the neural network model training begins. The quantization sensitivity calculation module is used to calculate the quantization sensitivity of each block of data and the gradient value corresponding to that block of data during the training process, and to represent the sensitivity as the optimal relative quantization bit width between blocks; The average quantization bit width target determination module is used to determine the current required average quantization bit width target based on the current number of training steps of the network. The absolute quantization bit width determination module is used to dynamically determine the absolute quantization bit width of each block of weight value and activation value data by combining the relative quantization bit width, the average quantization bit width target, and the preset maximum and minimum calculation bit width parameters supported by the hardware. The accelerated computing module is used to quantize the training data according to the absolute quantization bit width to obtain low-precision quantized data for computing, thereby accelerating the training and computing of the neural network model. The quantization sensitivity calculation module, after the neural network model training begins, acquires the block quantization range of the weight values ​​and activation values, as well as the weight value gradient and activation value gradient data for each batch of training. It then combines this data with historical data to obtain the average quantization range per block and the average gradient value for each data point in the current round. Each round includes multiple batches. After the training of each round ends, it calculates the relative sensitivity of each data point according to a set formula, expressed as the relative quantization bit width.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the neural network model training acceleration method based on dynamic precision quantization as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the neural network model training acceleration method based on dynamic precision quantization as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the neural network model training acceleration method based on dynamic precision quantization as described in any one of claims 1 to 5.

Images