A method, apparatus, and storage medium for training a neural network using hysteresis quantization perception.

Abstract

This invention relates to the field of deep learning technology and discloses a method, apparatus, and storage medium for hysteresis quantization-aware training of neural networks. The method enables training a pre-constructed neural network to obtain a floating-point network that satisfies a first preset convergence condition. Pseudo-quantization nodes are inserted into the network, and pre-acquired training data is input for PTQ calibration to obtain an initial quantized network. The initial quantized network is then trained using QAT according to a preset weight update strategy. If the model parameters satisfy a second preset convergence condition during QAT training, the quantization parameters of the initial quantized network are fixed as target quantization parameters, and a hysteresis strategy is added to the weight update strategy to obtain a weight hysteresis update strategy. Finally, the initial quantized network is trained using QAT according to the weight hysteresis update strategy to obtain a target quantized network that satisfies a third preset convergence condition. Therefore, implementing this invention can improve network accuracy, as well as the stability and reliability of network training.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method, apparatus, and storage medium for training neural networks using hysteresis quantization perception. Background Technology

[0002] In scenarios where there is a need for efficient deployment and operation of deep learning models, such as mobile devices, embedded systems, and edge computing, post-training quantization (PTQ) is usually adopted. After the model training is completed, based on quantization technology, the high-precision (such as 32-bit floating-point numbers) weights and activation values ​​in the model are converted into low-precision (such as 8-bit integers) representations.

[0003] In practice, traditional post-quantization training methods are prone to network accuracy decline due to information loss and nonlinear distortion during the quantization process. Quantization-Aware Training (QAT) was proposed because it can introduce quantization simulation during the training phase to allow the model to adapt to the quantized state, thereby ensuring that the network maintains high accuracy after quantization.

[0004] However, practice has shown that introducing a round function during model training in existing QAT techniques results in zero gradients during backpropagation, preventing weight updates. Currently, the industry commonly uses the Straight-Through Estimator (STE), which treats the gradient of the quantization operation as an identity mapping. This means that during backpropagation, the gradient of the quantization operation is treated as the gradient of the input, disregarding the nonlinear distortion introduced by the quantization operation itself. Essentially, this is gradient estimation rather than an exact mathematical solution, easily leading to training instability issues such as oscillations and non-convergence, thus limiting the model's stability and reliability. Therefore, proposing a technical solution to improve network accuracy and enhance the stability and reliability of network training is particularly important. Summary of the Invention

[0005] This invention provides a method, apparatus, and storage medium for hysteresis quantization sensing training of neural networks, which can improve network accuracy and enhance the stability and reliability of network training.

[0006] To address the aforementioned technical problems, the first aspect of this invention discloses a method for training a neural network using hysteresis quantization perception, the method comprising:

[0007] The pre-built neural network is trained to obtain a floating-point network that satisfies the first preset convergence condition;

[0008] A pseudo-quantization node is inserted into the floating-point network, and the pre-acquired training data is input into the floating-point network into which the pseudo-quantization node has been inserted for PTQ calibration to obtain the initial quantization network.

[0009] According to a preset weight update strategy, the initial quantization network is trained using QAT. The weight update strategy is used to update the weight parameters of the initial quantization network during the QAT training process, and to solve for the quantization parameters based on the weight parameters.

[0010] If the model parameters of the determined initial quantization network satisfy the second preset convergence condition during the QAT training process, then the quantization parameters of the initial quantization network are fixed as the target quantization parameters, and a hysteresis strategy is added to the weight update strategy to obtain a weight hysteresis update strategy. Then, the initial quantization network is trained using the weight hysteresis update strategy to obtain a target quantization network that satisfies the third preset convergence condition.

[0011] A second aspect of the present invention discloses a hysteresis quantization perception training device for a neural network, the device comprising:

[0012] The floating-point training unit is used to perform training operations on a pre-built neural network to obtain a floating-point network trained to meet a first preset convergence condition.

[0013] The PTQ calibration unit is used to insert pseudo-quantization nodes into the floating-point network and input the pre-acquired training data into the floating-point network with the pseudo-quantization nodes inserted to perform PTQ calibration, thereby obtaining the initial quantization network.

[0014] The QAT training unit is used to perform QAT training on the initial quantization network according to a preset weight update strategy. The weight update strategy is used to update the weight parameters of the initial quantization network during the QAT training process and solve the quantization parameters based on the weight parameters.

[0015] The hysteresis update unit is used to fix the quantization parameters of the initial quantization network as the target quantization parameters if the model parameters of the determined initial quantization network meet the second preset convergence condition during the QAT training process, and to add a hysteresis strategy to the weight update strategy to obtain a weight hysteresis update strategy.

[0016] The QAT training unit is also used to perform QAT training on the initial quantization network according to the weight hysteresis update strategy, so as to obtain a target quantization network trained to meet the third preset convergence condition.

[0017] As an optional implementation, in a second aspect of the invention, the model parameters of the initial quantization network during the QAT training process are determined in the following manner:

[0018] During QAT training of the initial quantization network, the floating-point weights for this inference in the weight input module of the initial quantization network are input to the weight quantization module of the initial quantization network for quantization processing to obtain the integer weights corresponding to the floating-point weights; and the floating-point input data received by the floating-point input module of the initial quantization network are input to the input quantization module of the initial quantization network for quantization processing to obtain the integer input data corresponding to the floating-point input data.

[0019] The integer weights and the integer input data are input into the convolution operation of the convolutional layer module of the initial quantization network, and the output is sent to the floating-point output module of the initial quantization network to obtain floating-point output data;

[0020] The floating-point output data is input to the output quantization module of the initial quantization network for quantization processing to obtain the integer output data corresponding to the floating-point output data.

[0021] The first loss parameter of the initial quantization network for this inference is determined, and the first loss parameter of this inference is backpropagated to the convolutional layer module by the output quantization module to perform convolution calculation to obtain the convolution calculation result;

[0022] The convolutional layer module transmits the convolution calculation result to the weight input module and the floating-point input module respectively, and updates it according to the weight update strategy to obtain the updated floating-point weight. The updated floating-point weight is used as the floating-point weight for the next inference and is applied to the floating-point output module during the next inference.

[0023] The model parameters of the initial quantization network include the model weights for each inference under the weight update strategy and the first loss parameter for each inference. The model weights include the floating-point weights and / or the integer weights corresponding to the floating-point weights.

[0024] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0025] The judgment unit is used to determine whether the current condition of the initial quantization network satisfies the second preset convergence condition based on the determined network inference result of the initial quantization network under the weight update strategy; and when the judgment unit determines that the current condition of the initial quantization network satisfies the second preset convergence condition, it triggers the hysteresis update unit to perform the operation of fixing the quantization parameters of the initial quantization network as the target quantization parameters and adding a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy, and triggers the QAT training unit to perform the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy until the initial quantization network satisfies the second preset convergence condition;

[0026] The QAT training unit is further configured to, when the judgment unit determines that the current conditions of the initial quantization network do not meet the second preset convergence condition, re-execute the operation of QAT training the initial quantization network according to the preset weight update strategy.

[0027] As an optional implementation, in a second aspect of the present invention, the method by which the determining unit determines whether the current conditions of the initial quantization network satisfy the second preset convergence condition based on the determined network inference results of the initial quantization network under the weight update strategy specifically includes:

[0028] Based on the first loss parameter included in the network inference result for multiple inferences under the weight update strategy, determine whether all first loss parameters within a preset number of continuous inferences are within a preset loss value range, and record this as the first sub-convergence index condition.

[0029] Based on the model weights included in the network inference results for multiple inferences under the weight update strategy, the weight update curve corresponding to the weight update strategy is determined; and the weight update curve is divided according to a preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and it is determined whether there is a preset high-frequency weight oscillation phenomenon in all the weight change phenomena corresponding to the target inference number range in the weight update curve, which is denoted as the second sub-convergence index condition.

[0030] Specifically, when both the first sub-convergence index condition and the second sub-convergence index condition are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; when the first sub-convergence index condition and / or the second sub-convergence index condition are not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0031] As an optional implementation, in the second aspect of the present invention, the method by which the QAT training unit performs QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain a target quantization network trained to meet the third preset convergence condition specifically includes:

[0032] According to the weight hysteresis update strategy, the initial quantization network is trained with QAT to obtain multiple weight quantization related parameters for the initial quantization network under the weight hysteresis update strategy. The weight quantization related parameters include parameters before weight quantization and parameters after weight quantization.

[0033] Based on all the weight quantization-related parameters, determine whether the target quantization parameters of the initial quantization network meet the preset update conditions; and when it is determined that the target quantization parameters of the initial quantization network meet the update conditions, re-execute the operation of QAT training of the initial quantization network according to the preset weight update strategy.

[0034] When it is determined that the target quantization parameters of the initial quantization network do not meet the update conditions, a second loss parameter for the initial quantization network under the weight hysteresis update strategy is determined, and based on the second loss parameter and the determined validation set results of the initial quantization network, it is determined whether the initial quantization network meets the third preset convergence condition.

[0035] When it is determined that the initial quantization network does not meet the third preset convergence condition, the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy is re-executed to obtain multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and the operation of determining whether the target quantization parameter of the initial quantization network meets the preset update condition based on all the weight quantization related parameters is re-executed.

[0036] When it is determined that the initial quantization network satisfies the third preset convergence condition, the initial quantization network that satisfies the third preset convergence condition is determined as the target quantization network.

[0037] As an optional implementation, in a second aspect of the present invention, the method by which the QAT training unit determines whether the target quantization parameters of the initial quantization network satisfy preset update conditions based on all the weight quantization-related parameters specifically includes:

[0038] Based on the parameters before and after weight quantization included in each of the weight quantization related parameters, calculate the rounding error value of the corresponding quantization operation introduced before each weight quantization.

[0039] Based on the rounding error values ​​of all the pre-quantization weighted parameters, determine the target parameter quantity of the target pre-quantization weighted parameter whose rounding error value is a preset error threshold, and calculate the ratio of the target pre-quantization weighted parameter to all the pre-quantization weighted parameters based on the target parameter quantity and the total parameter quantity of all the pre-quantization weighted parameters.

[0040] Determine whether the ratio of the target weight pre-quantization parameter among all the weight pre-quantization parameters is greater than a preset ratio;

[0041] When it is determined that the ratio of the target weight pre-quantization parameter to all the weight pre-quantization parameters is greater than the preset ratio, it is determined that the target quantization parameter of the initial quantization network satisfies the preset update condition.

[0042] When it is determined that the ratio of the target weight pre-quantization parameter to all the weight pre-quantization parameters is less than or equal to the preset ratio, it is determined that the target quantization parameter of the initial quantization network does not meet the preset update condition.

[0043] As an optional implementation, in a second aspect of the present invention, the method by which the QAT training unit determines whether the initial quantization network satisfies the third preset convergence condition based on the second loss parameter and the determined validation set results of the initial quantization network specifically includes:

[0044] Based on the second loss parameter, determine whether the second loss parameter has converged, and based on the validation set results of the determined initial quantization network, determine whether the validation set results meet the preset network accuracy requirements;

[0045] When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, it is determined that the initial quantization network satisfies the third preset convergence condition.

[0046] When it is determined that the second loss parameter has not converged, and / or the validation set results do not meet the network accuracy requirements, it is determined that the initial quantization network does not meet the third preset convergence condition.

[0047] A third aspect of the present invention discloses another hysteresis quantization sensing training device for neural networks, the device comprising:

[0048] Memory containing executable program code;

[0049] A processor coupled to the memory;

[0050] The processor calls the executable program code stored in the memory to execute the hysteresis quantization perception training method for neural networks disclosed in the first aspect of the present invention.

[0051] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the hysteresis quantization perception training method for neural networks disclosed in the first aspect of the present invention.

[0052] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0053] In this embodiment of the invention, a pre-constructed neural network is trained to obtain a floating-point network that meets a first preset convergence condition. A pseudo-quantization node is inserted into the floating-point network, and pre-acquired training data is input into the floating-point network with the inserted pseudo-quantization node for PTQ calibration to obtain an initial quantization network. The initial quantization network is then trained using a preset weight update strategy, whereby the weight update strategy updates the weight parameters of the initial quantization network during the QAT training process and solves for the quantization parameters based on the weight parameters. If the model parameters of the determined initial quantization network meet a second preset convergence condition during the QAT training process, the quantization parameters of the initial quantization network are fixed as target quantization parameters, and a hysteresis strategy is added to the weight update strategy to obtain a weight hysteresis update strategy. The initial quantization network is then trained using the weight hysteresis update strategy to obtain a target quantization network that meets a third preset convergence condition. As can be seen, implementing this invention can train a pre-constructed neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes and feeding in training data, PTQ calibration is completed, followed by QAT training according to the weight update strategy. This combined training method of PTQ and QAT can improve the accuracy and precision of network training. Furthermore, if QAT training converges, the quantization parameters are fixed and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. This allows for phased QAT training using different quantization parameters, reducing the instability caused by changes in quantization parameters. Moreover, by adding a hysteresis strategy during weight parameter updates, the stability after weight updates can be improved, which is beneficial to improving the stability and reliability of network training. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1This is a flowchart illustrating a method for training a neural network using hysteresis quantization perception, as disclosed in an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of a floating-point quantization loop based on QAT training disclosed in an embodiment of the present invention;

[0057] Figure 3 This is a flowchart illustrating another method for training a neural network using hysteresis quantization perception, as disclosed in an embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram of a weight update curve disclosed in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of a rounding method for a weight quantization process disclosed in an embodiment of the present invention;

[0060] Figure 6 This is a histogram of the parameter quantities of the weighted parameters before quantization under different rounding error values, as disclosed in an embodiment of the present invention.

[0061] Figure 7 This is a flowchart illustrating another method for training a neural network using hysteresis quantization perception, as disclosed in an embodiment of the present invention.

[0062] Figure 8 This is a schematic diagram of the structure of a neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention;

[0063] Figure 9 This is a schematic diagram of the structure of another neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention;

[0064] Figure 10 This is a schematic diagram of the structure of another neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0067] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0068] This invention discloses a method, apparatus, and storage medium for hysteresis quantization-aware training of neural networks. It can train a pre-constructed neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes and feeding in training data, PTQ calibration is completed, followed by QAT training according to a weight update strategy. This combined PTQ and QAT training method improves the accuracy and precision of network training. Furthermore, if QAT training converges, the quantization parameters are fixed, and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. This allows for staged QAT training using different quantization parameters, reducing instability caused by quantization parameter changes. Adding a hysteresis strategy during weight parameter updates improves the stability after weight updates, thus enhancing the stability and reliability of network training. Detailed descriptions follow.

[0069] Example 1

[0070] Please see Figure 1 , Figure 1 This is a flowchart illustrating a hysteresis quantization perception training method for a neural network disclosed in an embodiment of the present invention. Figure 1 The described hysteresis quantization sensing training method for neural networks can be applied to a hysteresis quantization sensing training device for neural networks. This device may include a training equipment or a training server, where the training server may include a cloud server or a local server; this embodiment of the invention does not impose any limitations. Figure 1 As shown, the hysteresis quantization perception training method for this neural network can include the following operations:

[0071] 101. Perform training operations on the pre-constructed neural network to obtain a floating-point network trained to meet the first preset convergence condition.

[0072] In this embodiment of the invention, the first preset convergence condition is used to represent the condition that the weights in the neural network stably converge to the pre-weight range, the loss parameters during the training process continuously decrease to the preset loss range, and the condition that the neural network meets the accuracy requirements on the validation set.

[0073] 102. Insert pseudo-quantization nodes into the floating-point network, and input the pre-acquired training data into the floating-point network with inserted pseudo-quantization nodes for PTQ calibration to obtain the initial quantization network.

[0074] 103. Perform QAT training on the initial quantized network according to the preset weight update strategy.

[0075] In this embodiment of the invention, the weight update strategy is used to update the weight parameters of the initial quantized network during QAT training, and to solve for the quantization parameters based on the weight parameters. The quantization parameters map the range of floating-point values ​​to a discrete integer range, and may include scale and offset. Scale represents the quantization scale / scaling factor, used to determine the proportional relationship between floating-point and integer values, and offset represents the quantization offset / zero point, used to specify the integer value corresponding to the zero point of the floating-point number. Specifically, QAT training begins, the model weights and activation value ranges are statistically analyzed during training, and a reasonable scale and offset are calculated.

[0076] 104. If the model parameters of the determined initial quantization network meet the second preset convergence condition during QAT training, then fix the quantization parameters of the initial quantization network as the target quantization parameters, and add a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy.

[0077] In this embodiment of the invention, optionally, the model parameters of the initial quantization network include the model weights for each inference under the weight update strategy and the first loss parameter for each inference. The model weights include floating-point weights and / or integer weights corresponding to the floating-point weights. This embodiment of the invention does not limit the model weights.

[0078] In this embodiment of the invention, the second preset convergence condition is used to represent the condition that the loss parameter is reduced to a fixed level and the weight exhibits a preset high-frequency oscillation phenomenon during the QAT training process.

[0079] In this embodiment of the invention, the hysteresis strategy is used to represent adding hysteresis damping when updating the weight parameters, which can improve the stability after the weight update.

[0080] 105. Following the weight hysteresis update strategy, perform QAT training on the initial quantization network to obtain the target quantization network trained to meet the third preset convergence condition.

[0081] In this embodiment of the invention, the third preset convergence condition is used to represent the condition that the loss parameter converges during the training process and the validation set result meets the preset accuracy requirement.

[0082] In this embodiment of the invention, optionally, the hysteresis quantization perception training method of the neural network can be used to train an image processing model (such as the AIISP model), or to train a speech recognition model, a natural language processing model, or any other model capable of quantization processing. This embodiment of the invention does not limit the scope of the method.

[0083] For example, for AIISP (Artificial Intelligence Image Signal Processing) models, it is possible to base them on... Figure 1 The described training process trains a pre-constructed neural network using multiple raw image signals and target image signals obtained by processing each raw image signal, thereby obtaining the AIISP model. The target image signal obtained by processing each raw image signal can be obtained by performing at least one of the following processing steps: white balance, exposure control, color correction, sharpening, and noise reduction. Specifically, the raw image signals are used as the input to the neural network, and the target image signals are used as the output of the neural network. Figure 1 The described training process is used to train the AIISP model, which optimizes the traditional ISP (Image Signal Processing) process through neural network training. This enables processing such as white balance, exposure control, color correction, sharpening, and noise reduction, significantly improving image clarity and fidelity.

[0084] It is evident that implementation Figure 1 The described hysteresis quantization-aware training method for neural networks can train a pre-built neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes, feeding in training data, and completing PTQ calibration, QAT training is then performed according to the weight update strategy. This combined PTQ and QAT training approach improves the accuracy and precision of network training. If QAT training converges, the quantization parameters are fixed, and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. This allows for phased QAT training using different quantization parameters, reducing the instability caused by quantization parameter changes. Furthermore, by adding a hysteresis strategy during weight parameter updates, the stability after weight updates is improved, which is beneficial for enhancing the stability and reliability of network training.

[0085] In an optional embodiment, the model parameters of the initial quantized network during QAT training are determined in the following way:

[0086] During QAT training of the initial quantization network, the floating-point weights of this inference in the weight input module of the initial quantization network are input to the weight quantization module of the initial quantization network for quantization processing to obtain the integer weights corresponding to the floating-point weights; and the floating-point input data received by the floating-point input module of the initial quantization network are input to the input quantization module of the initial quantization network for quantization processing to obtain the integer input data corresponding to the floating-point input data.

[0087] The integer weights and integer input data are input into the convolutional layer module of the initial quantization network, and the output is sent to the floating-point output module of the initial quantization network to obtain floating-point output data.

[0088] The floating-point output data is input into the output quantization module of the initial quantization network for quantization processing to obtain the integer output data corresponding to the floating-point output data.

[0089] The first loss parameter of the initial quantization network for this inference is determined, and the first loss parameter of this inference is backpropagated by the output quantization module to the convolutional layer module for convolution calculation to obtain the convolution calculation result;

[0090] The convolutional layer module passes the convolution calculation results to the weight input module and the floating-point input module respectively. The weights are updated according to the weight update strategy to obtain the updated floating-point weights. The updated floating-point weights are used as the floating-point weights for the next inference and are applied to the floating-point output module during the next inference.

[0091] In this embodiment of the invention, the floating-point input data can be input data from a pre-input training set, and the training set can also include the real floating-point output data corresponding to each floating-point input data. The first loss parameter for this inference can be calculated by the real floating-point output data and the floating-point output data for this inference.

[0092] For example, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating a floating-point quantization loop based on QAT training, as disclosed in an embodiment of the present invention. Figure 2As shown, the floating-point weight Wf and the floating-point input data If are each scaled and quantized to obtain integer weight Wq and integer input data Iq, respectively. The integer weight Wq and integer input data Iq are then convolved in the convolutional layer module conv (during which the quantized integer data is converted back to floating-point data to complete the convolution calculation), resulting in floating-point output data Of. The floating-point output data Of is then quantized to obtain integer output data Oq. Subsequently, the loss parameter loss is backpropagated. Specifically, the output quantization module corresponding to the integer output data Oq backpropagates the loss parameter loss to the convolutional layer module conv for convolution calculation. The convolution calculation result is then passed to the weight input module corresponding to the floating-point weight Wf and the floating-point input module corresponding to the floating-point input data If. After updating the floating-point weight Wf based on the convolution calculation result, the updated floating-point weight Wf is applied to the floating-point output data Of in the next inference iteration to reduce the loss parameter loss in the next inference iteration.

[0093] Specifically, the quantization formula for converting floating-point weights Wf into integer weights Wq can be Wq=clip(round(Wf / scale),qmin,qmax), where Wf represents the floating-point ground truth during network training and Wq is the quantized value.

[0094] The scale is obtained statistically using Wf, most commonly through the minmax method. Changes in scale lead to changes in Wq, causing network instability. Therefore, this invention divides network training into two states: scale transient and scale steady state. The scale transient corresponds to the QAT training phase of the aforementioned weight update strategy. Its main task is to statistically determine a reasonable and stable scale to fix the scale and provide a stable network state for entering the scale steady state. The scale steady state corresponds to the QAT training phase of the aforementioned weight hysteresis update strategy. Its main task is to train floating-point weights Wf to obtain floating-point weights Wf with small quantization loss and low fluctuation.

[0095] As can be seen, this optional embodiment can improve the accuracy and reliability of quantizing floating-point data into integer weights and integer input data during QAT training of the initial quantization network. The initial quantization network quantizes floating-point weights and floating-point input data into integer weights and integer input data respectively, performs convolution operations to obtain floating-point output data, and then quantizes the floating-point output data into integer output data. Subsequently, the loss parameters for this inference are determined and backpropagated for convolution calculation. Based on the convolution calculation results, the floating-point weights are updated according to the weight update strategy, improving the accuracy and reliability of floating-point weight updates. Furthermore, the updated floating-point weights are applied to the next inference, which helps reduce the loss parameters for the next inference, thereby improving the accuracy and reliability of network training.

[0096] Example 2

[0097] Please see Figure 3 , Figure 3 This is a flowchart illustrating a hysteresis quantization perception training method for a neural network disclosed in an embodiment of the present invention. Figure 3 The described hysteresis quantization sensing training method for neural networks can be applied to a hysteresis quantization sensing training device for neural networks. This device may include a training equipment or a training server, where the training server may include a cloud server or a local server; this embodiment of the invention does not impose any limitations. Figure 3 As shown, the hysteresis quantization perception training method for this neural network can include the following operations:

[0098] 201. Perform training operations on the pre-constructed neural network to obtain a floating-point network trained to meet the first preset convergence condition.

[0099] 202. Insert pseudo-quantization nodes into the floating-point network, and input the pre-acquired training data into the floating-point network with inserted pseudo-quantization nodes for PTQ calibration to obtain the initial quantization network.

[0100] 203. Perform QAT training on the initial quantized network according to the preset weight update strategy.

[0101] 204. Based on the determined network inference results of the initial quantization network under the weight update strategy, determine whether the current conditions of the initial quantization network satisfy the second preset convergence condition.

[0102] In this embodiment of the invention, the network inference result may include: a second loss parameter for multiple inferences of the initial quantized network under the weight update strategy, and model weights for multiple inferences of the initial quantized network under the weight update strategy.

[0103] In this embodiment of the invention, when the judgment result of step 204 is yes, that is, when it is determined that the current condition of the initial quantization network meets the second preset convergence condition, step 205 is triggered; when the judgment result of step 204 is no, that is, when it is determined that the current condition of the initial quantization network does not meet the second preset convergence condition, the operation of QAT training of the initial quantization network according to the preset weight update strategy in step 203 is re-executed.

[0104] 205. Fix the initial quantization parameters of the quantization network as the target quantization parameters, and add a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy.

[0105] 206. Following the weight hysteresis update strategy, perform QAT training on the initial quantization network to obtain the target quantization network trained to meet the third preset convergence condition.

[0106] For further descriptions of steps 201-203, 205, and 206, please refer to the detailed description of steps 101-105 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0107] It is evident that implementation Figure 3 The described hysteresis quantization-aware training method for neural networks can train a pre-built neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes, feeding in training data, and completing PTQ calibration, QAT training is then performed according to the weight update strategy. This combined PTQ and QAT training approach improves the accuracy and precision of network training. If QAT training converges, the quantization parameters are fixed, and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. This allows for phased QAT training using different quantization parameters, reducing the instability caused by quantization parameter changes. Furthermore, by adding a hysteresis strategy during weight parameter updates, the stability after weight updates is improved, which is beneficial for enhancing the stability and reliability of network training. Furthermore, it can quickly and accurately determine whether the current conditions of the initial quantized network have converged based on the network inference results of the initial quantized network under the determined weight update strategy. If it does not converge, it continues to perform QAT training according to the original weight update strategy. If it converges, it performs QAT training according to the new weight hysteresis update strategy. This is beneficial for flexibly choosing to perform QAT training according to the original weight update strategy or the new weight hysteresis update strategy based on accurate judgment results.

[0108] In an optional embodiment, step 204 above, which determines whether the current conditions of the initial quantization network satisfy the second preset convergence condition based on the determined network inference results for the initial quantization network under the weight update strategy, may include:

[0109] Based on the first loss parameter for multiple inferences under the weight update strategy included in the network inference results, determine whether all first loss parameters within the preset number of continuous inferences are within the preset loss value range, and record it as the first sub-convergence index condition.

[0110] Based on the model weights for multiple inferences under the weight update strategy contained in the network inference results, the weight update curve corresponding to the weight update strategy is determined; and the weight update curve is divided according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and it is determined whether the weight change phenomenon corresponding to all target inference number ranges in the weight update curve has the preset high-frequency weight oscillation phenomenon, which is denoted as the second sub-convergence index condition.

[0111] Specifically, when both the first sub-convergence index condition and the second sub-convergence index condition are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; when the first sub-convergence index condition and / or the second sub-convergence index condition are not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0112] Specifically, when it is determined that all first loss parameters within a preset number of continuous inferences are within a preset loss value range, and there is a high-frequency oscillation phenomenon in the weight update curve, the current condition of the initial quantization network is determined to satisfy the second preset convergence condition; when it is determined that at least one first loss parameter within a preset number of continuous inferences is not within a preset loss value range, and / or there is no high-frequency oscillation phenomenon in the weight update curve, the current condition of the initial quantization network is determined not to satisfy the second preset convergence condition.

[0113] In this embodiment of the invention, specifically, multiple target model weights are extracted from the model weights of multiple inferences included in the network inference results according to a preset extraction method (such as random extraction), and a weight update curve is determined based on all target model weights. When high-frequency oscillations in the weights are observed, such as more than 6 repetitive cyclic updates within a target inference number range (e.g., 20 iterations), ... Figure 4 The blue curve shown indicates that the network has reached convergence. Figure 4 This is a schematic diagram of a weight update curve disclosed in an embodiment of the present invention.

[0114] As can be seen, this optional embodiment can determine whether all second loss parameters within a preset number of continuous inferences are within a preset loss value range based on the second loss parameters of multiple inferences included in the network inference results, denoted as the first sub-convergence index condition; and determine the weight update curve corresponding to the weight update strategy based on the model weights of the sub-inferences included in the network inference results; and divide the weight update curve according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and determine whether there is a preset high-frequency weight oscillation phenomenon in the weight change phenomenon corresponding to all target inference number ranges in the weight update curve, denoted as the second sub-convergence index condition. When all sub-convergence index conditions are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition, which improves the accuracy and reliability of determining that the current condition of the initial quantization network satisfies the second preset convergence condition. When at least one sub-convergence index condition is not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition, which improves the efficiency and speed of determining that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0115] In another optional embodiment, step 206 above, which involves performing QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain a target quantization network trained to meet a third preset convergence condition, includes:

[0116] According to the weight hysteresis update strategy, QAT training is performed on the initial quantization network to obtain multiple weight quantization related parameters for the initial quantization network under the weight hysteresis update strategy. The weight quantization related parameters include the parameters before weight quantization and the parameters after weight quantization.

[0117] Based on all weighting-related parameters, determine whether the target quantization parameters of the initial quantization network meet the preset update conditions; and when it is determined that the target quantization parameters of the initial quantization network meet the update conditions, re-execute the operation of QAT training of the initial quantization network according to the preset weight update strategy in step 203.

[0118] When it is determined that the target quantization parameters of the initial quantization network do not meet the update conditions, the second loss parameter of the initial quantization network under the weight hysteresis update strategy is determined, and based on the second loss parameter and the validation set results of the determined initial quantization network, it is determined whether the initial quantization network meets the third preset convergence condition.

[0119] When it is determined that the initial quantization network does not meet the third preset convergence condition, the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy in step 206 is re-executed to obtain the operation of multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and the operation of determining whether the target quantization parameter of the initial quantization network meets the preset update condition based on all weight quantization related parameters in step 206 is re-executed.

[0120] When it is determined that the initial quantization network satisfies the third preset convergence condition, the initial quantization network that satisfies the third preset convergence condition is determined as the target quantization network.

[0121] In this embodiment of the invention, the rounding method for the weight quantization process corresponding to the weight update strategy uses the round function. The round function rounds numbers according to a preset value (e.g., 0.5). For example, numbers less than 0.5 are rounded to zero, and numbers greater than or equal to 0.5 are rounded to 1. The rounding method for the weight hysteresis update strategy corresponding to the weight quantization process uses the hesitation-round function. The hesitation-round function rounds numbers according to a preset range (e.g., 0.3-0.8) containing the preset value. For example, numbers less than 0.3 are rounded to zero, numbers greater than 0.8 are rounded to 1, all numbers within the range less than 0.3 gradually increasing to 0.8 are rounded to zero, and all numbers within the range greater than 0.8 gradually decreasing to 0.3 are rounded to 1. Figure 5 As shown, Figure 5 This is a schematic diagram of a rounding method for a weight quantization process disclosed in an embodiment of the present invention, wherein... Figure 5 (a) indicates the rounding method used in the weight quantization process corresponding to the weight update strategy. Figure 5 (b) indicates the rounding method of the weight quantization process corresponding to the weight hysteresis update strategy. By using the rounding method of the weight quantization process corresponding to the weight hysteresis update strategy, it is beneficial to reduce the occurrence of weight update overshoot or invalid update due to the non-continuous Round function during the quantization process, which may cause the weight update to fail to reach the ideal value.

[0122] As can be seen, this optional embodiment can perform QAT training on the initial quantized network according to the weight hysteresis update strategy to obtain multiple weight quantization-related parameters of the initial quantized network under the weight hysteresis update strategy. This allows it to determine whether the target quantization parameters of the initial quantized network meet the update conditions. If the update conditions are met, QAT training is performed again according to the weight update strategy. This can effectively avoid frequent weight jumps caused by quantization after the quantization parameters have stabilized. By setting the update conditions that trigger the quantization parameters, the accuracy and timeliness of updating the target quantization parameters can be improved, reducing the occurrence of network oscillations caused by errors due to frequent updates of the target quantization parameters. If the update conditions are not met, it is further determined whether the network has converged. If it has not converged, QAT training continues according to the weight hysteresis update strategy. If it converges, the training is complete. This not only improves the accuracy of the target quantization parameters but also improves the accuracy of the network training convergence determination, which is beneficial to improving the network accuracy, training accuracy, stability, and reliability of the final trained target quantized network.

[0123] In this optional embodiment, as an optional implementation method, determining whether the target quantization parameters of the initial quantization network meet the preset update conditions based on all quantization-related parameters may include:

[0124] Based on the parameters related to each weight quantization, including the parameters before and after weight quantization, calculate the rounding error value introduced by the corresponding quantization operation before each weight quantization.

[0125] Based on the rounding error values ​​of all pre-quantization parameters, determine the target parameter quantity of the target pre-quantization parameter with a rounding error value of a preset error threshold, and calculate the ratio of the target pre-quantization parameter to all pre-quantization parameters based on the target parameter quantity and the total parameter quantity of all pre-quantization parameters.

[0126] Determine whether the ratio of the target weight's unquantized parameters to all unquantized weight parameters is greater than a preset ratio;

[0127] When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is greater than a preset ratio, the target quantization parameters of the initial quantization network are determined to meet the preset update conditions.

[0128] When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is less than or equal to a preset ratio, it is determined that the target quantization parameters of the initial quantization network do not meet the preset update conditions.

[0129] For example, by plotting a histogram of the parameters before weight quantization (e.g., weights and activation values ​​before quantization), the number of parameters before weight quantization under different rounding error values ​​can be statistically analyzed. Figure 6 As shown, Figure 6 This is a histogram of the parameter quantities of the weighted parameters before quantization under different rounding error values, as disclosed in an embodiment of the present invention. Figure 6 The histogram shown uses different rounding error values ​​on the horizontal axis and the number of parameters of the weighted pre-quantization parameters under different rounding error values ​​on the vertical axis. Based on this, it is determined whether the ratio of the target weighted pre-quantization parameters among all weighted pre-quantization parameters is greater than a preset ratio (e.g., whether the ratio of elements appearing in the first and last bins of the histogram is greater than 10%). If it is greater than or equal to the preset ratio, it is determined that the target quantization parameters of the initial quantization network meet the preset update conditions, and then the quantization parameter update operation is triggered. If it is less than the preset ratio, it is determined that the target quantization parameters of the initial quantization network do not meet the preset update conditions.

[0130] As can be seen, this optional implementation can calculate the rounding error value of the corresponding quantization operation introduced before each weight quantization based on the parameters before and after weight quantization, which are included in the relevant parameters of each weight quantization. This accurately determines the target parameter quantity of the target weight before quantization, with the rounding error value being a preset error threshold. Based on the target parameter quantity and the total parameter quantity of all weight before quantization parameters, the ratio of the target weight before quantization parameter to all weight before quantization parameters is calculated. This determines whether the ratio is greater than a preset ratio. If it is determined to be greater than the preset ratio, it is determined that the target quantization parameter of the initial quantization network meets the preset update condition, improving the accuracy and efficiency of determining whether the target quantization parameter of the initial quantization network meets the preset update condition, thereby improving the triggering accuracy and timeliness of updating the target quantization parameter. If it is determined to be less than or equal to the preset ratio, it is determined that the target quantization parameter of the initial quantization network does not meet the preset update condition, improving the accuracy and reliability of determining whether the target quantization parameter of the initial quantization network does not meet the preset update condition. This allows for accurate subsequent judgment of whether the network has been trained to convergence even when the target quantization parameter does not need to be updated, which is beneficial to improving the accuracy and reliability of network training to a certain extent.

[0131] In this optional embodiment, as another optional implementation, determining whether the initial quantization network satisfies the third preset convergence condition based on the second loss parameter and the validation set results of the determined initial quantization network may include:

[0132] Based on the second loss parameter, determine whether the second loss parameter has converged, and based on the validation set results of the determined initial quantized network, determine whether the validation set results meet the preset network accuracy requirements.

[0133] When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, the initial quantization network is determined to satisfy the third preset convergence condition.

[0134] When it is determined that the second loss parameter has not converged, and / or the validation set results do not meet the network accuracy requirements, it is determined that the initial quantization network does not meet the third preset convergence condition.

[0135] In this embodiment of the invention, specifically, after the initial quantization network is trained using QAT according to the weight hysteresis update strategy, the initial quantization network is validated based on the pre-obtained validation set to obtain the validation set result of the initial quantization network.

[0136] As can be seen, this optional implementation can, when it is determined that the target quantization parameters of the initial quantization network do not need to be updated, further determine whether the second loss parameter of the initial quantization network under the weight hysteresis update strategy has converged, and determine whether the validation set results of the determined initial quantization network meet the preset network accuracy requirements. When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, it is determined that the initial quantization network meets the third preset convergence condition, which improves the accuracy and reliability of determining that the initial quantization network meets the third preset convergence condition. Thus, training is stopped and the initial quantization network is determined as the target quantization network after training, which is beneficial to improving the accuracy and reliability of network training. When it is determined that the second loss parameter has not converged and / or the validation set results do not meet the network accuracy requirements, it is directly determined that the initial quantization network does not meet the third preset convergence condition, which improves the efficiency and speed of determining that the initial quantization network does not meet the third preset convergence condition. Thus, QAT training continues according to the weight hysteresis update strategy, which is beneficial to improving the training accuracy and reliability of the neural network through multiple conditions.

[0137] For example, such as Figure 7 As shown, Figure 7 This is a flowchart illustrating another method for training a neural network using hysteresis quantization perception, as disclosed in an embodiment of the present invention. This method may include the following steps:

[0138] S1. Training a floating-point network until the network converges is characterized by the following: the weights in the network converge stably to a certain range, the training loss continuously decreases to a reasonable range, and the network meets the accuracy requirements on the validation set.

[0139] S2. Insert pseudo-quantization nodes into the model, feed in training data, and complete PTQ.

[0140] S3. Start QAT training, count the model weights and activation value ranges during training, and solve for a reasonable scale and offset. The scale calculated at this time is recorded as scale0.

[0141] S4. Determine if QAT training has converged. The convergence criteria are: a. The training loss decreases to a fixed level without significant, clearly directional updates. b. Randomly sample multiple (5-10) weights from the model and plot their update curves. It is observed that the weights exhibit high-frequency oscillations, with more than 6 repetitive cyclical updates in 20 iterations. Figure 4 The solid blue line indicates that the network has entered convergence. If the convergence is successful, the process proceeds to S5; otherwise, QAT training continues in S3.

[0142] S5. Fix the quantization parameters and enter stage 1. At this point, the scale and offset are no longer updated based on the weights and activation statistics. Continue QAT training and perform hesitation-round on the weight update strategy according to the step and lr inheritance steps (i.e., the continuation of the standard gradient descent optimization process in QAT).

[0143] S6. During training, draw a histogram of weights and activation values ​​before quantization and determine whether the scale needs to be updated. The judgment condition is: the number of elements appearing in the first and last bins is greater than 10% of the total number, i.e., whether the scale adjustment operation is triggered. If yes, proceed to step S3; if no (i.e., no update is needed), proceed to S7.

[0144] S7. Determine if the training loss has converged and if the validation set results meet the requirements. If they do, stop training; otherwise, return to step S5 for QAT training. This method trains a pre-built neural network to convergence, inserts pseudo-quantization nodes, feeds in training data, completes PTQ calibration, and then performs QAT training according to the weight update strategy. This combined PTQ and QAT training approach improves network training accuracy and precision. If QAT training converges, the quantization parameters are fixed, and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new hysteresis strategy until convergence. This allows for phased QAT training using different quantization parameters, reducing instability caused by quantization parameter changes. Furthermore, adding a hysteresis strategy during weight parameter updates improves stability after weight updates, enhancing network training stability and reliability.

[0145] Example 3

[0146] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a hysteresis quantization perception training device for a neural network disclosed in an embodiment of the present invention. Figure 8The described neural network hysteresis quantization perception training device may include a training device or a training server, wherein the training server may include a cloud server or a local server, and the embodiments of the present invention are not limited thereto. Figure 8 As shown, the hysteresis quantization sensing training device for this neural network may include:

[0147] The floating-point training unit 301 is used to perform training operations on a pre-built neural network to obtain a floating-point network trained to meet a first preset convergence condition.

[0148] The PTQ calibration unit 302 is used to insert pseudo-quantization nodes into the floating-point network and input the pre-acquired training data into the floating-point network with inserted pseudo-quantization nodes for PTQ calibration to obtain the initial quantization network.

[0149] QAT training unit 303 is used to perform QAT training on the initial quantized network according to a preset weight update strategy. The weight update strategy is used to update the weight parameters of the initial quantized network during the QAT training process and solve the quantization parameters based on the weight parameters.

[0150] The hysteresis update unit 304 is used to fix the quantization parameters of the initial quantization network as the target quantization parameters if the model parameters of the determined initial quantization network meet the second preset convergence condition during QAT training, and to add a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy.

[0151] The QAT training unit 303 is also used to perform QAT training on the initial quantization network according to the weight hysteresis update strategy, so as to obtain the target quantization network trained to meet the third preset convergence condition.

[0152] It is evident that implementation Figure 8 The described neural network hysteresis quantization-aware training device can train a pre-built neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes, feeding in training data, and completing PTQ calibration, it then performs QAT training according to the weight update strategy. This combined PTQ and QAT training method can improve the accuracy and precision of network training. If QAT training converges, the quantization parameters are fixed and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. It can perform QAT training in stages using different quantization parameters, reducing the instability of network training caused by changes in quantization parameters. Furthermore, by adding a hysteresis strategy during weight parameter updates, the stability after weight updates can be improved, which is beneficial to improving the stability and reliability of network training.

[0153] In an optional embodiment, the model parameters of the initial quantized network during QAT training are determined in the following way:

[0154] During QAT training of the initial quantization network, the floating-point weights of this inference in the weight input module of the initial quantization network are input to the weight quantization module of the initial quantization network for quantization processing to obtain the integer weights corresponding to the floating-point weights; and the floating-point input data received by the floating-point input module of the initial quantization network are input to the input quantization module of the initial quantization network for quantization processing to obtain the integer input data corresponding to the floating-point input data.

[0155] The integer weights and integer input data are input into the convolutional layer module of the initial quantization network, and the output is sent to the floating-point output module of the initial quantization network to obtain floating-point output data.

[0156] The floating-point output data is input into the output quantization module of the initial quantization network for quantization processing to obtain the integer output data corresponding to the floating-point output data.

[0157] The first loss parameter of the initial quantization network for this inference is determined, and the first loss parameter of this inference is backpropagated by the output quantization module to the convolutional layer module for convolution calculation to obtain the convolution calculation result;

[0158] The convolutional layer module passes the convolution calculation results to the weight input module and the floating-point input module respectively. The weights are updated according to the weight update strategy to obtain the updated floating-point weights. The updated floating-point weights are used as the floating-point weights for the next inference and are applied to the floating-point output module in the next inference.

[0159] The model parameters of the initial quantization network include the model weights for each inference under the weight update strategy and the first loss parameter for each inference. The model weights include floating-point weights and / or integer weights corresponding to the floating-point weights.

[0160] As can be seen, this optional embodiment can improve the accuracy and reliability of quantizing floating-point data into integer weights and integer input data during QAT training of the initial quantization network. The initial quantization network quantizes floating-point weights and floating-point input data into integer weights and integer input data respectively, performs convolution operations to obtain floating-point output data, and then quantizes the floating-point output data into integer output data. Subsequently, the loss parameters for this inference are determined and backpropagated for convolution calculation. Based on the convolution calculation results, the floating-point weights are updated according to the weight update strategy, improving the accuracy and reliability of floating-point weight updates. Furthermore, the updated floating-point weights are applied to the next inference, which helps reduce the loss parameters for the next inference, thereby improving the accuracy and reliability of network training.

[0161] In this optional embodiment, as an optional implementation method, such as Figure 9 As shown, Figure 9 This is a schematic diagram of another neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention, wherein the device may further include:

[0162] The judgment unit 305 is used to determine whether the current condition of the initial quantization network meets the second preset convergence condition based on the determined network inference result of the initial quantization network under the weight update strategy; and when the judgment unit 305 determines that the current condition of the initial quantization network meets the second preset convergence condition, it triggers the hysteresis update unit 304 to perform the operation of fixing the quantization parameters of the initial quantization network as the target quantization parameters and adding a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy, and triggers the QAT training unit 303 to perform the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy until the initial quantization network meets the second preset convergence condition.

[0163] The QAT training unit 303 is also used to re-execute the operation of QAT training the initial quantization network according to the preset weight update strategy when the judgment unit 305 determines that the current conditions of the initial quantization network do not meet the second preset convergence condition.

[0164] As can be seen, this optional implementation can quickly and accurately determine whether the current conditions of the initial quantized network have converged based on the determined network inference results of the initial quantized network under the weight update strategy. If it does not converge, QAT training continues according to the original weight update strategy. If it converges, QAT training is performed according to the new weight hysteresis update strategy. This is beneficial for flexibly choosing to perform QAT training according to the original weight update strategy or the new weight hysteresis update strategy based on accurate judgment results.

[0165] In this optional implementation, the method by which the judging unit 305 judges whether the current conditions of the initial quantization network satisfy the second preset convergence condition based on the determined network inference results for the initial quantization network under the weight update strategy specifically includes:

[0166] Based on the first loss parameter for multiple inferences under the weight update strategy included in the network inference results, determine whether all first loss parameters within the preset number of continuous inferences are within the preset loss value range, and record it as the first sub-convergence index condition.

[0167] Based on the model weights for multiple inferences under the weight update strategy contained in the network inference results, the weight update curve corresponding to the weight update strategy is determined; and the weight update curve is divided according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and it is determined whether the weight change phenomenon corresponding to all target inference number ranges in the weight update curve has the preset high-frequency weight oscillation phenomenon, which is denoted as the second sub-convergence index condition.

[0168] Specifically, when both the first sub-convergence index condition and the second sub-convergence index condition are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; when the first sub-convergence index condition and / or the second sub-convergence index condition are not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0169] As can be seen, this optional implementation can also determine whether all second loss parameters within a preset number of continuous inferences are within a preset loss value range based on the second loss parameters of multiple inferences included in the network inference results, and record this as the first sub-convergence index condition; and determine the weight update curve corresponding to the weight update strategy based on the model weights of the sub-inferences included in the network inference results; and divide the weight update curve according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and determine whether there is a preset high-frequency weight oscillation phenomenon in the weight change phenomenon corresponding to all target inference number ranges in the weight update curve, and record this as the second sub-convergence index condition. When all sub-convergence index conditions are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition, which improves the accuracy and reliability of determining that the current condition of the initial quantization network satisfies the second preset convergence condition. When at least one sub-convergence index condition is not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition, which improves the efficiency and speed of determining that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0170] In another optional embodiment, the QAT training unit 303 performs QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain the target quantization network that satisfies the third preset convergence condition. Specifically, this includes:

[0171] According to the weight hysteresis update strategy, QAT training is performed on the initial quantization network to obtain multiple weight quantization related parameters for the initial quantization network under the weight hysteresis update strategy. The weight quantization related parameters include the parameters before weight quantization and the parameters after weight quantization.

[0172] Based on all weighting-related parameters, determine whether the target quantization parameters of the initial quantization network meet the preset update conditions; and when it is determined that the target quantization parameters of the initial quantization network meet the update conditions, re-execute the operation of QAT training on the initial quantization network according to the preset weight update strategy.

[0173] When it is determined that the target quantization parameters of the initial quantization network do not meet the update conditions, the second loss parameter of the initial quantization network under the weight hysteresis update strategy is determined, and based on the second loss parameter and the validation set results of the determined initial quantization network, it is determined whether the initial quantization network meets the third preset convergence condition.

[0174] When it is determined that the initial quantization network does not meet the third preset convergence condition, the QAT training of the initial quantization network is re-executed according to the weight hysteresis update strategy to obtain the operation of multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and the operation of determining whether the target quantization parameter of the initial quantization network meets the preset update condition based on all weight quantization related parameters is re-executed.

[0175] When it is determined that the initial quantization network satisfies the third preset convergence condition, the initial quantization network that satisfies the third preset convergence condition is determined as the target quantization network.

[0176] As can be seen, this optional embodiment can perform QAT training on the initial quantized network according to the weight hysteresis update strategy to obtain multiple weight quantization-related parameters of the initial quantized network under the weight hysteresis update strategy. This allows it to determine whether the target quantization parameters of the initial quantized network meet the update conditions. If the update conditions are met, QAT training is performed again according to the weight update strategy. This can effectively avoid frequent weight jumps caused by quantization after the quantization parameters have stabilized. By setting the update conditions that trigger the quantization parameters, the accuracy and timeliness of updating the target quantization parameters can be improved, reducing the occurrence of network oscillations caused by errors due to frequent updates of the target quantization parameters. If the update conditions are not met, it is further determined whether the network has converged. If it has not converged, QAT training continues according to the weight hysteresis update strategy. If it converges, the training is complete. This not only improves the accuracy of the target quantization parameters but also improves the accuracy of the network training convergence determination, which is beneficial to improving the network accuracy, training accuracy, stability, and reliability of the final trained target quantized network.

[0177] In this optional embodiment, as an optional implementation, the QAT training unit 303 determines whether the target quantization parameters of the initial quantization network meet the preset update conditions based on all quantization-related parameters in the following specific ways:

[0178] Based on the parameters related to each weight quantization, including the parameters before and after weight quantization, calculate the rounding error value introduced by the corresponding quantization operation before each weight quantization.

[0179] Based on the rounding error values ​​of all pre-quantization parameters, determine the target parameter quantity of the target pre-quantization parameter with a rounding error value of a preset error threshold, and calculate the ratio of the target pre-quantization parameter to all pre-quantization parameters based on the target parameter quantity and the total parameter quantity of all pre-quantization parameters.

[0180] Determine whether the ratio of the target weight's unquantized parameters to all unquantized weight parameters is greater than a preset ratio;

[0181] When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is greater than a preset ratio, the target quantization parameters of the initial quantization network are determined to meet the preset update conditions.

[0182] When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is less than or equal to a preset ratio, it is determined that the target quantization parameters of the initial quantization network do not meet the preset update conditions.

[0183] As can be seen, this optional implementation can calculate the rounding error value of the corresponding quantization operation introduced before each weight quantization based on the parameters before and after weight quantization, which are included in the relevant parameters of each weight quantization. This accurately determines the target parameter quantity of the target weight before quantization, with the rounding error value being a preset error threshold. Based on the target parameter quantity and the total parameter quantity of all weight before quantization parameters, the ratio of the target weight before quantization parameter to all weight before quantization parameters is calculated. This determines whether the ratio is greater than a preset ratio. If it is determined to be greater than the preset ratio, it is determined that the target quantization parameter of the initial quantization network meets the preset update condition, improving the accuracy and efficiency of determining whether the target quantization parameter of the initial quantization network meets the preset update condition, thereby improving the triggering accuracy and timeliness of updating the target quantization parameter. If it is determined to be less than or equal to the preset ratio, it is determined that the target quantization parameter of the initial quantization network does not meet the preset update condition, improving the accuracy and reliability of determining whether the target quantization parameter of the initial quantization network does not meet the preset update condition. This allows for accurate subsequent judgment of whether the network has been trained to convergence even when the target quantization parameter does not need to be updated, which is beneficial to improving the accuracy and reliability of network training to a certain extent.

[0184] In this optional embodiment, as another optional implementation, the QAT training unit 303 determines whether the initial quantization network satisfies the third preset convergence condition based on the second loss parameter and the determined validation set results of the initial quantization network. Specifically, this includes:

[0185] Based on the second loss parameter, determine whether the second loss parameter has converged, and based on the validation set results of the determined initial quantized network, determine whether the validation set results meet the preset network accuracy requirements.

[0186] When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, the initial quantization network is determined to satisfy the third preset convergence condition.

[0187] When it is determined that the second loss parameter has not converged, and / or the validation set results do not meet the network accuracy requirements, it is determined that the initial quantization network does not meet the third preset convergence condition.

[0188] As can be seen, this optional implementation can, when it is determined that the target quantization parameters of the initial quantization network do not need to be updated, further determine whether the second loss parameter of the initial quantization network under the weight hysteresis update strategy has converged, and determine whether the validation set results of the determined initial quantization network meet the preset network accuracy requirements. When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, it is determined that the initial quantization network meets the third preset convergence condition, which improves the accuracy and reliability of determining that the initial quantization network meets the third preset convergence condition. Thus, training is stopped and the initial quantization network is determined as the target quantization network after training, which is beneficial to improving the accuracy and reliability of network training. When it is determined that the second loss parameter has not converged and / or the validation set results do not meet the network accuracy requirements, it is directly determined that the initial quantization network does not meet the third preset convergence condition, which improves the efficiency and speed of determining that the initial quantization network does not meet the third preset convergence condition. Thus, QAT training continues according to the weight hysteresis update strategy, which is beneficial to improving the training accuracy and reliability of the neural network through multiple conditions.

[0189] Example 4

[0190] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of another neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention. Figure 10 As shown, the hysteresis quantization sensing training device for this neural network may include:

[0191] Memory 401 storing executable program code;

[0192] Processor 402 coupled to memory 401;

[0193] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the hysteresis quantization perception training method for neural networks described in Embodiment 1 or Embodiment 2 of the present invention.

[0194] Example 5

[0195] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the hysteresis quantization perception training method for neural networks described in Embodiment 1 or Embodiment 2 of this invention.

[0196] Example 6

[0197] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the hysteresis quantization perception training method for a neural network described in Embodiment 1 or Embodiment 2.

[0198] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0199] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0200] Finally, it should be noted that the hysteresis quantization perception training method and apparatus for neural networks and the storage medium disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a neural network using hysteresis quantization perception, characterized in that, The method includes: Training operations are performed on a pre-constructed neural network using multiple original image signals and target image signals obtained by processing each of the original image signals, to obtain a floating-point network trained to meet a first preset convergence condition. The original image signals are used as the input of the neural network, and the target image signals are used as the output of the neural network. The neural network is an image processing model. A pseudo-quantization node is inserted into the floating-point network, and the pre-acquired training data is input into the floating-point network into which the pseudo-quantization node has been inserted for PTQ calibration to obtain an initial quantization network. The training data includes the original image signal and the target image signal. According to a preset weight update strategy, the initial quantization network is trained using QAT. The weight update strategy is used to update the weight parameters of the initial quantization network during the QAT training process, and to solve for the quantization parameters based on the weight parameters. If the model parameters of the determined initial quantization network meet the second preset convergence condition during the QAT training process, then the quantization parameters of the initial quantization network are fixed as the target quantization parameters, and a hysteresis strategy is added to the weight update strategy to obtain a weight hysteresis update strategy. Then, according to the weight hysteresis update strategy, the initial quantization network is trained with QAT to obtain a target quantization network trained to meet the third preset convergence condition. The target quantization network is a trained image processing model. The trained image processing model is used to perform at least one of white balance, exposure control, color correction, sharpening, and noise reduction on the image signal to be processed to obtain the processed image signal. The second preset convergence condition includes a first sub-convergence index condition and a second sub-convergence index condition. The first sub-convergence index condition is used to indicate that all first loss parameters within the determined preset number of continuous inferences are within the preset loss range. The second sub-convergence index condition is used to indicate that the weight change phenomenon corresponding to all target inference number ranges in the determined weight update curve has a preset high-frequency weight oscillation phenomenon. The weight quantization process of the weight hysteresis update strategy adopts hesitation as the rounding method. The round function, where hesitation The round function is used to round numbers according to a preset range of preset values. The rounding method of the weight quantization process specifically includes: rounding numbers less than a first preset value to zero, rounding numbers greater than a second preset value to 1, rounding all numbers within a range of numbers less than the first preset value and gradually increasing to the second preset value to zero, and rounding all numbers within a range of numbers greater than the second preset value and gradually decreasing to the first preset value to 1, wherein the first preset value is less than the second preset value.

2. The method for training a neural network with hysteresis quantization perception according to claim 1, characterized in that, The model parameters of the initial quantized network during the QAT training process are determined in the following way: During QAT training of the initial quantization network, the floating-point weights of this inference in the weight input module of the initial quantization network are input to the weight quantization module of the initial quantization network for quantization processing to obtain the integer weights corresponding to the floating-point weights. In addition, the floating-point input data received by the floating-point input module included in the initial quantization network is input to the input quantization module included in the initial quantization network for quantization processing to obtain the integer input data corresponding to the floating-point input data; The integer weights and the integer input data are input into the convolutional layer module of the initial quantization network for convolution operation, and then output to the floating-point output module of the initial quantization network to obtain floating-point output data; The floating-point output data is input to the output quantization module of the initial quantization network for quantization processing to obtain the integer output data corresponding to the floating-point output data. The first loss parameter of the initial quantization network for this inference is determined, and the first loss parameter of this inference is backpropagated to the convolutional layer module by the output quantization module to perform convolution calculation to obtain the convolution calculation result; The convolutional layer module transmits the convolution calculation result to the weight input module and the floating-point input module respectively, and updates it according to the weight update strategy to obtain the updated floating-point weight. The updated floating-point weight is used as the floating-point weight for the next inference and is applied to the floating-point output module during the next inference. The model parameters of the initial quantization network include the model weights for each inference under the weight update strategy and the first loss parameter for each inference. The model weights include the floating-point weights and / or the integer weights corresponding to the floating-point weights.

3. The method for training a neural network with hysteresis quantization perception according to claim 2, characterized in that, The method further includes: Based on the determined network inference results of the initial quantization network under the weight update strategy, it is determined whether the current condition of the initial quantization network satisfies the second preset convergence condition; and when it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition, the operation of fixing the quantization parameters of the initial quantization network as the target quantization parameters and adding a hysteresis strategy to the weight update strategy to obtain a weight hysteresis update strategy is triggered, and the operation of performing QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain a target quantization network trained to satisfy the third preset convergence condition is triggered. When it is determined that the current conditions of the initial quantization network do not meet the second preset convergence condition, the operation of QAT training of the initial quantization network according to the preset weight update strategy is re-executed.

4. The hysteresis quantization perception training method for neural networks according to claim 3, characterized in that, The step of determining whether the current conditions of the initial quantization network satisfy the second preset convergence condition based on the determined network inference results of the initial quantization network under the weight update strategy includes: Based on the first loss parameter included in the network inference result for multiple inferences under the weight update strategy, determine whether all first loss parameters within a preset number of continuous inferences are within a preset loss value range, and record this as the first sub-convergence index condition. Based on the model weights included in the network inference results for multiple inferences under the weight update strategy, the weight update curve corresponding to the weight update strategy is determined; and the weight update curve is divided according to a preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and it is determined whether there is a preset high-frequency weight oscillation phenomenon in all the weight change phenomena corresponding to the target inference number range in the weight update curve, which is denoted as the second sub-convergence index condition. Specifically, when both the first sub-convergence index condition and the second sub-convergence index condition are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; when the first sub-convergence index condition and / or the second sub-convergence index condition are not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

5. The method for training a neural network with hysteresis quantization perception according to any one of claims 1-4, characterized in that, The step of training the initial quantization network using QAT according to the weight hysteresis update strategy to obtain the target quantization network trained to meet the third preset convergence condition includes: According to the weight hysteresis update strategy, the initial quantization network is trained with QAT to obtain multiple weight quantization related parameters for the initial quantization network under the weight hysteresis update strategy. The weight quantization related parameters include parameters before weight quantization and parameters after weight quantization. Based on all the weight quantization-related parameters, determine whether the target quantization parameters of the initial quantization network meet the preset update conditions; and when it is determined that the target quantization parameters of the initial quantization network meet the update conditions, re-execute the operation of QAT training of the initial quantization network according to the preset weight update strategy. When it is determined that the target quantization parameters of the initial quantization network do not meet the update conditions, a second loss parameter for the initial quantization network under the weight hysteresis update strategy is determined, and based on the second loss parameter and the determined validation set results of the initial quantization network, it is determined whether the initial quantization network meets the third preset convergence condition. When it is determined that the initial quantization network does not meet the third preset convergence condition, the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy is re-executed to obtain multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and the operation of determining whether the target quantization parameter of the initial quantization network meets the preset update condition based on all the weight quantization related parameters is re-executed. When it is determined that the initial quantization network satisfies the third preset convergence condition, the initial quantization network that satisfies the third preset convergence condition is determined as the target quantization network.

6. The method for training a neural network with hysteresis quantization perception according to claim 5, characterized in that, The step of determining whether the target quantization parameters of the initial quantization network meet the preset update conditions based on all the weight quantization-related parameters includes: Based on the parameters before and after weight quantization included in each of the weight quantization related parameters, calculate the rounding error value of the corresponding quantization operation introduced before each weight quantization. Based on the rounding error values ​​of all the pre-quantization weighted parameters, determine the target parameter quantity of the target pre-quantization weighted parameter whose rounding error value is a preset error threshold, and calculate the ratio of the target pre-quantization weighted parameter to all the pre-quantization weighted parameters based on the target parameter quantity and the total parameter quantity of all the pre-quantization weighted parameters. Determine whether the ratio of the target weight pre-quantization parameter among all the weight pre-quantization parameters is greater than a preset ratio; When it is determined that the ratio of the target weight pre-quantization parameter to all the weight pre-quantization parameters is greater than the preset ratio, it is determined that the target quantization parameter of the initial quantization network satisfies the preset update condition. When it is determined that the ratio of the target weight pre-quantization parameter to all the weight pre-quantization parameters is less than or equal to the preset ratio, it is determined that the target quantization parameter of the initial quantization network does not meet the preset update condition.

7. The method for training a neural network with hysteresis quantization perception according to claim 5, characterized in that, The step of determining whether the initial quantization network satisfies the third preset convergence condition based on the second loss parameter and the determined validation set results of the initial quantization network includes: Based on the second loss parameter, determine whether the second loss parameter has converged, and based on the validation set results of the determined initial quantization network, determine whether the validation set results meet the preset network accuracy requirements; When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, it is determined that the initial quantization network satisfies the third preset convergence condition. When it is determined that the second loss parameter has not converged, and / or the validation set results do not meet the network accuracy requirements, it is determined that the initial quantization network does not meet the third preset convergence condition.

8. A hysteresis quantization sensing training device for a neural network, characterized in that, The apparatus is used to perform the hysteresis quantization perception training method for a neural network as described in any one of claims 1-7, and the apparatus comprises: A floating-point training unit is used to perform training operations on a pre-constructed neural network using multiple original image signals and a target image signal obtained by processing each of the original image signals, to obtain a floating-point network trained to meet a first preset convergence condition. The original image signals are used as the input of the neural network, and the target image signals are used as the output of the neural network. The neural network is an image processing model. The PTQ calibration unit is used to insert pseudo-quantization nodes into the floating-point network and input the pre-acquired training data into the floating-point network with the pseudo-quantization nodes inserted to perform PTQ calibration, thereby obtaining an initial quantization network. The training data includes the original image signal and the target image signal. The QAT training unit is used to perform QAT training on the initial quantization network according to a preset weight update strategy. The weight update strategy is used to update the weight parameters of the initial quantization network during the QAT training process and solve the quantization parameters based on the weight parameters. The hysteresis update unit is used to fix the quantization parameters of the initial quantization network as the target quantization parameters if the model parameters of the determined initial quantization network meet the second preset convergence condition during the QAT training process, and to add a hysteresis strategy to the weight update strategy to obtain a weight hysteresis update strategy. The QAT training unit is further configured to perform QAT training on the initial quantization network according to the weight hysteresis update strategy, to obtain a target quantization network trained to meet the third preset convergence condition. The target quantization network is a trained image processing model. The trained image processing model is used to perform at least one of white balance, exposure control, color correction, sharpening and noise reduction on the image signal to be processed, to obtain the processed image signal.

9. A hysteresis quantization sensing training device for a neural network, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the hysteresis quantization perception training method for the neural network as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the hysteresis quantization perception training method for the neural network as described in any one of claims 1-7.

Images