Neural network hysteresis quantization perception training method and device, and storage medium

Abstract

The invention relates to the technical field of deep learning, and discloses a hysteresis quantitative perception training method and device for a neural network, and a storage medium, and the method comprises the steps: training a pre-constructed neural network to obtain a floating point network meeting a first preset convergence condition; inserting a pseudo quantization node into the network, and inputting pre-acquired training data to carry out PTQ calibration to obtain an initial quantization network; performing QAT training on the initial quantization network according to a preset weight updating strategy; and if the model parameter meets a second preset convergence condition in the QAT training process, fixing a quantization parameter of the initial quantization network as a target quantization parameter, adding a hysteresis strategy in a weight updating strategy to obtain a weight hysteresis updating strategy, and performing QAT training on the initial quantization network according to the weight hysteresis updating strategy to obtain a target quantization parameter. And obtaining a target quantization network meeting a third preset convergence condition. Therefore, by implementing the method, the network precision can be improved, and the network training stability and reliability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a neural network hysteresis quantization-aware training method and device and a storage medium. BACKGROUND

[0002] In scenarios such as mobile devices, embedded systems and edge computing that require efficient deployment and operation of deep learning models, a post-training quantization (PTQ) method is usually used. After the model training is completed, based on quantization technology, the high-precision (such as 32-bit floating-point number) weights and activation values in the model are converted to low-precision (such as 8-bit integer) to represent.

[0003] In practice, based on the traditional post-quantization training method, the network accuracy is easily reduced due to the introduction of information loss and nonlinear distortion in the quantization process. The quantization-aware training (QAT) method is proposed because it can introduce quantization simulation in the training phase to make the model adapt to the state after quantization, thereby ensuring that the network accuracy remains high after quantization.

[0004] However, it is found in practice that the introduction of the round function based on the existing QAT technology in model training makes the backpropagation gradient zero, which causes the weights to be unable to update. The current industry generally uses the straight-through estimator (STE) method, which equates the gradient of the quantization operation to the identity mapping, that is, in the backpropagation, the gradient of the quantization operation is considered as the gradient of the input without considering the nonlinear distortion caused by the quantization operation itself. Its essence is gradient estimation rather than accurate mathematical solution, which easily leads to unstable training problems such as oscillation and non-convergence in the training process, thereby limiting the stability and reliability of the model. Therefore, it is particularly important to propose a technical solution to improve network accuracy and improve network training stability and reliability. SUMMARY The present application provides a neural network hysteresis quantization-aware training method and device, and a storage medium, which can improve network accuracy and improve network training stability and reliability.

[0005] To solve the above technical problems, the first aspect of the present application discloses a neural network hysteresis quantization-aware training method, which comprises: performing a training operation on a pre-constructed neural network to obtain a floating-point network trained to satisfy a first preset convergence condition; inserting a pseudo-quantization node in the floating-point network, and inputting pre-acquired training data into the floating-point network in which the pseudo-quantization node is inserted to perform PTQ calibration, to obtain an initial quantization network; updating the initial quantization network according to a preset weight update strategy, the weight update strategy being used to update weight parameters of the initial quantization network in the process of the QAT training, and quantization parameters being solved according to the weight parameters; if the model parameters of the initial quantization network determined in the process of the QAT training satisfy a second preset convergence condition, fixing the quantization parameters of the initial quantization network as target quantization parameters, adding a hysteresis strategy in the weight update strategy to obtain a weight hysteresis update strategy, and updating the initial quantization network according to the weight hysteresis update strategy to obtain a target quantization network trained to satisfy a third preset convergence condition.

[0006] The second aspect of the present application discloses a hysteresis quantization-aware training device of a neural network, the device comprising: a floating-point training unit configured to perform a training operation on a pre-constructed neural network to obtain a floating-point network trained to satisfy a first preset convergence condition; a PTQ calibration unit configured to insert a pseudo-quantization node in the floating-point network, and input pre-acquired training data into the floating-point network in which the pseudo-quantization node is inserted to perform PTQ calibration, to obtain an initial quantization network; a QAT training unit configured to update the initial quantization network according to a preset weight update strategy, the weight update strategy being used to update weight parameters of the initial quantization network in the process of the QAT training, and quantization parameters being solved according to the weight parameters; a hysteresis update unit configured to, if the model parameters of the initial quantization network determined in the process of the QAT training satisfy a second preset convergence condition, fix the quantization parameters of the initial quantization network as target quantization parameters, and add a hysteresis strategy in the weight update strategy to obtain a weight hysteresis update strategy; the QAT training unit is further configured to update the initial quantization network according to the weight hysteresis update strategy to obtain a target quantization network trained to satisfy a third preset convergence condition.

[0007] As an optional implementation, in the second aspect of the present application, the model parameters of the initial quantization network in the process of the QAT training are determined in the following manner: In the process of QAT training of the initial quantization network, the floating point weight of this inference in the weight input module included in the initial quantization network is input to the weight quantization module included in the initial quantization network for quantization processing, to obtain the integer type weight corresponding to the floating point weight; and the floating point input data received by the floating point input module included in the initial quantization network is input to the input quantization module included in the initial quantization network for quantization processing, to obtain the integer type input data corresponding to the floating point input data; The integer type weight and the integer type input data are input to the convolution operation of the convolution layer module of the initial quantization network, and the convolution operation output is output to the floating point output module of the initial quantization network, to obtain floating point output data; The floating point output data is input to the output quantization module of the initial quantization network for quantization processing, to obtain the integer type output data corresponding to the floating point output data; A first loss parameter of this inference of the initial quantization network is determined, and the first loss parameter of this inference is back propagated to the convolution layer module by the output quantization module for convolution calculation, to obtain a convolution calculation result; The convolution calculation result is transmitted to the weight input module and the floating point input module respectively by the convolution layer module, and is updated according to the weight update strategy, to obtain updated floating point weight, and the updated floating point weight is used as the floating point weight of the next inference, and acts on the floating point output module in the next inference; The model parameters of the initial quantization network include the model weight of each inference and the first loss parameter of each inference under the weight update strategy, and the model weight includes the floating point weight and / or the integer type weight corresponding to the floating point weight.

[0008] As an optional implementation, in the second aspect of the present application, the device further comprises: The judging unit is configured to judge whether the current condition of the initial quantization network meets a second preset convergence condition according to the determined network inference result of the initial quantization network under the weight update strategy, and when the judging unit judges that the current condition of the initial quantization network meets the second preset convergence condition, trigger the hysteresis updating unit to execute the operation of fixing the quantization parameter of the initial quantization network as a target quantization parameter, and adding a hysteresis strategy in the weight update strategy to obtain a weight hysteresis update strategy, and trigger the QAT training unit to execute the operation of performing QAT training on the initial quantization network according to the weight hysteresis update strategy until the initial quantization network meets the second preset convergence condition; The QAT training unit is further configured to re-perform the operation of performing QAT training on the initial quantization network according to the preset weight update strategy when the judgment unit determines that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0009] As an optional implementation form, in the second aspect, the manner in which the judgment unit determines whether the current condition of the initial quantization network satisfies the second preset convergence condition according to the determined network inference result of the initial quantization network under the weight update strategy specifically includes: determining, according to the first loss parameters contained in the network inference result and corresponding to multiple inferences under the weight update strategy, whether all the first loss parameters within a preset continuous inference number range are within a preset loss value range, denoted as a first sub-convergence indicator condition; determining a weight update curve corresponding to the weight update strategy according to the model weights contained in the network inference result and corresponding to multiple inferences under the weight update strategy, and dividing the weight update curve according to a preset target inference number range to determine a model weight change phenomenon corresponding to each of the target inference number ranges in the weight update curve, and determining whether there is a preset weight high-frequency vibration phenomenon in the weight change phenomena corresponding to all the target inference number ranges in the weight update curve, denoted as a second sub-convergence indicator condition; wherein, when the first sub-convergence indicator condition and the second sub-convergence indicator condition are both satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; and when the first sub-convergence indicator condition and / or the second sub-convergence indicator condition is not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0010] As an optional implementation form, in the second aspect, the manner in which the QAT training unit performs QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain a target quantization network that satisfies a third preset convergence condition specifically includes: performing QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, the weight quantization related parameters including pre-weight quantization parameters and post-weight quantization parameters; determining whether the target quantization parameters of the initial quantization network satisfy a preset update condition according to all the weight quantization related parameters, and re-performing the operation of performing QAT training on the initial quantization network according to the preset weight update strategy when it is determined that the target quantization parameters of the initial quantization network satisfy the update condition; determining a second loss parameter of the initial quantization network under the weight hysteresis update strategy, and determining whether the initial quantization network satisfies a third preset convergence condition according to the second loss parameter and the determined validation set result of the initial quantization network when it is determined that the target quantization parameter of the initial quantization network does not satisfy the update condition; re-executing the QAT training of the initial quantization network according to the weight hysteresis update strategy to obtain a plurality of weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and re-executing the operation of determining whether the target quantization parameter of the initial quantization network satisfies the preset update condition according to all the weight quantization related parameters when it is determined that the initial quantization network does not satisfy the third preset convergence condition; determining the initial quantization network satisfying the third preset convergence condition as the target quantization network when it is determined that the initial quantization network satisfies the third preset convergence condition.

[0011] As an optional implementation, in the second aspect of the present application, the manner in which the QAT training unit determines whether the target quantization parameter of the initial quantization network satisfies the preset update condition according to all the weight quantization related parameters specifically comprises: calculating a rounding error value of each weight pre-quantization parameter introducing a corresponding quantization operation according to the weight pre-quantization parameter and the weight post-quantization parameter included in each weight quantization related parameter; determining a target parameter quantity of the target weight pre-quantization parameter whose rounding error value is a preset error threshold according to the rounding error values of all the weight pre-quantization parameters, and calculating a ratio of the target weight pre-quantization parameter in all the weight pre-quantization parameters according to the target parameter quantity and a total parameter quantity of all the weight pre-quantization parameters; determining whether the ratio of the target weight pre-quantization parameter in all the weight pre-quantization parameters is greater than a preset ratio; determining that the target quantization parameter of the initial quantization network satisfies the preset update condition when it is determined that the ratio of the target weight pre-quantization parameter in all the weight pre-quantization parameters is greater than the preset ratio; determining that the target quantization parameter of the initial quantization network does not satisfy the preset update condition when it is determined that the ratio of the target weight pre-quantization parameter in all the weight pre-quantization parameters is less than or equal to the preset ratio.

[0012] As an optional implementation, in the second aspect of the present application, the manner in which the QAT training unit determines whether the initial quantization network satisfies the third preset convergence condition according to the second loss parameter and the determined validation set result of the initial quantization network specifically comprises: determine whether the second loss parameter converges according to the second loss parameter, and determine whether the validation set result of the initial quantization network meets a preset network precision requirement according to the determined validation set result; When it is determined that the second loss parameter converges and the validation set result meets the network precision requirement, it is determined that the initial quantization network meets a third preset convergence condition; When it is determined that the second loss parameter does not converge and / or the validation set result does not meet the network precision requirement, it is determined that the initial quantization network does not meet the third preset convergence condition.

[0013] A third aspect of the present application discloses another device for hysteresis quantization-aware training of a neural network, the device comprising: a memory storing executable program codes; a processor coupled to the memory; The processor invokes the executable program codes stored in the memory to execute the method for hysteresis quantization-aware training of a neural network disclosed in the first aspect of the present application.

[0014] A fourth aspect of the present application discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the method for hysteresis quantization-aware training of a neural network disclosed in the first aspect of the present application.

[0015] Compared with the prior art, the embodiments of the present application have the following beneficial effects: In this embodiment of the invention, a pre-constructed neural network is trained to obtain a floating-point network that meets a first preset convergence condition. A pseudo-quantization node is inserted into the floating-point network, and pre-acquired training data is input into the floating-point network with the inserted pseudo-quantization node for PTQ calibration to obtain an initial quantization network. The initial quantization network is then trained using a preset weight update strategy, whereby the weight update strategy updates the weight parameters of the initial quantization network during the QAT training process and solves for the quantization parameters based on the weight parameters. If the model parameters of the determined initial quantization network meet a second preset convergence condition during the QAT training process, the quantization parameters of the initial quantization network are fixed as target quantization parameters, and a hysteresis strategy is added to the weight update strategy to obtain a weight hysteresis update strategy. The initial quantization network is then trained using the weight hysteresis update strategy to obtain a target quantization network that meets a third preset convergence condition. As can be seen, implementing this invention can train a pre-constructed neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes and feeding in training data, PTQ calibration is completed, followed by QAT training according to the weight update strategy. This combined training method of PTQ and QAT can improve the accuracy and precision of network training. Furthermore, if QAT training converges, the quantization parameters are fixed and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. This allows for phased QAT training using different quantization parameters, reducing the instability caused by changes in quantization parameters. Moreover, by adding a hysteresis strategy during weight parameter updates, the stability after weight updates can be improved, which is beneficial to improving the stability and reliability of network training. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for training a neural network with hysteresis quantization perception, as disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of a floating-point quantization loop based on QAT training disclosed in an embodiment of the present invention; Figure 3 This is a flowchart illustrating another method for training a neural network using hysteresis quantization perception, as disclosed in an embodiment of the present invention. Figure 4is a schematic diagram of a weight update curve disclosed by an embodiment of the present application; Figure 5 is a schematic diagram of a weight quantization process rounding method disclosed by an embodiment of the present application; Figure 6 is a histogram of the parameter quantity of the parameter before weight quantization in different rounding error value cases disclosed by an embodiment of the present application; Figure 7 is a flowchart of another hysteresis quantization-aware training method of a neural network disclosed by an embodiment of the present application; Figure 8 is a structural schematic diagram of a hysteresis quantization-aware training device of a neural network disclosed by an embodiment of the present application; Figure 9 is a structural schematic diagram of another hysteresis quantization-aware training device of a neural network disclosed by an embodiment of the present application; Figure 10 is a structural schematic diagram of another hysteresis quantization-aware training device of a neural network disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor fall within the scope of protection of the present application.

[0019] The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or end.

[0020] In this document, the reference to "embodiments" means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment is referred to, nor does it mean that the embodiments are mutually exclusive or alternative to each other. The person skilled in the art explicitly and implicitly understands that the embodiments described herein can be combined with other embodiments.

[0021] The application discloses a hysteresis quantization perception training method and device of a neural network, a storage medium, and can train a pre-constructed neural network to obtain a floating-point network trained to convergence, and can complete PTQ calibration by inserting a pseudo-quantization node and filling in training data, and then can perform QAT training according to a weight update strategy, can improve the accuracy and network precision of network training through the training mode combining PTQ and QAT, and if the QAT training is to convergence, the quantization parameters are fixed and a hysteresis strategy is added in the weight update strategy to perform QAT training according to a new weight hysteresis update strategy until convergence, can realize QAT training in stages through different quantization parameters, reduces the instability of network training caused by changes in quantization parameters, and through adding the hysteresis strategy in weight parameter updating, can improve the stability after weight updating, and is beneficial to improving the network training stability and reliability. The following is described in detail.

[0022] Embodiment one Please refer to Figure 1 , Figure 1 is a flowchart of a hysteresis quantization perception training method of a neural network disclosed by the embodiment of the application. Wherein, Figure 1 The hysteresis quantization perception training method of the neural network described can be applied to a hysteresis quantization perception training device of a neural network, wherein the device can include a training device or a training server, wherein the training server can include a cloud server or a local server, and the embodiment of the application does not limit. As shown in Figure 1 The hysteresis quantization perception training method of the neural network can include the following operations: 101, performing a training operation on a pre-constructed neural network to obtain a floating-point network trained to meet a first preset convergence condition.

[0023] In the embodiment of the application, the first preset convergence condition is used to represent that the weights in the neural network are stably converged to a pre-weight range, the loss parameter in the training process is continuously reduced to a preset loss range, and the neural network meets the accuracy requirement on the validation set.

[0024] 102, inserting a pseudo-quantization node in the floating-point network, and inputting pre-acquired training data into the floating-point network with the inserted pseudo-quantization node to perform PTQ calibration to obtain an initial quantization network.

[0025] 103, performing QAT training on the initial quantization network according to a preset weight update strategy.

[0026] In the embodiment of the present application, the weight updating strategy is used to update the weight parameters of the initial quantization network in the process of QAT training, and the quantization parameters are solved according to the weight parameters. The quantization parameters are used to map the floating-point value range to the discrete integer range, and the quantization parameters can include scale and offset. The scale represents the quantization scale / factor, which is used to determine the proportional relationship between the floating-point value and the integer value, and the offset represents the quantization offset / zero point, which is used to specify the integer value corresponding to the floating-point zero point. Specifically, the QAT training is started, the model weight and activation value range in the training process are counted, and the reasonable scale and offset are solved.

[0027] 104. If it is determined that the model parameters of the initial quantization network in the process of QAT training satisfy the second preset convergence condition, the quantization parameters of the initial quantization network are fixed as the target quantization parameters, and a hysteresis strategy is added to the weight updating strategy to obtain a weight hysteresis updating strategy.

[0028] In the embodiment of the present application, optionally, the model parameters of the initial quantization network include the model weight for each inference under the weight updating strategy and the first loss parameter for each inference, and the model weight includes the floating-point weight and / or the integer weight corresponding to the floating-point weight, which is not limited in the embodiment of the present application.

[0029] In the embodiment of the present application, the second preset convergence condition is used to represent the condition that the loss parameter in the process of QAT training is reduced to a fixed level and the weight appears a preset high-frequency vibration phenomenon.

[0030] In the embodiment of the present application, the hysteresis strategy is used to represent that the hysteresis damping is increased when the weight parameters are updated, so as to improve the stability of the updated weight.

[0031] 105. According to the weight hysteresis updating strategy, the initial quantization network is trained by QAT to obtain a target quantization network trained to satisfy a third preset convergence condition.

[0032] In the embodiment of the present application, the third preset convergence condition is used to represent the condition that the loss parameter converges in the training process and the verification set result meets the preset accuracy requirement.

[0033] In the embodiment of the present application, optionally, the hysteresis quantization perception training method of the neural network can be used to train an image processing model (such as an AIISP model), can be used to train a speech recognition model, can be used to train a natural language processing model, and can be used to train any other model capable of quantization processing, which is not limited in the embodiment of the present application.

[0034] For example, for an AIISP (artificial intelligence image signal processing) model, the model can be trained based on the hysteresis quantization perception training method of the neural network. Figure 1The described training process trains the pre-constructed neural network through a plurality of original image signals and target image signals obtained by processing each original image signal, thereby obtaining an AIISP model. The target image signal obtained by processing each original image signal can be obtained by performing at least one of white balance, exposure control, color correction, sharpening, and noise reduction on the original image signal. Specifically, the original image signal is taken as the input of the neural network, and the target image signal is taken as the output of the neural network. The neural network is trained through the input and output, and the trained neural network is taken as the AIISP model. Figure 1 The described training process trains to obtain an AIISP model, which optimizes the traditional ISP (image signal processing) process through the neural network training method, and can realize white balance, exposure control, color correction, sharpening, and noise reduction, thereby significantly improving the image clarity and restoration.

[0035] It can be seen that the implementation Figure 1 The described hysteresis quantization perception training method of the neural network can train the pre-constructed neural network to obtain a floating-point network trained to converge, and can complete PTQ calibration by inserting a pseudo-quantization node and filling training data. Subsequently, the QAT training is performed according to the weight update strategy. The training accuracy and network precision can be improved through the combination of PTQ and QAT training methods. If the QAT training converges, the quantization parameters are fixed, and the hysteresis strategy is added in the weight update strategy. The QAT training is performed according to the new weight hysteresis update strategy until convergence. The QAT training can be performed in stages through different quantization parameters, which can reduce the instability of network training caused by changes in quantization parameters. The stability of the weight update can be improved by adding the hysteresis strategy in the weight parameter update, which is conducive to improving the network training stability and reliability.

[0036] In an optional embodiment, the model parameters of the initial quantization network in the process of QAT training are determined in the following manner: In the process of QAT training of the initial quantization network, the floating-point weight of the inference this time in the weight input module included in the initial quantization network is input to the weight quantization module included in the initial quantization network for quantization processing to obtain the integer weight corresponding to the floating-point weight; and the floating-point input data received by the floating-point input module included in the initial quantization network is input to the input quantization module included in the initial quantization network for quantization processing to obtain the integer input data corresponding to the floating-point input data; The integer weight and the integer input data are input to the convolution operation of the convolution layer module of the initial quantization network, and the output is output to the floating-point output module of the initial quantization network to obtain floating-point output data; The floating-point output data is input to the output quantization module of the initial quantization network for quantization processing to obtain the integer output data corresponding to the floating-point output data; The first loss parameter of the initial quantization network for this inference is determined, and the first loss parameter of this inference is backpropagated by the output quantization module to the convolutional layer module for convolution calculation to obtain the convolution calculation result; The convolutional layer module passes the convolution calculation results to the weight input module and the floating-point input module respectively. The weights are updated according to the weight update strategy to obtain the updated floating-point weights. The updated floating-point weights are used as the floating-point weights for the next inference and are applied to the floating-point output module during the next inference.

[0037] In this embodiment of the invention, the floating-point input data can be input data from a pre-input training set, and the training set can also include the real floating-point output data corresponding to each floating-point input data. The first loss parameter for this inference can be calculated by the real floating-point output data and the floating-point output data for this inference.

[0038] For example, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating a floating-point quantization loop based on QAT training, as disclosed in an embodiment of the present invention. Figure 2 As shown, the floating-point weight Wf and the floating-point input data If are each scaled and quantized to obtain integer weight Wq and integer input data Iq, respectively. The integer weight Wq and integer input data Iq are then convolved in the convolutional layer module conv (during which the quantized integer data is converted back to floating-point data to complete the convolution calculation), resulting in floating-point output data Of. The floating-point output data Of is then quantized to obtain integer output data Oq. Subsequently, the loss parameter loss is backpropagated. Specifically, the output quantization module corresponding to the integer output data Oq backpropagates the loss parameter loss to the convolutional layer module conv for convolution calculation. The convolution calculation result is then passed to the weight input module corresponding to the floating-point weight Wf and the floating-point input module corresponding to the floating-point input data If. After updating the floating-point weight Wf based on the convolution calculation result, the updated floating-point weight Wf is applied to the floating-point output data Of in the next inference iteration to reduce the loss parameter loss in the next inference iteration.

[0039] Specifically, the quantization formula for converting floating-point weights Wf into integer weights Wq can be Wq=clip(round(Wf / scale),qmin,qmax), where Wf represents the floating-point ground truth during network training and Wq is the quantized value.

[0040] The scale is obtained statistically using Wf, most commonly through the minmax method. Changes in scale lead to changes in Wq, causing network instability. Therefore, this invention divides network training into two states: scale transient and scale steady state. The scale transient corresponds to the QAT training phase of the aforementioned weight update strategy. Its main task is to statistically determine a reasonable and stable scale to fix the scale and provide a stable network state for entering the scale steady state. The scale steady state corresponds to the QAT training phase of the aforementioned weight hysteresis update strategy. Its main task is to train floating-point weights Wf to obtain floating-point weights Wf with small quantization loss and low fluctuation.

[0041] As can be seen, this optional embodiment can improve the accuracy and reliability of quantizing floating-point data into integer weights and integer input data during QAT training of the initial quantization network. The initial quantization network quantizes floating-point weights and floating-point input data into integer weights and integer input data respectively, performs convolution operations to obtain floating-point output data, and then quantizes the floating-point output data into integer output data. Subsequently, the loss parameters for this inference are determined and backpropagated for convolution calculation. Based on the convolution calculation results, the floating-point weights are updated according to the weight update strategy, improving the accuracy and reliability of floating-point weight updates. Furthermore, the updated floating-point weights are applied to the next inference, which helps reduce the loss parameters for the next inference, thereby improving the accuracy and reliability of network training.

[0042] Example 2 Please see Figure 3 , Figure 3 This is a flowchart illustrating a hysteresis quantization perception training method for a neural network disclosed in an embodiment of the present invention. Figure 3 The described hysteresis quantization sensing training method for neural networks can be applied to a hysteresis quantization sensing training device for neural networks. This device may include a training equipment or a training server, where the training server may include a cloud server or a local server; this embodiment of the invention does not impose any limitations. Figure 3 As shown, the hysteresis quantization perception training method for this neural network can include the following operations: 201. Perform training operations on the pre-constructed neural network to obtain a floating-point network trained to meet the first preset convergence condition.

[0043] 202. Insert pseudo-quantization nodes into the floating-point network, and input the pre-acquired training data into the floating-point network with inserted pseudo-quantization nodes for PTQ calibration to obtain the initial quantization network.

[0044] 203. Perform QAT training on the initial quantized network according to the preset weight update strategy.

[0045] 204. Based on the determined network inference results of the initial quantization network under the weight update strategy, determine whether the current conditions of the initial quantization network satisfy the second preset convergence condition.

[0046] In this embodiment of the invention, the network inference result may include: a second loss parameter for multiple inferences of the initial quantized network under the weight update strategy, and model weights for multiple inferences of the initial quantized network under the weight update strategy.

[0047] In this embodiment of the invention, when the judgment result of step 204 is yes, that is, when it is determined that the current condition of the initial quantization network meets the second preset convergence condition, step 205 is triggered; when the judgment result of step 204 is no, that is, when it is determined that the current condition of the initial quantization network does not meet the second preset convergence condition, the operation of QAT training of the initial quantization network according to the preset weight update strategy in step 203 is re-executed.

[0048] 205. Fix the initial quantization parameters of the quantization network as the target quantization parameters, and add a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy.

[0049] 206. Following the weight hysteresis update strategy, QAT training is performed on the initial quantization network to obtain the target quantization network trained to meet the third preset convergence condition.

[0050] For further descriptions of steps 201-203, 205, and 206, please refer to the detailed description of steps 101-105 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0051] It is evident that implementation Figure 3The described hysteresis quantization-aware training method for neural networks can train a pre-built neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes, feeding in training data, and completing PTQ calibration, QAT training is then performed according to the weight update strategy. This combined PTQ and QAT training approach improves the accuracy and precision of network training. If QAT training converges, the quantization parameters are fixed, and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. This allows for phased QAT training using different quantization parameters, reducing the instability caused by quantization parameter changes. Furthermore, by adding a hysteresis strategy during weight parameter updates, the stability after weight updates is improved, which is beneficial for enhancing the stability and reliability of network training. Furthermore, it can quickly and accurately determine whether the current conditions of the initial quantized network have converged based on the network inference results of the initial quantized network under the determined weight update strategy. If it does not converge, it continues to perform QAT training according to the original weight update strategy. If it converges, it performs QAT training according to the new weight hysteresis update strategy. This is beneficial for flexibly choosing to perform QAT training according to the original weight update strategy or the new weight hysteresis update strategy based on accurate judgment results.

[0052] In an optional embodiment, step 204 above, which determines whether the current conditions of the initial quantization network satisfy the second preset convergence condition based on the determined network inference results for the initial quantization network under the weight update strategy, may include: Based on the first loss parameter for multiple inferences under the weight update strategy included in the network inference results, determine whether all first loss parameters within the preset number of continuous inferences are within the preset loss value range, and record it as the first sub-convergence index condition. Based on the model weights for multiple inferences under the weight update strategy contained in the network inference results, the weight update curve corresponding to the weight update strategy is determined; and the weight update curve is divided according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and it is determined whether the weight change phenomenon corresponding to all target inference number ranges in the weight update curve has the preset high-frequency weight oscillation phenomenon, which is denoted as the second sub-convergence index condition. Specifically, when both the first sub-convergence index condition and the second sub-convergence index condition are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; when the first sub-convergence index condition and / or the second sub-convergence index condition are not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0053] Specifically, when it is determined that all first loss parameters within a preset number of continuous inferences are within a preset loss value range, and there is a high-frequency oscillation phenomenon in the weight update curve, the current condition of the initial quantization network is determined to satisfy the second preset convergence condition; when it is determined that at least one first loss parameter within a preset number of continuous inferences is not within a preset loss value range, and / or there is no high-frequency oscillation phenomenon in the weight update curve, the current condition of the initial quantization network is determined not to satisfy the second preset convergence condition.

[0054] In this embodiment of the invention, specifically, multiple target model weights are extracted from the model weights of multiple inferences included in the network inference results according to a preset extraction method (such as random extraction), and a weight update curve is determined based on all target model weights. When high-frequency oscillations in the weights are observed, such as more than 6 repetitive cyclic updates within a target inference number range (e.g., 20 iterations), ... Figure 4 The blue curve shown indicates that the network has reached convergence. Figure 4 This is a schematic diagram of a weight update curve disclosed in an embodiment of the present invention.

[0055] As can be seen, this optional embodiment can determine whether all second loss parameters within a preset number of continuous inferences are within a preset loss value range based on the second loss parameters of multiple inferences included in the network inference results, denoted as the first sub-convergence index condition; and determine the weight update curve corresponding to the weight update strategy based on the model weights of the sub-inferences included in the network inference results; and divide the weight update curve according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and determine whether there is a preset high-frequency weight oscillation phenomenon in the weight change phenomenon corresponding to all target inference number ranges in the weight update curve, denoted as the second sub-convergence index condition. When all sub-convergence index conditions are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition, which improves the accuracy and reliability of determining that the current condition of the initial quantization network satisfies the second preset convergence condition. When at least one sub-convergence index condition is not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition, which improves the efficiency and speed of determining that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0056] In another optional embodiment, step 206 above, which involves performing QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain a target quantization network trained to meet a third preset convergence condition, includes: According to the weight hysteresis update strategy, QAT training is performed on the initial quantization network to obtain multiple weight quantization related parameters for the initial quantization network under the weight hysteresis update strategy. The weight quantization related parameters include the parameters before weight quantization and the parameters after weight quantization. Based on all weighting-related parameters, determine whether the target quantization parameters of the initial quantization network meet the preset update conditions; and when it is determined that the target quantization parameters of the initial quantization network meet the update conditions, re-execute the operation of QAT training of the initial quantization network according to the preset weight update strategy in step 203. When it is determined that the target quantization parameters of the initial quantization network do not meet the update conditions, the second loss parameter of the initial quantization network under the weight hysteresis update strategy is determined, and based on the second loss parameter and the validation set results of the determined initial quantization network, it is determined whether the initial quantization network meets the third preset convergence condition. When it is determined that the initial quantization network does not meet the third preset convergence condition, the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy in step 206 is re-executed to obtain the operation of multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and the operation of determining whether the target quantization parameter of the initial quantization network meets the preset update condition based on all weight quantization related parameters in step 206 is re-executed. When it is determined that the initial quantization network satisfies the third preset convergence condition, the initial quantization network that satisfies the third preset convergence condition is determined as the target quantization network.

[0057] In this embodiment of the invention, the rounding method for the weight quantization process corresponding to the weight update strategy uses the round function. The round function rounds numbers according to a preset value (e.g., 0.5). For example, numbers less than 0.5 are rounded to zero, and numbers greater than or equal to 0.5 are rounded to 1. The rounding method for the weight hysteresis update strategy corresponding to the weight quantization process uses the hesitation-round function. The hesitation-round function rounds numbers according to a preset range (e.g., 0.3-0.8) containing the preset value. For example, numbers less than 0.3 are rounded to zero, numbers greater than 0.8 are rounded to 1, all numbers within the range less than 0.3 gradually increasing to 0.8 are rounded to zero, and all numbers within the range greater than 0.8 gradually decreasing to 0.3 are rounded to 1. Figure 5 As shown, Figure 5 This is a schematic diagram of a rounding method for a weight quantization process disclosed in an embodiment of the present invention, wherein... Figure 5 (a) indicates the rounding method used in the weight quantization process corresponding to the weight update strategy. Figure 5(b) indicates the rounding method of the weight quantization process corresponding to the weight hysteresis update strategy. By using the rounding method of the weight quantization process corresponding to the weight hysteresis update strategy, it is beneficial to reduce the occurrence of weight update overshoot or invalid update due to the non-continuous Round function during the quantization process, which may cause the weight update to fail to reach the ideal value.

[0058] As can be seen, this optional embodiment can perform QAT training on the initial quantized network according to the weight hysteresis update strategy to obtain multiple weight quantization-related parameters of the initial quantized network under the weight hysteresis update strategy. This allows it to determine whether the target quantization parameters of the initial quantized network meet the update conditions. If the update conditions are met, QAT training is performed again according to the weight update strategy. This can effectively avoid frequent weight jumps caused by quantization after the quantization parameters have stabilized. By setting the update conditions that trigger the quantization parameters, the accuracy and timeliness of updating the target quantization parameters can be improved, reducing the occurrence of network oscillations caused by errors due to frequent updates of the target quantization parameters. If the update conditions are not met, it is further determined whether the network has converged. If it has not converged, QAT training continues according to the weight hysteresis update strategy. If it converges, the training is complete. This not only improves the accuracy of the target quantization parameters but also improves the accuracy of the network training convergence determination, which is beneficial to improving the network accuracy, training accuracy, stability, and reliability of the final trained target quantized network.

[0059] In this optional embodiment, as an optional implementation method, determining whether the target quantization parameters of the initial quantization network meet the preset update conditions based on all quantization-related parameters may include: Based on the parameters related to each weight quantization, including the parameters before and after weight quantization, calculate the rounding error value introduced by the corresponding quantization operation before each weight quantization. Based on the rounding error values ​​of all pre-quantization parameters, determine the target parameter quantity of the target pre-quantization parameter with a rounding error value of a preset error threshold, and calculate the ratio of the target pre-quantization parameter to all pre-quantization parameters based on the target parameter quantity and the total parameter quantity of all pre-quantization parameters. Determine whether the ratio of the target weight's unquantized parameters to all unquantized weight parameters is greater than a preset ratio; When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is greater than a preset ratio, the target quantization parameters of the initial quantization network are determined to meet the preset update conditions. When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is less than or equal to a preset ratio, it is determined that the target quantization parameters of the initial quantization network do not meet the preset update conditions.

[0060] For example, by plotting a histogram of the parameters before weight quantization (e.g., weights and activation values ​​before quantization), the number of parameters before weight quantization under different rounding error values ​​can be statistically analyzed. Figure 6 As shown, Figure 6 This is a histogram of the parameter quantities of the weighted parameters before quantization under different rounding error values, as disclosed in an embodiment of the present invention. Figure 6 The histogram shown uses different rounding error values ​​on the horizontal axis and the number of parameters of the weighted pre-quantization parameters under different rounding error values ​​on the vertical axis. Based on this, it is determined whether the ratio of the target weighted pre-quantization parameters among all weighted pre-quantization parameters is greater than a preset ratio (e.g., whether the ratio of elements appearing in the first and last bins of the histogram is greater than 10%). If it is greater than or equal to the preset ratio, it is determined that the target quantization parameters of the initial quantization network meet the preset update conditions, and then the quantization parameter update operation is triggered. If it is less than the preset ratio, it is determined that the target quantization parameters of the initial quantization network do not meet the preset update conditions.

[0061] As can be seen, this optional implementation can calculate the rounding error value of the corresponding quantization operation introduced before each weight quantization based on the parameters before and after weight quantization, which are included in the relevant parameters of each weight quantization. This accurately determines the target parameter quantity of the target weight before quantization, with the rounding error value being a preset error threshold. Based on the target parameter quantity and the total parameter quantity of all weight before quantization parameters, the ratio of the target weight before quantization parameter to all weight before quantization parameters is calculated. This determines whether the ratio is greater than a preset ratio. If it is determined to be greater than the preset ratio, it is determined that the target quantization parameter of the initial quantization network meets the preset update condition, improving the accuracy and efficiency of determining whether the target quantization parameter of the initial quantization network meets the preset update condition, thereby improving the triggering accuracy and timeliness of updating the target quantization parameter. If it is determined to be less than or equal to the preset ratio, it is determined that the target quantization parameter of the initial quantization network does not meet the preset update condition, improving the accuracy and reliability of determining whether the target quantization parameter of the initial quantization network does not meet the preset update condition. This allows for accurate subsequent judgment of whether the network has been trained to convergence even when the target quantization parameter does not need to be updated, which is beneficial to improving the accuracy and reliability of network training to a certain extent.

[0062] In this optional embodiment, as another optional implementation, determining whether the initial quantization network satisfies the third preset convergence condition based on the second loss parameter and the validation set results of the determined initial quantization network may include: Based on the second loss parameter, determine whether the second loss parameter has converged, and based on the validation set results of the determined initial quantized network, determine whether the validation set results meet the preset network accuracy requirements. When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, the initial quantization network is determined to satisfy the third preset convergence condition. When it is determined that the second loss parameter has not converged, and / or the validation set results do not meet the network accuracy requirements, it is determined that the initial quantization network does not meet the third preset convergence condition.

[0063] In this embodiment of the invention, specifically, after the initial quantization network is trained using QAT according to the weight hysteresis update strategy, the initial quantization network is validated based on the pre-obtained validation set to obtain the validation set result of the initial quantization network.

[0064] As can be seen, this optional implementation can, when it is determined that the target quantization parameters of the initial quantization network do not need to be updated, further determine whether the second loss parameter of the initial quantization network under the weight hysteresis update strategy has converged, and determine whether the validation set results of the determined initial quantization network meet the preset network accuracy requirements. When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, it is determined that the initial quantization network meets the third preset convergence condition, which improves the accuracy and reliability of determining that the initial quantization network meets the third preset convergence condition. Thus, training is stopped and the initial quantization network is determined as the target quantization network after training, which is beneficial to improving the accuracy and reliability of network training. When it is determined that the second loss parameter has not converged and / or the validation set results do not meet the network accuracy requirements, it is directly determined that the initial quantization network does not meet the third preset convergence condition, which improves the efficiency and speed of determining that the initial quantization network does not meet the third preset convergence condition. Thus, QAT training continues according to the weight hysteresis update strategy, which is beneficial to improving the training accuracy and reliability of the neural network through multiple conditions.

[0065] For example, such as Figure 7 As shown, Figure 7 This is a flowchart illustrating another method for training a neural network using hysteresis quantization perception, as disclosed in an embodiment of the present invention. This method may include the following steps: S1. Training a floating-point network until the network converges is characterized by the following: the weights in the network converge stably to a certain range, the training loss continuously decreases to a reasonable range, and the network meets the accuracy requirements on the validation set.

[0066] S2. Insert pseudo-quantization nodes into the model, feed in training data, and complete PTQ.

[0067] S3. Start QAT training, count the model weights and activation value ranges during training, and solve for a reasonable scale and offset. The scale solved at this time is recorded as scale0.

[0068] S4. Determine if QAT training has converged. The convergence criteria are: a. The training loss decreases to a fixed level without significant, clearly directional updates. b. Randomly sample multiple (5-10) weights from the model and plot their update curves. It is observed that the weights exhibit high-frequency oscillations, with more than 6 repetitive cyclical updates in 20 iterations. Figure 4 The solid blue line indicates that the network has entered convergence. If the convergence is successful, the process proceeds to S5; otherwise, QAT training continues in S3.

[0069] S5. Fix the quantization parameters and enter stage 1. At this point, the scale and offset are no longer updated based on the weights and activation statistics. Continue QAT training and perform hesitation-round on the weight update strategy according to the step and lr inheritance steps (i.e., the continuation of the standard gradient descent optimization process in QAT).

[0070] S6. During training, draw a histogram of weights and activation values ​​before quantization and determine whether the scale needs to be updated. The judgment condition is: the number of elements appearing in the first and last bins is greater than 10% of the total number, i.e., whether the scale adjustment operation is triggered. If yes, proceed to step S3; if no (i.e., no update is needed), proceed to S7.

[0071] S7. Determine if the training loss has converged and if the validation set results meet the requirements. If they do, stop training; otherwise, return to step S5 for QAT training. This method trains a pre-built neural network to convergence, inserts pseudo-quantization nodes, feeds in training data, completes PTQ calibration, and then performs QAT training according to the weight update strategy. This combined PTQ and QAT training approach improves network training accuracy and precision. If QAT training converges, the quantization parameters are fixed, and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new hysteresis strategy until convergence. This allows for phased QAT training using different quantization parameters, reducing instability caused by quantization parameter changes. Furthermore, adding a hysteresis strategy during weight parameter updates improves stability after weight updates, enhancing network training stability and reliability.

[0072] Example 3 Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention. Figure 8The described neural network hysteresis quantization perception training device may include a training device or a training server, wherein the training server may include a cloud server or a local server, and the embodiments of the present invention are not limited thereto. Figure 8 As shown, the hysteresis quantization sensing training device for this neural network may include: The floating-point training unit 301 is used to perform training operations on a pre-built neural network to obtain a floating-point network trained to meet a first preset convergence condition. The PTQ calibration unit 302 is used to insert pseudo-quantization nodes into the floating-point network and input the pre-acquired training data into the floating-point network with inserted pseudo-quantization nodes for PTQ calibration to obtain the initial quantization network. QAT training unit 303 is used to perform QAT training on the initial quantized network according to a preset weight update strategy. The weight update strategy is used to update the weight parameters of the initial quantized network during the QAT training process and solve the quantization parameters based on the weight parameters. The hysteresis update unit 304 is used to fix the quantization parameters of the initial quantization network as the target quantization parameters if the model parameters of the determined initial quantization network meet the second preset convergence condition during QAT training, and to add a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy. The QAT training unit 303 is also used to perform QAT training on the initial quantization network according to the weight hysteresis update strategy, so as to obtain the target quantization network trained to meet the third preset convergence condition.

[0073] It is evident that implementation Figure 8 The described neural network hysteresis quantization-aware training device can train a pre-built neural network to obtain a converged floating-point network. By inserting pseudo-quantization nodes, feeding in training data, and completing PTQ calibration, it then performs QAT training according to the weight update strategy. This combined PTQ and QAT training method can improve the accuracy and precision of network training. If QAT training converges, the quantization parameters are fixed and a hysteresis strategy is added to the weight update strategy to perform QAT training according to the new weight hysteresis update strategy until convergence. It can perform QAT training in stages using different quantization parameters, reducing the instability of network training caused by changes in quantization parameters. Furthermore, by adding a hysteresis strategy during weight parameter updates, the stability after weight updates can be improved, which is beneficial to improving the stability and reliability of network training.

[0074] In an optional embodiment, the model parameters of the initial quantized network during QAT training are determined in the following way: During QAT training of the initial quantization network, the floating-point weights of this inference in the weight input module of the initial quantization network are input to the weight quantization module of the initial quantization network for quantization processing to obtain the integer weights corresponding to the floating-point weights; and the floating-point input data received by the floating-point input module of the initial quantization network are input to the input quantization module of the initial quantization network for quantization processing to obtain the integer input data corresponding to the floating-point input data. The integer weights and integer input data are input into the convolutional layer module of the initial quantization network, and the output is sent to the floating-point output module of the initial quantization network to obtain floating-point output data. The floating-point output data is input into the output quantization module of the initial quantization network for quantization processing to obtain the integer output data corresponding to the floating-point output data. The first loss parameter of the initial quantization network for this inference is determined, and the first loss parameter of this inference is backpropagated by the output quantization module to the convolutional layer module for convolution calculation to obtain the convolution calculation result; The convolutional layer module passes the convolution calculation results to the weight input module and the floating-point input module respectively. The weights are updated according to the weight update strategy to obtain the updated floating-point weights. The updated floating-point weights are used as the floating-point weights for the next inference and are applied to the floating-point output module in the next inference. The model parameters of the initial quantization network include the model weights for each inference under the weight update strategy and the first loss parameter for each inference. The model weights include floating-point weights and / or integer weights corresponding to the floating-point weights.

[0075] As can be seen, this optional embodiment can improve the accuracy and reliability of quantizing floating-point data into integer weights and integer input data during QAT training of the initial quantization network. The initial quantization network quantizes floating-point weights and floating-point input data into integer weights and integer input data respectively, performs convolution operations to obtain floating-point output data, and then quantizes the floating-point output data into integer output data. Subsequently, the loss parameters for this inference are determined and backpropagated for convolution calculation. Based on the convolution calculation results, the floating-point weights are updated according to the weight update strategy, improving the accuracy and reliability of floating-point weight updates. Furthermore, the updated floating-point weights are applied to the next inference, which helps reduce the loss parameters for the next inference, thereby improving the accuracy and reliability of network training.

[0076] In this optional embodiment, as an optional implementation method, such as Figure 9 As shown, Figure 9 This is a schematic diagram of another neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention, wherein the device may further include: The judgment unit 305 is used to determine whether the current condition of the initial quantization network meets the second preset convergence condition based on the determined network inference result of the initial quantization network under the weight update strategy; and when the judgment unit 305 determines that the current condition of the initial quantization network meets the second preset convergence condition, it triggers the hysteresis update unit 304 to perform the operation of fixing the quantization parameters of the initial quantization network as the target quantization parameters and adding a hysteresis strategy to the weight update strategy to obtain the weight hysteresis update strategy, and triggers the QAT training unit 303 to perform the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy until the initial quantization network meets the second preset convergence condition. The QAT training unit 303 is also used to re-execute the operation of QAT training the initial quantization network according to the preset weight update strategy when the judgment unit 305 determines that the current conditions of the initial quantization network do not meet the second preset convergence condition.

[0077] As can be seen, this optional implementation can quickly and accurately determine whether the current conditions of the initial quantized network have converged based on the determined network inference results of the initial quantized network under the weight update strategy. If it does not converge, QAT training continues according to the original weight update strategy. If it converges, QAT training is performed according to the new weight hysteresis update strategy. This is beneficial for flexibly choosing to perform QAT training according to the original weight update strategy or the new weight hysteresis update strategy based on accurate judgment results.

[0078] In this optional implementation, the method by which the judging unit 305 judges whether the current conditions of the initial quantization network satisfy the second preset convergence condition based on the determined network inference results for the initial quantization network under the weight update strategy specifically includes: Based on the first loss parameter for multiple inferences under the weight update strategy included in the network inference results, determine whether all first loss parameters within the preset number of continuous inferences are within the preset loss value range, and record it as the first sub-convergence index condition. Based on the model weights for multiple inferences under the weight update strategy contained in the network inference results, the weight update curve corresponding to the weight update strategy is determined; and the weight update curve is divided according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and it is determined whether the weight change phenomenon corresponding to all target inference number ranges in the weight update curve has the preset high-frequency weight oscillation phenomenon, which is denoted as the second sub-convergence index condition. Specifically, when both the first sub-convergence index condition and the second sub-convergence index condition are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; when the first sub-convergence index condition and / or the second sub-convergence index condition are not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0079] As can be seen, this optional implementation can also determine whether all second loss parameters within a preset number of continuous inferences are within a preset loss value range based on the second loss parameters of multiple inferences included in the network inference results, and record this as the first sub-convergence index condition; and determine the weight update curve corresponding to the weight update strategy based on the model weights of the sub-inferences included in the network inference results; and divide the weight update curve according to the preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and determine whether there is a preset high-frequency weight oscillation phenomenon in the weight change phenomenon corresponding to all target inference number ranges in the weight update curve, and record this as the second sub-convergence index condition. When all sub-convergence index conditions are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition, which improves the accuracy and reliability of determining that the current condition of the initial quantization network satisfies the second preset convergence condition. When at least one sub-convergence index condition is not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition, which improves the efficiency and speed of determining that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

[0080] In another optional embodiment, the QAT training unit 303 performs QAT training on the initial quantization network according to the weight hysteresis update strategy to obtain the target quantization network that satisfies the third preset convergence condition. Specifically, this includes: According to the weight hysteresis update strategy, QAT training is performed on the initial quantization network to obtain multiple weight quantization related parameters for the initial quantization network under the weight hysteresis update strategy. The weight quantization related parameters include the parameters before weight quantization and the parameters after weight quantization. Based on all weighting-related parameters, determine whether the target quantization parameters of the initial quantization network meet the preset update conditions; and when it is determined that the target quantization parameters of the initial quantization network meet the update conditions, re-execute the operation of QAT training on the initial quantization network according to the preset weight update strategy. When it is determined that the target quantization parameters of the initial quantization network do not meet the update conditions, the second loss parameter of the initial quantization network under the weight hysteresis update strategy is determined, and based on the second loss parameter and the validation set results of the determined initial quantization network, it is determined whether the initial quantization network meets the third preset convergence condition. When it is determined that the initial quantization network does not meet the third preset convergence condition, the QAT training of the initial quantization network is re-executed according to the weight hysteresis update strategy to obtain the operation of multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and the operation of determining whether the target quantization parameter of the initial quantization network meets the preset update condition based on all weight quantization related parameters is re-executed. When it is determined that the initial quantization network satisfies the third preset convergence condition, the initial quantization network that satisfies the third preset convergence condition is determined as the target quantization network.

[0081] As can be seen, this optional embodiment can perform QAT training on the initial quantized network according to the weight hysteresis update strategy to obtain multiple weight quantization-related parameters of the initial quantized network under the weight hysteresis update strategy. This allows it to determine whether the target quantization parameters of the initial quantized network meet the update conditions. If the update conditions are met, QAT training is performed again according to the weight update strategy. This can effectively avoid frequent weight jumps caused by quantization after the quantization parameters have stabilized. By setting the update conditions that trigger the quantization parameters, the accuracy and timeliness of updating the target quantization parameters can be improved, reducing the occurrence of network oscillations caused by errors due to frequent updates of the target quantization parameters. If the update conditions are not met, it is further determined whether the network has converged. If it has not converged, QAT training continues according to the weight hysteresis update strategy. If it converges, the training is complete. This not only improves the accuracy of the target quantization parameters but also improves the accuracy of the network training convergence determination, which is beneficial to improving the network accuracy, training accuracy, stability, and reliability of the final trained target quantized network.

[0082] In this optional embodiment, as an optional implementation, the QAT training unit 303 determines whether the target quantization parameters of the initial quantization network meet the preset update conditions based on all quantization-related parameters in the following specific ways: Based on the parameters related to each weight quantization, including the parameters before and after weight quantization, calculate the rounding error value introduced by the corresponding quantization operation before each weight quantization. Based on the rounding error values ​​of all pre-quantization parameters, determine the target parameter quantity of the target pre-quantization parameter with a rounding error value of a preset error threshold, and calculate the ratio of the target pre-quantization parameter to all pre-quantization parameters based on the target parameter quantity and the total parameter quantity of all pre-quantization parameters. Determine whether the ratio of the target weight's unquantized parameters to all unquantized weight parameters is greater than a preset ratio; When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is greater than a preset ratio, the target quantization parameters of the initial quantization network are determined to meet the preset update conditions. When it is determined that the ratio of the target weight's unquantized parameters to the ratio of all unquantized parameters is less than or equal to a preset ratio, it is determined that the target quantization parameters of the initial quantization network do not meet the preset update conditions.

[0083] As can be seen, this optional implementation can calculate the rounding error value of the corresponding quantization operation introduced before each weight quantization based on the parameters before and after weight quantization, which are included in the relevant parameters of each weight quantization. This accurately determines the target parameter quantity of the target weight before quantization, with the rounding error value being a preset error threshold. Based on the target parameter quantity and the total parameter quantity of all weight before quantization parameters, the ratio of the target weight before quantization parameter to all weight before quantization parameters is calculated. This determines whether the ratio is greater than a preset ratio. If it is determined to be greater than the preset ratio, it is determined that the target quantization parameter of the initial quantization network meets the preset update condition, improving the accuracy and efficiency of determining whether the target quantization parameter of the initial quantization network meets the preset update condition, thereby improving the triggering accuracy and timeliness of updating the target quantization parameter. If it is determined to be less than or equal to the preset ratio, it is determined that the target quantization parameter of the initial quantization network does not meet the preset update condition, improving the accuracy and reliability of determining whether the target quantization parameter of the initial quantization network does not meet the preset update condition. This allows for accurate subsequent judgment of whether the network has been trained to convergence even when the target quantization parameter does not need to be updated, which is beneficial to improving the accuracy and reliability of network training to a certain extent.

[0084] In this optional embodiment, as another optional implementation, the QAT training unit 303 determines whether the initial quantization network satisfies the third preset convergence condition based on the second loss parameter and the determined validation set results of the initial quantization network. Specifically, this includes: Based on the second loss parameter, determine whether the second loss parameter has converged, and based on the validation set results of the determined initial quantized network, determine whether the validation set results meet the preset network accuracy requirements. When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, the initial quantization network is determined to satisfy the third preset convergence condition. When it is determined that the second loss parameter has not converged, and / or the validation set results do not meet the network accuracy requirements, it is determined that the initial quantization network does not meet the third preset convergence condition.

[0085] As can be seen, this optional implementation can, when it is determined that the target quantization parameters of the initial quantization network do not need to be updated, further determine whether the second loss parameter of the initial quantization network under the weight hysteresis update strategy has converged, and determine whether the validation set results of the determined initial quantization network meet the preset network accuracy requirements. When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, it is determined that the initial quantization network meets the third preset convergence condition, which improves the accuracy and reliability of determining that the initial quantization network meets the third preset convergence condition. Thus, training is stopped and the initial quantization network is determined as the target quantization network after training, which is beneficial to improving the accuracy and reliability of network training. When it is determined that the second loss parameter has not converged and / or the validation set results do not meet the network accuracy requirements, it is directly determined that the initial quantization network does not meet the third preset convergence condition, which improves the efficiency and speed of determining that the initial quantization network does not meet the third preset convergence condition. Thus, QAT training continues according to the weight hysteresis update strategy, which is beneficial to improving the training accuracy and reliability of the neural network through multiple conditions.

[0086] Example 4 Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of another neural network hysteresis quantization perception training device disclosed in an embodiment of the present invention. Figure 10 As shown, the hysteresis quantization sensing training device for this neural network may include: Memory 401 storing executable program code; Processor 402 coupled to memory 401; The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the hysteresis quantization perception training method for neural networks described in Embodiment 1 or Embodiment 2 of the present invention.

[0087] Example 5 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the hysteresis quantization perception training method for neural networks described in Embodiment 1 or Embodiment 2 of this invention.

[0088] Example 6 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the hysteresis quantization perception training method for a neural network described in Embodiment 1 or Embodiment 2.

[0089] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0090] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0091] Finally, it should be noted that the hysteresis quantization perception training method and apparatus for neural networks and the storage medium disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a neural network using hysteresis quantization perception, characterized in that, The method includes: The pre-built neural network is trained to obtain a floating-point network that satisfies the first preset convergence condition; A pseudo-quantization node is inserted into the floating-point network, and the pre-acquired training data is input into the floating-point network into which the pseudo-quantization node has been inserted for PTQ calibration to obtain the initial quantization network; According to a preset weight update strategy, the initial quantization network is trained using QAT. The weight update strategy is used to update the weight parameters of the initial quantization network during the QAT training process, and to solve for the quantization parameters based on the weight parameters. If the model parameters of the determined initial quantization network satisfy the second preset convergence condition during the QAT training process, then the quantization parameters of the initial quantization network are fixed as the target quantization parameters, and a hysteresis strategy is added to the weight update strategy to obtain a weight hysteresis update strategy. Then, the initial quantization network is trained using the weight hysteresis update strategy to obtain a target quantization network that satisfies the third preset convergence condition.

2. The method for training a neural network with hysteresis quantization perception according to claim 1, characterized in that, The model parameters of the initial quantized network during the QAT training process are determined in the following way: During QAT training of the initial quantization network, the floating-point weights of this inference in the weight input module of the initial quantization network are input to the weight quantization module of the initial quantization network for quantization processing to obtain the integer weights corresponding to the floating-point weights. In addition, the floating-point input data received by the floating-point input module included in the initial quantization network is input to the input quantization module included in the initial quantization network for quantization processing to obtain the integer input data corresponding to the floating-point input data; The integer weights and the integer input data are input into the convolution operation of the convolutional layer module of the initial quantization network, and the output is sent to the floating-point output module of the initial quantization network to obtain floating-point output data; The floating-point output data is input to the output quantization module of the initial quantization network for quantization processing to obtain the integer output data corresponding to the floating-point output data. The first loss parameter of the initial quantization network for this inference is determined, and the first loss parameter of this inference is backpropagated to the convolutional layer module by the output quantization module to perform convolution calculation to obtain the convolution calculation result; The convolutional layer module transmits the convolution calculation result to the weight input module and the floating-point input module respectively, and updates it according to the weight update strategy to obtain the updated floating-point weight. The updated floating-point weight is used as the floating-point weight for the next inference and is applied to the floating-point output module during the next inference. The model parameters of the initial quantization network include the model weights for each inference under the weight update strategy and the first loss parameter for each inference. The model weights include the floating-point weights and / or the integer weights corresponding to the floating-point weights.

3. The method for training a neural network with hysteresis quantization perception according to claim 2, characterized in that, The method further includes: Based on the determined network inference results of the initial quantization network under the weight update strategy, it is determined whether the current condition of the initial quantization network satisfies the second preset convergence condition; and when it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition, the operation of fixing the quantization parameters of the initial quantization network as the target quantization parameters and adding a hysteresis strategy to the weight update strategy to obtain a weight hysteresis update strategy is triggered, and the operation of performing QAT training on the initial quantization network according to the weight hysteresis update strategy is triggered until the initial quantization network satisfies the second preset convergence condition is executed. When it is determined that the current conditions of the initial quantization network do not meet the second preset convergence condition, the operation of QAT training of the initial quantization network according to the preset weight update strategy is re-executed.

4. The hysteresis quantization perception training method for neural networks according to claim 3, characterized in that, The step of determining whether the current conditions of the initial quantization network satisfy the second preset convergence condition based on the determined network inference results of the initial quantization network under the weight update strategy includes: Based on the first loss parameter included in the network inference result for multiple inferences under the weight update strategy, determine whether all first loss parameters within a preset number of continuous inferences are within a preset loss value range, and record this as the first sub-convergence index condition. Based on the model weights included in the network inference results for multiple inferences under the weight update strategy, the weight update curve corresponding to the weight update strategy is determined; and the weight update curve is divided according to a preset target inference number range to determine the model weight change phenomenon corresponding to each target inference number range in the weight update curve; and it is determined whether there is a preset high-frequency weight oscillation phenomenon in all the weight change phenomena corresponding to the target inference number range in the weight update curve, which is denoted as the second sub-convergence index condition. Specifically, when both the first sub-convergence index condition and the second sub-convergence index condition are satisfied, it is determined that the current condition of the initial quantization network satisfies the second preset convergence condition; when the first sub-convergence index condition and / or the second sub-convergence index condition are not satisfied, it is determined that the current condition of the initial quantization network does not satisfy the second preset convergence condition.

5. The method for training a neural network with hysteresis quantization perception according to any one of claims 1-4, characterized in that, The step of training the initial quantization network using QAT according to the weight hysteresis update strategy to obtain the target quantization network trained to meet the third preset convergence condition includes: According to the weight hysteresis update strategy, the initial quantization network is trained with QAT to obtain multiple weight quantization related parameters for the initial quantization network under the weight hysteresis update strategy. The weight quantization related parameters include parameters before weight quantization and parameters after weight quantization. Based on all the weight quantization-related parameters, determine whether the target quantization parameters of the initial quantization network meet the preset update conditions; and when it is determined that the target quantization parameters of the initial quantization network meet the update conditions, re-execute the operation of QAT training of the initial quantization network according to the preset weight update strategy. When it is determined that the target quantization parameters of the initial quantization network do not meet the update conditions, a second loss parameter for the initial quantization network under the weight hysteresis update strategy is determined, and based on the second loss parameter and the determined validation set results of the initial quantization network, it is determined whether the initial quantization network meets the third preset convergence condition. When it is determined that the initial quantization network does not meet the third preset convergence condition, the operation of QAT training on the initial quantization network according to the weight hysteresis update strategy is re-executed to obtain multiple weight quantization related parameters of the initial quantization network under the weight hysteresis update strategy, and the operation of determining whether the target quantization parameter of the initial quantization network meets the preset update condition based on all the weight quantization related parameters is re-executed. When it is determined that the initial quantization network satisfies the third preset convergence condition, the initial quantization network that satisfies the third preset convergence condition is determined as the target quantization network.

6. The method for training a neural network with hysteresis quantization perception according to claim 5, characterized in that, The step of determining whether the target quantization parameters of the initial quantization network meet the preset update conditions based on all the weight quantization-related parameters includes: Based on the parameters before and after weight quantization included in each of the weight quantization related parameters, calculate the rounding error value of the corresponding quantization operation introduced before each weight quantization. Based on the rounding error values ​​of all the pre-quantization weighted parameters, determine the target parameter quantity of the target pre-quantization weighted parameter whose rounding error value is a preset error threshold, and calculate the ratio of the target pre-quantization weighted parameter to all the pre-quantization weighted parameters based on the target parameter quantity and the total parameter quantity of all the pre-quantization weighted parameters. Determine whether the ratio of the target weight pre-quantization parameter among all the weight pre-quantization parameters is greater than a preset ratio; When it is determined that the ratio of the target weight pre-quantization parameter to all the weight pre-quantization parameters is greater than the preset ratio, it is determined that the target quantization parameter of the initial quantization network satisfies the preset update condition. When it is determined that the ratio of the target weight pre-quantization parameter to all the weight pre-quantization parameters is less than or equal to the preset ratio, it is determined that the target quantization parameter of the initial quantization network does not meet the preset update condition.

7. The method for training a neural network with hysteresis quantization perception according to claim 5, characterized in that, The step of determining whether the initial quantization network satisfies the third preset convergence condition based on the second loss parameter and the determined validation set results of the initial quantization network includes: Based on the second loss parameter, determine whether the second loss parameter has converged, and based on the validation set results of the determined initial quantization network, determine whether the validation set results meet the preset network accuracy requirements; When it is determined that the second loss parameter has converged and the validation set results meet the network accuracy requirements, it is determined that the initial quantization network satisfies the third preset convergence condition. When it is determined that the second loss parameter has not converged, and / or the validation set results do not meet the network accuracy requirements, it is determined that the initial quantization network does not meet the third preset convergence condition.

8. A hysteresis quantization sensing training device for a neural network, characterized in that, The device includes: The floating-point training unit is used to perform training operations on a pre-built neural network to obtain a floating-point network trained to meet a first preset convergence condition. The PTQ calibration unit is used to insert pseudo-quantization nodes into the floating-point network and input the pre-acquired training data into the floating-point network with the pseudo-quantization nodes inserted to perform PTQ calibration, thereby obtaining the initial quantization network. The QAT training unit is used to perform QAT training on the initial quantization network according to a preset weight update strategy. The weight update strategy is used to update the weight parameters of the initial quantization network during the QAT training process and solve for the quantization parameters based on the weight parameters. The hysteresis update unit is used to fix the quantization parameters of the initial quantization network as the target quantization parameters if the model parameters of the determined initial quantization network meet the second preset convergence condition during the QAT training process, and to add a hysteresis strategy to the weight update strategy to obtain a weight hysteresis update strategy. The QAT training unit is also used to perform QAT training on the initial quantization network according to the weight hysteresis update strategy, so as to obtain a target quantization network trained to meet the third preset convergence condition.

9. A hysteresis quantization sensing training device for a neural network, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the hysteresis quantization perception training method for the neural network as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the hysteresis quantization perception training method for the neural network as described in any one of claims 1-7.

Images