Training Method and Device of Neural Network Model, Electronic Device and Storage Medium

Abstract

The present disclosure relates to a training method and apparatus for a neural network model, an electronic device, and a storage medium. The method includes: updating a first network layer in a first neural network model to be trained to obtain a second network layer with a quantization simulation structure added, thereby obtaining a second neural network model, and training the second neural network model according to a second model optimizer to obtain a trained second neural network model; wherein the second model optimizer is determined based on the quantization simulation structure and an initial first model optimizer, and is used to adjust the update step size of the weight parameters of at least one second network layer. The quantization simulation structure is used to perform a simulated quantization operation on the weight parameters and adjust the quantization bits of the simulated quantization operation according to the quantization error brought by the simulated quantization operation. Embodiments of the present disclosure can simulate the error brought by the quantization operation during the training process, maintain the accuracy of the quantized model, and reduce the time and labor costs of quantization.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a method and apparatus for training a neural network model, an electronic device, and a storage medium. Background Art

[0002] With the continuous development of artificial intelligence technologies, the model structures related to artificial intelligence are designed to be more and more complex. The corresponding weight parameters of the models are getting larger and larger, the inference time of the models is getting longer and longer, and the computing power requirements for hardware devices are getting higher and higher, which poses greater challenges to hardware devices.

[0003] To accelerate the inference speed of a model, the trained model can be quantized to a lower bit width. However, the quantization process will introduce quantization errors, resulting in a decrease in the inference accuracy of the quantized model. To reduce the quantization errors, related technologies usually need to perform quantitative analysis on the model weights and fine-tune the model weights. At present, when neural network models (such as large language models) are becoming more and more widely used, the above methods will incur high time and labor costs in practice. Summary of the Invention

[0004] The present disclosure proposes a technical solution for training a neural network model.

[0005] According to one aspect of the present disclosure, there is provided a method for training a neural network model, the method including: obtaining a first neural network model to be trained; updating a first network layer in the first neural network model to a second network layer to obtain a second neural network model, where the second network layer has a quantization simulation structure added compared with the first network layer; training the second neural network model according to a second model optimizer to obtain a trained second neural network model; where the second model optimizer is determined according to the quantization simulation structure and an initial first model optimizer, the second model optimizer is used to adjust the update step size of at least one weight parameter of the second network layer, and the quantization simulation structure is used to perform a simulated quantization operation on the weight parameter and adjust the quantization bit width of the simulated quantization operation according to the quantization error brought by the simulated quantization operation.

[0006] In a possible implementation, the training process of any second network layer in the second neural network model includes: in response to the training step reaching a preset verification step, updating the first quantization label according to a verification data set to obtain a second quantization label; wherein, the first quantization label refers to the quantization label of the second network layer shared in the previous round of multi-step training, and the second quantization label refers to the quantization label of the second network layer shared in the current round of multi-step training; using the second model optimizer to determine the update step size of the second network layer in each step of the current round of training according to the second quantization label and the training data used in each step of the current round; updating the weight parameters of the second network layer according to the update step size of the second network layer in each step of the current round to obtain an updated second network layer; and when the preset training end condition is satisfied, determining the updated second network layer obtained in the current step of training as the trained second network layer.

[0007] In a possible implementation, updating the first quantization label according to a verification data set to obtain a second quantization label includes: determining a first result and a second result; wherein, the first result and the second result respectively indicate verification results obtained according to the verification data in the verification data set and the weight parameters of the second network layer when the quantization simulation structure is enabled and disabled; determining the loss between the second result and the first result as a third result; and updating the quantization label of the second network layer according to the third result.

[0008] In a possible implementation, the quantization label of the second network layer includes a first label, a second label, and a third label. The first label is used to mark whether the second network layer enables or disables the quantization simulation structure; the second label is used to mark the quantization error of the simulated quantization operation; the third label is used to mark the quantization bit number of the simulated quantization operation. Updating the quantization label of the second network layer according to the third result includes: in response to the difference between the third result and the second label being greater than or equal to a first preset threshold, when the third label is the preset maximum value, setting the first label to a first identifier, and when the third label is less than the preset maximum value, expanding the third label by a first preset multiple, wherein the first identifier represents that the second network layer disables the quantization simulation structure to perform the simulated quantization operation; in response to the difference between the third result and the second label being less than or equal to a second preset threshold, shrinking the third label by a second preset multiple and setting the first label to a second identifier, wherein the second identifier represents that the second network layer enables the quantization simulation structure to perform the simulated quantization operation; in response to the difference between the third result and the second label being less than the first preset threshold and greater than the second preset threshold, setting the second label to the third result.

[0009] In a possible implementation, the second model optimizer determines the update step size of the second network layer in each step of the current round of training according to the second quantization label and the training data used in each step of the current round of training, including: when the first label is the second identifier, the second model optimizer performs a quantization operation on the gradient of the weight parameter of the second network layer according to the third label and the training data to obtain a quantization gradient; the second model optimizer determines the update step size of the second network layer according to the quantization gradient.

[0010] In a possible implementation, the quantization simulation structure includes a first quantization parameter, a second quantization parameter, a first quantization operator, and a second quantization operator. The quantization simulation structure is used to: input the weight parameter of the second network layer, the first quantization parameter, and the second quantization parameter into the first quantization operator to obtain a first quantization result; input the first quantization result, the first quantization parameter, and the second quantization parameter into the second quantization operator to obtain a second quantization result, where the second quantization result is the weight parameter after the simulated quantization operation; where the first quantization parameter represents a scaling ratio, and the second quantization parameter represents a zero point position; the first quantization operator is used to add the clamping operation result of the quotient of the weight parameter of the second network layer and the first quantization parameter to the second quantization parameter; the second quantization operator is used to multiply the difference between the first quantization result and the second quantization parameter by the first quantization parameter.

[0011] In a possible implementation, the method further includes: quantizing the trained second neural network model according to the quantization label to obtain a quantized model; inputting the data to be processed into the quantized model for processing to obtain a processing result, where the quantized model is a model for performing at least one task among image classification, object detection, image segmentation, and speech recognition, and the data to be processed includes at least one of image data, speech data, and text data.

[0012] According to one aspect of the present disclosure, there is provided a neural network model training apparatus, including: an acquisition module configured to acquire a first neural network model to be trained; an update module configured to update a first network layer in the first neural network model to a second network layer, obtaining a second neural network model, wherein the second network layer has an additional quantization simulation structure compared to the first network layer; a training module configured to train the second neural network model according to a second model optimizer, obtaining a trained second neural network model; wherein, the second model optimizer is determined according to the quantization simulation structure and an initial first model optimizer, and the second model optimizer is used to adjust an update step size of weight parameters of at least one of the second network layers, and the quantization simulation structure is used to perform a simulated quantization operation on the weight parameters and adjust a quantization bit number of the simulated quantization operation according to a quantization error brought by the simulated quantization operation.

[0013] In a possible implementation, the training module is configured to train any second network layer in the second neural network model, and a training process of any second network layer in the second neural network model includes: in response to a training step reaching a preset verification step, updating a first quantization label according to a verification data set, obtaining a second quantization label; wherein, the first quantization label refers to the quantization label of the second network layer shared in the previous round of multi-step training, and the second quantization label refers to the quantization label of the second network layer shared in the current round of multi-step training; using the second model optimizer to determine an update step size of the second network layer in each step of the current round of training according to the second quantization label and training data used in each step of the current round; updating weight parameters of the second network layer according to the update step size of the second network layer in each step of the current round, obtaining an updated second network layer; in a case where a preset training end condition is satisfied, determining the updated second network layer obtained in the current step of training as a trained second network layer.

[0014] In a possible implementation, updating the first quantization label according to the verification data set to obtain the second quantization label includes: determining a first result and a second result; wherein, the first result and the second result respectively indicate verification results obtained according to verification data in the verification data set and weight parameters of the second network layer in cases where the quantization simulation structure is enabled and disabled; determining a loss between the second result and the first result as a third result; updating the quantization label of the second network layer according to the third result.

[0015] In a possible implementation, the quantization tags of the second network layer include a first tag, a second tag, and a third tag. The first tag is used to mark whether the second network layer enables or disables the quantization simulation structure; the second tag is used to mark the quantization error of the analog quantization operation; the third tag is used to mark the quantization bit number of the analog quantization operation. Updating the quantization tags of the second network layer according to the third result includes: in response to the difference between the third result and the second tag being greater than or equal to a first preset threshold, when the third tag is the preset maximum value, setting the first tag to a first identifier, and when the third tag is less than the preset maximum value, expanding the third tag by a first preset multiple, where the first identifier represents that the second network layer disables the quantization simulation structure to perform the analog quantization operation; in response to the difference between the third result and the second tag being less than or equal to a second preset threshold, shrinking the third tag by a second preset multiple and setting the first tag to a second identifier, where the second identifier represents that the second network layer enables the quantization simulation structure to perform the analog quantization operation; in response to the difference between the third result and the second tag being less than the first preset threshold and greater than the second preset threshold, setting the second tag to the third result.

[0016] In a possible implementation, the second model optimizer determines the update step size of the second network layer in each step of the current round of training according to the second quantization tag and the training data used in each step of the current round of training, including: when the first tag is the second identifier, the second model optimizer performs a quantization operation on the gradient of the weight parameters of the second network layer according to the third tag and the training data to obtain a quantized gradient; the second model optimizer determines the update step size of the second network layer according to the quantized gradient.

[0017] In a possible implementation, the quantization simulation structure includes a first quantization parameter, a second quantization parameter, a first quantization operator, and a second quantization operator. The quantization simulation structure is used for: inputting the weight parameters of the second network layer, the first quantization parameter, and the second quantization parameter into the first quantization operator to obtain a first quantization result; inputting the first quantization result, the first quantization parameter, and the second quantization parameter into the second quantization operator to obtain a second quantization result, and the second quantization result is the weight parameter after the analog quantization operation; where the first quantization parameter represents a scaling ratio, and the second quantization parameter represents a zero-point position; the first quantization operator is used to add the clamping operation result of the quotient of the weight parameters of the second network layer and the first quantization parameter to the second quantization parameter; the second quantization operator is used to multiply the difference between the first quantization result and the second quantization parameter by the first quantization parameter.

[0018] In a possible implementation, the device further includes a quantization module, configured to: quantize the trained second neural network model according to the quantization tag to obtain a quantized model; input the data to be processed into the quantized model for processing to obtain a processing result, where the quantized model is a model for performing at least one of image classification, object detection, image segmentation, and speech recognition, and the data to be processed includes at least one of image data, speech data, and text data.

[0019] According to one aspect of the present disclosure, a neural network model is provided. The neural network model includes a second network layer, and the second network layer includes a quantization simulation structure. The neural network model is trained based on a second model optimizer; the second model optimizer is determined according to the quantization simulation structure and an initial first model optimizer, and the second model optimizer is used to adjust the update step size of the weight parameters of at least one of the second network layers. The quantization simulation structure is used to perform a simulated quantization operation on the weight parameters and adjust the quantization bits of the simulated quantization operation according to the quantization error brought by the simulated quantization operation.

[0020] According to one aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0021] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented.

[0022] According to one aspect of the present disclosure, a computer program product is provided, including computer program instructions, and when the computer program instructions are executed by a processor, the above method is implemented.

[0023] In the embodiments of the present disclosure, by setting up a quantization simulation structure, it is not necessary to load the first neural network model and the second neural network model simultaneously. By loading a single second neural network model, quantization-aware simulation can be performed during the training phase of the second neural network model. According to the quantization error brought about by the simulated quantization operation, the quantization bit number of the simulated quantization operation can be adaptively adjusted, enabling the maintenance of the same accuracy as the original first neural network model during the training process and still not losing accuracy after quantization. Moreover, the second model optimizer is determined based on the quantization simulation structure and the initial first model optimizer. The second model optimizer can adaptively adjust the update step size of the weight parameters, taking into account the error brought about by the simulated quantization operation during the training process of the second neural network model and accelerating the model's convergence speed. The trained second neural network model can be conveniently quantized to low bits while minimizing the quantization error, reducing time and labor costs.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear based on the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into the specification and form a part of this specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0026] Figure 1 A flowchart showing a method for training a neural network model according to an embodiment of the present disclosure.

[0027] Figure 2 A comparison schematic diagram showing a second network layer and a first network layer according to an embodiment of the present disclosure.

[0028] Figure 3 A schematic diagram showing a method for training a second network layer according to an embodiment of the present disclosure.

[0029] Figure 4 A schematic diagram showing a determination method for a first result and a second result according to an embodiment of the present disclosure.

[0030] Figure 5 A schematic diagram showing a method for updating quantization tags according to an embodiment of the present disclosure.

[0031] Figure 6 A block diagram showing a neural network model training apparatus according to an embodiment of the present disclosure.

[0032] Figure 7 A block diagram showing an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. Identical reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0034] As used herein, the term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0035] As used herein, the term "and / or" is merely a description of an association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. Additionally, as used herein, the term "at least one" means any one of a plurality or any combination of at least two of a plurality. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set composed of A, B, and C.

[0036] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can be implemented without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0037] Most of the quantization-aware training methods in the related art rely on the convolution and BatchNorm structures in deep learning models. However, some current neural network models (such as large language models) are mainly composed of linear layers and do not have the above two structural types, so they cannot be directly applied to various neural network models.

[0038] Moreover, the methods in the related art need to load both the pre-quantization model and the model to be quantized on the model training device, and update the model to be quantized by comparing the two. Neural network models usually consume a large amount of device resources. Loading two models simultaneously will increase the device consumption, and it is also time-consuming to compare the differences between different models on neural network models, resulting in increased time and cost.

[0039] In view of this, embodiments of the present disclosure propose a method for training a neural network model, which updates a first network layer in a first neural network model to be trained to a second network layer with a quantization simulation structure added, obtaining a second neural network model, and modifying an initial first model optimizer according to the quantization simulation structure to obtain a modified second model optimizer; training the second neural network model according to the quantization simulation structure and the second model optimizer to obtain a trained second neural network model; wherein, during the training process of the second neural network model, the second model optimizer is used to adjust the update step size of at least one weight parameter of the second network layer, and the quantization simulation structure is used to perform a simulated quantization operation on the weight parameter and adjust the quantization bits of the simulated quantization operation according to the quantization error brought by the simulated quantization operation.

[0040] By setting the quantization simulation structure, it is not necessary to load the first neural network model and the second neural network model at the same time. By loading one copy of the second neural network model, quantization-aware simulation can be performed during the training stage of the second neural network model. According to the quantization error brought by the simulated quantization operation, the quantization bits of the simulated quantization operation can be adaptively adjusted, and the same accuracy as the original first neural network model can be maintained during the training process, and the accuracy is still not lost after quantization. Moreover, the second model optimizer can adaptively adjust the update step size of the weight parameter, and can take the error brought by the simulated quantization operation into account in the training process of the second neural network model, accelerating the convergence speed of the model. The trained second neural network model can be conveniently quantized to low bits and keep the quantization error minimized, reducing the time and labor costs.

[0041] Figure 1 The flowchart showing the method for training a neural network model according to an embodiment of the present disclosure is as Figure 1 shown, and the method for training the neural network model includes:

[0042] In step S11, a first neural network model to be trained is obtained.

[0043] In step S12, the first network layer in the first neural network model is updated to a second network layer, obtaining a second neural network model, and the second network layer has a quantization simulation structure added compared with the first network layer.

[0044] In step S13, the second neural network model is trained according to the second model optimizer to obtain a trained second neural network model.

[0045] Among them, during the training process of the second neural network model, the second model optimizer is determined according to the quantization simulation structure and the initial first model optimizer. The second model optimizer is used to adjust the update step size of at least one (such as each or part of) the weight parameters of the second network layer. The quantization simulation structure is used to perform simulated quantization operations on the weight parameters and adjust the quantization bits of the simulated quantization operations according to the quantization errors brought by the simulated quantization operations.

[0046] In a possible implementation manner, the training method of the neural network model can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by the processor calling computer-readable instructions stored in the memory. Alternatively, the method can be executed by the server.

[0047] In a possible implementation manner, in step S11, the first neural network model to be trained obtained may include multiple convolutional layers, pooling layers, fully connected layers, etc. The first neural network model may be at least one of a convolutional neural network (CNN), a deep learning neural network (DNN), a recurrent neural network (RNN), a residual network (ResNets), a backpropagation neural network (BP), a backbone neural network, a large language model (LLM), etc. The present disclosure does not specifically limit the network structure of the first neural network model.

[0048] Optionally, the first neural network model to be trained may be a neural network model for any one of tasks such as image classification, object detection, image segmentation, speech recognition, natural language processing, machine translation, and question answering systems. The present disclosure does not limit this.

[0049] In step S11, the first neural network model to be trained is obtained. In step S12, the first network layer in the first neural network model to be trained can be updated to a second network layer with an added quantization simulation structure to obtain a second neural network model.

[0050] In a possible implementation, the first network layer in the first neural network model can be any layer in the first neural network model. In some specific embodiments, according to the actual application scenario, some or all of the first network layers in the first neural network model can be determined as the second network layer with a quantization simulation structure added, to obtain the second neural network model to be trained. When there are multiple second network layers in the second neural network model, the second network layers can be continuous or discontinuous. Depending on the second neural network model, the types of the second network layers with the quantization simulation structure added can also be different. For example, the second network layer can be a convolutional layer, a fully connected layer, or an input layer, a hidden layer, an output layer, etc. The number and type of the layers to be quantized are not specifically limited herein.

[0051] In a possible implementation, most of the first network layers to be updated are the weight parameter layers in the first neural network model, such as convolutional layers and linear layers (also called fully connected layers). Adding a quantization simulation structure to the first network layer to be trained can be used to perform a simulated quantization operation on the weight parameters of the updated second network layer in the subsequent training process. First, perform a low-bit conversion on it, and then convert it back to the original number of bits. The weight parameters originally represented by high-precision floating-point numbers can be converted to low-precision representation, so as to reduce the computational complexity, improve the inference speed, and reduce the power consumption.

[0052] Optionally, in the subsequent training process, the quantization simulation structure of at least one second network layer can adaptively adjust the quantization bits of the simulated quantization operation according to the quantization error brought by the simulated quantization operation. The quantization bits refer to the number of bits for storing the weight parameters corresponding to each second network layer. Optionally, the quantization bits can be 4bit, 6bit, 8bit, etc., and the quantization bits to be used for each second network layer can be set according to the simulated quantization error of each second network layer; optionally, the quantization bits of each second network layer can be the same or different, and can be set according to actual needs, which are not specifically limited herein.

[0053] Optionally, since the second network layer is substantially equivalent to the first network layer when the quantization simulation structure is not enabled, the same input data can be input to the second network layer with the quantization simulation structure enabled and the second network layer without the quantization simulation structure respectively. By comparing the differences in the processing results in these two cases, the simulated quantization error brought by performing the simulated quantization operation on the weight parameters can be obtained.

[0054] In step S12, the second neural network model with the quantization simulation structure added is obtained. In step S13, the second neural network model is trained according to the second model optimizer to obtain the trained second neural network model. The second model optimizer can be determined according to the quantization simulation structure and the initial first model optimizer.

[0055] Optionally, the first optimizer is used to update and adjust the network parameters (such as the weight parameters of the second network layer) of the neural network model during the training process. The first optimizer may include, for example, Stochastic Gradient Descent (SGD), Batch Gradient Descent (BGD), Mini Batch Gradient Descent (MBGD), Momentum, Adaptive Gradient (Adagrad), Root Mean Square Prop (RMSProp), Adaptive Moment Estimation (Adam), etc. The embodiments of the present disclosure do not specifically limit the type of the first optimizer.

[0056] Optionally, the learning rate of the first model optimizer can be modified according to the quantization simulation structure to obtain the modified second model optimizer. The modified second model optimizer can adaptively adjust the update step of the weight parameters of at least one second network layer according to the number of bits to be quantized of at least one second network layer determined by the quantization simulation structure. It should be understood that in actual applications, the update step of the weight parameters of each second network layer can be adjusted, or the update step of the weight parameters of some second network layers can be adjusted. The embodiments of the present disclosure do not limit this.

[0057] In step S12, the first network layer in the first neural network model is updated to the second network layer with the quantization simulation structure added to obtain the second neural network model. In step S13, the modified second model optimizer can be obtained according to the quantization simulation structure, and the second neural network model can be trained according to the quantization simulation structure and the second model optimizer to obtain the trained second neural network model.

[0058] In a possible implementation manner, the second neural network model can be subjected to quantization-aware training according to the quantization simulation structure and the second model optimizer. Optionally, the quantization-aware training method can include, but is not limited to, the Learned Step-size Quantization (LSQ), the Parameterized Clipping Activation (PACT) algorithm, the Additive Powers-of-Two (APoT), the Differentiable Soft Quantization (DSQ), the Learned Quantization for Highly Accurate and Compact Deep Neural Networks (LQ-net), etc. The embodiments of the present disclosure do not limit this.

[0059] In a possible implementation manner, the second neural network model can be subjected to quantization-aware training according to the training task and the training data to obtain a trained second neural network model. The second neural network model can be a model for any one of the tasks of image classification, object detection, image segmentation, and speech recognition, which is not limited herein.

[0060] In a possible implementation manner, during the training of the second neural network model, each second network layer quantization simulation structure can be used to perform a simulated quantization operation on its respective weight parameters, and the weight parameters originally represented in high precision can be converted into low-precision representation. Among them, each second network layer quantization simulation structure can adaptively adjust the quantization bits of the simulated quantization operation according to the quantization error brought by the simulated quantization operation. The quantization bits can be used to dynamically adjust the precision of the second network layer. During the training process, if the quantization error is relatively small, the quantization bits can be set relatively small; if the quantization error is relatively large, the quantization bits can be set relatively large, and even the quantization simulation structure can be disabled to cancel the simulated quantization operation on the weight parameters.

[0061] Among them, in order to cooperate with the work of the quantization simulation structure, a quantization label can be set for the second network layer. The quantization label can be used to indicate whether to perform a simulated quantization operation on the weight parameters of the second network layer during the training process, and in the case of determining to perform a simulated quantization operation on the weight parameters of the second network layer, the quantization bits used for the simulated quantization operation.

[0062] In a possible implementation, during the training process of the second neural network model, the second model optimizer can adaptively adjust the update step size of the weight parameters of the second network layer according to the quantization bits corresponding to the second network layer determined based on the quantization simulation structure, and take the error brought by quantization into account in the model training process.

[0063] By setting up the quantization simulation structure, it is not necessary to load the first neural network model and the second neural network model simultaneously. By loading a single second neural network model, quantization-aware simulation can be performed during the training phase of the second neural network model. According to the quantization error brought by the simulated quantization operation, the quantization bits of the simulated quantization operation can be adaptively adjusted, so that the same accuracy as the original first neural network model can be maintained during the training process, and the accuracy is still not lost after quantization. Moreover, the second model optimizer can adaptively adjust the update step size of the weight parameters, and can take the error brought by the simulated quantization operation into account in the training process of the second neural network model, accelerating the convergence speed of the model. The trained second neural network model can be conveniently quantized to low bits, and the quantization error can be minimized, reducing the time and labor costs.

[0064] The training method of the neural network model according to the embodiments of the present disclosure will be described in detail below.

[0065] In step S11, when the first neural network model to be trained is obtained, in step S12, the first network layer in the first neural network model can be updated to a second network layer to obtain a second neural network model, and the second network layer has a quantization simulation structure added compared with the first network layer.

[0066] Figure 2 A comparison schematic diagram of the second network layer and the first network layer according to the embodiments of the present disclosure is shown. As Figure 2 shown, the left part is the first network layer, and the right part is the second network layer. The second network layer has a quantization simulation structure added compared with the first network layer. In this way, if the quantization simulation structure is not enabled in the second network layer, the structure of the second network layer is the same as that of the first network layer, and the quantization of the weight parameters may not be involved during the training process; if the quantization simulation structure is enabled in the second network layer, a simulated quantization operation will be performed on the weight parameters during the training process, converting the high-precision weight parameters into a lower-precision representation form for accelerating model training and reducing the storage space requirement.

[0067] Among them, in order to cooperate with the work of the quantization simulation structure, a quantization label can be set for the second network layer, and this quantization label can be used to indicate whether to perform a simulated quantization operation on the weight parameters of the second network layer during the training process, and in the case of determining to perform a simulated quantization operation on the weight parameters of the second network layer, the quantization bits used for this simulated quantization operation.

[0068] In a possible implementation, the quantization tags of the second network layer include a first tag, a second tag, and a third tag. The first tag is used to mark whether the second network layer enables or disables the quantization analog structure to perform analog quantization operations; the second tag is used to mark the quantization error of the analog quantization operation; the third tag is used to mark the number of quantization bits of the analog quantization operation.

[0069] In a possible implementation, as Figure 2 shown in the right part, the quantization analog structure includes a first quantization parameter, a second quantization parameter, a first quantization operator, and a second quantization operator. Among them, when the quantization analog structure is enabled in the second network layer, the weight parameters of the second network layer, the first quantization parameter, and the second quantization parameter can be input into the first quantization operator to obtain a first quantization result; the first quantization result, the first quantization parameter, and the second quantization parameter are input into the second quantization operator to obtain a second quantization result, and the second quantization result is the weight parameter after the analog quantization operation, converting the high-precision weight parameter into a lower-precision representation form. Or, when the quantization analog structure is not enabled in the second network layer, the weight parameters of the second network layer can directly participate in training without performing analog quantization operations and will retain their original data precision.

[0070] Exemplarily, the first quantization parameter represents a scaling ratio, which is determined according to the maximum and minimum values of the input data of the second network layer and the quantization tags, and can be expressed as:

[0071] scale= (max - min) / 2 quant _ bit (1)

[0072] In formula (1), scale is the first quantization parameter representing the scaling ratio, quant_bit represents the third tag in the quantization tags used to mark the number of quantization bits of the analog quantization operation, max represents the maximum value of the input data of the second network layer, and min represents the minimum value of the input data of the second network layer. For example, assuming the input data is a matrix of M rows and N columns, max represents the maximum value among the M×N elements in the matrix, and min represents the minimum value among the M×N elements in the matrix.

[0073] Exemplarily, the second quantization parameter represents the zero point position, which is determined according to the maximum value of the input data of the second network layer, the first quantization parameter, and the quantization tags, and can be expressed as:

[0074] zero_point = 2 quant _ bit- max / scale (2)

[0075] In formula (2), zero_point is the second quantization parameter representing the zero-point position, scale is the first quantization parameter representing the scaling ratio, quant_bit is the third label in the quantization label used to mark the quantization bit number for the analog quantization operation, and max represents the maximum value of the input data of the second network layer.

[0076] Exemplarily, the first quantization operator is used to determine a first quantization result obtained by adding the clipping operation result and the second quantization parameter. The clipping operation result is obtained by performing a clipping operation on the quotient of the weight parameter of the second network layer and the first quantization parameter based on the quantization label, and can be expressed as:

[0077] Q = quant(weight, quant_bit) = clamp(weight / scale, quant_bit) + zero_point (3)

[0078] In formula (3), weight represents the weight parameter of the second network layer, scale is the first quantization parameter representing the scaling ratio, quant_bit is the third label in the quantization label used to mark the quantization bit number for the analog quantization operation, zero_point is the second quantization parameter representing the zero-point position, and the clipping operation function clamp() can output the clipping operation result, indicating that the element value of the quotient of the weight parameter weight of the second network layer and the first quantization parameter scale is limited to a number of bits with the value of quant_bit. For example, clamp(260, 8) = 255, where the maximum value of an 8-bit integer is 255 and 260 is truncated to 255. quant() represents the first quantization operator. By adding the clipping operation result clamp(weight / scale, quant_bit) and the second quantization parameter zero_point, the first quantization result Q of the first quantization operator quant(weight, quant_bit) can be obtained.

[0079] Exemplarily, the second quantization operator is used to determine a second quantization result obtained by multiplying the difference between the first quantization result and the second quantization parameter by the first quantization parameter. The second quantization result is the weight parameter after the analog quantization operation.

[0080] dequant(Q) = (Q - zero_point) × scale (4)

[0081] In formula (4), Q represents the first quantization result obtained from formula (3), zero_point is the second quantization parameter representing the zero-point position, scale is the first quantization parameter representing the scaling ratio, and dequant() represents the second quantization operator, which is used to determine the second quantization result of the product of the difference between the first quantization result Q and the second quantization parameter zero_point and the first quantization parameter scale. This second quantization result is the weight parameter after the quantization simulation structure is enabled in the second network layer to perform the simulation quantization operation.

[0082] By setting the quantization simulation structure in the second network layer, it is beneficial to introduce quantization during the training phase, convert the high-precision weight parameters into a lower-precision representation form, which can more effectively reduce the size and memory occupancy of the model, and accelerate the inference speed of the model.

[0083] In step S12, the first network layer in the first neural network model is updated to the second network layer with the quantization simulation structure added. In step S13, according to the quantization simulation structure, the initial first model optimizer is modified to obtain the modified second model optimizer.

[0084] In the example, the initial first model optimizer can be expressed as:

[0085] weight = weight - learning_rate × grad (5)

[0086] In formula (5), weight represents the weight parameter of the second network layer, grad represents the gradient of the weight parameter weight, and learning_rate represents the learning rate, which is a hyperparameter used to control the adjustment speed of the weight parameter and can be set manually according to experience before training. The present disclosure does not limit the specific value of the learning rate.

[0087] Comparing with the first model optimizer shown in formula (5), the modified second model optimizer can be expressed as:

[0088] weight = weight - learning_rate × (grad + H -1 ) (6)

[0089] In formula (6), weight represents the weight parameter of the second network layer, grad represents the gradient of the weight parameter weight, and learning_rate represents the learning rate, which is a hyperparameter used to control the adjustment speed of the weight parameter and can be set manually according to experience before training. The present disclosure places no restrictions on the specific value of the learning rate. Moreover, compared with the first model optimizer shown in formula (5), an approximate Hessian inverse matrix H is added to the weight parameter weight update part -1 , which is used to adaptively adjust the step size of the weight parameter update during the training process. In addition to being able to take into account the error brought by the simulated quantization operation in the model training process, it can also accelerate the convergence speed of the model.

[0090] In step S12, the first network layer in the first neural network model is updated to the second network layer with a quantization simulation structure added, obtaining the second neural network model. In step S13, a modified second model optimizer can be obtained according to the quantization simulation structure, and the second neural network model is trained according to the quantization simulation structure and the second model optimizer to obtain the trained second neural network model.

[0091] In a possible implementation Figure 3 A schematic diagram showing the training method of the second network layer according to an embodiment of the present disclosure is as Figure 3 shown. The training process of any second network layer in the second neural network model includes: in step S131, in response to the training step number reaching a preset verification step number, the first quantization label is updated according to the verification data set to obtain a second quantization label; wherein, the first quantization label refers to the quantization label of the second network layer shared by the previous round of multi-step training, and the second quantization label refers to the quantization label of the second network layer shared by the current round of multi-step training. That is, the quantization label of the second network layer shared by the previous round of multi-step training is adaptively updated according to the verification data set to obtain the quantization label of the second network layer shared by the current round of multi-step training; wherein, the verification data set can be the same as the training data set or different from the training data set. The verification data set can be a set of data for a certain field or a certain task, and the embodiments of the present disclosure place no restrictions on this.

[0092] In step S132, the second model optimizer is used to determine the update step size of the second network layer in each step of the current round according to the second quantization label (the quantization label of the second network layer shared by the current round of multi-step training) and the training data used in each step of the current round;

[0093] In step S133, the weight parameter of the second network layer is updated according to the update step size of the second network layer in each step of the current round to obtain the updated second network layer;

[0094] In step S134, when the preset training end condition is satisfied, the updated second network layer obtained by training in the current step is determined as the trained second network layer.

[0095] In this way, during the training process, the quantization labels of each second network layer in the second neural network model and the update step size of the weight parameters of the second network layer can be dynamically adjusted, which is beneficial to considering the error caused by quantization in the training process, accelerating the convergence speed of the second neural network model, and maintaining the same accuracy as the original first neural network model during the training process, reducing the error caused by the simulated quantization operation.

[0096] Exemplarily, in step S131, the verification step number of the second neural network model can be preset, so that the second neural network model can verify at least one (e.g., each) second network layer in the neural network model after each training verification step number. During the verification process, the quantization labels of at least one (e.g., each) second network layer are adaptively updated. The quantization label can be used to indicate whether to perform a simulated quantization operation on the weight parameters of the second network layer in the next round of multi-step training for at least one (e.g., each) second network layer, and in the case of determining to perform a simulated quantization operation on the weight parameters of the second network layer, the quantization bit number used in the simulated quantization operation. For example, the quantization label can include a first label, a second label, and a third label. The first label is used to mark whether the second network layer enables or disables the quantization simulation structure to perform the simulated quantization operation; the second label is used to mark the quantization error of the simulated quantization operation; the third label is used to mark the quantization bit number of the simulated quantization operation.

[0097] For a certain second network layer in the second neural network model, assuming that the preset verification step number is K (e.g., K = 1000), when training to the j-th step of the i-th round, if j is less than the verification step number K, the second network layer can be trained for the j-th step according to the training data and the quantization label of the second network layer. Among them, the quantization labels used in each step of the i-th round are the same, and the value of the quantization label in the i-th round is adaptively updated according to the verification data set for the quantization label of the second network layer in the previous round (the i - 1-th round) when the previous round (the i - 1-th round) is executed to the K-th step. Among them, the present disclosure does not limit the specific values of the verification step number K, the training round number i, and the training step number j.

[0098] Optionally, if the quantization label in the i-th round indicates to stop performing the simulated quantization operation on the weight parameters of the second network layer, the quantization simulation structure is disabled in the iterative training of each step in the i-th round of the second network layer, and the weight parameters of the second network layer are no longer quantized, and the weight parameters can be represented by floating-point numbers with higher precision.

[0099] Optionally, if the quantization tag in the i-th round indicates a simulated quantization operation on the weight parameters of the second network layer, and the quantization tag also indicates the quantization bits used in the simulated quantization operation, in each step of the iterative training in the i-th round, the second network layer will use a quantization simulation structure. By performing a simulated quantization operation on the weight parameters of the second network layer, the weight parameters represented by higher-precision floating-point numbers can be converted into weight parameters represented by lower-precision with the quantization bits indicated by the quantization tag.

[0100] If the j-th step in the i-th round reaches the verification step number K, the training of the (i + 1)-th round can be entered, and the quantization tag of the second network layer in the i-th round can be adaptively updated according to the verification data set to obtain the quantization tag of the second network layer shared by each step of the (i + 1)-th round of training.

[0101] In a possible implementation, the first quantization tag can be updated according to the verification data set in step S131 to obtain a second quantization tag, which may include: determining a first result and a second result; wherein, the first result and the second result respectively indicate verification results obtained according to the verification data in the verification data set and the weight parameters of the second network layer in the case of enabling and disabling the quantization simulation structure; that is, in the case where the second network layer disables the quantization simulation structure, the first result can be determined according to the verification data in the verification data set and the weight parameters of the second network layer; in the case where the second network layer enables the quantization simulation structure, the second result can be determined according to the verification data and the weight parameters of the second network layer; determining the loss between the second result and the first result as a third result; and adaptively updating the quantization tag of the second network layer according to the third result.

[0102] Figure 4 A schematic diagram showing the determination method of the first result and the second result according to an embodiment of the present disclosure is as Figure 4 shown. The second network layer can be a linear layer (also called a fully connected layer). For any second network layer, the quantization simulation structure can be disabled (see Figure 4 the left part). First, calculate its calculation without simulated quantization error, that is, directly multiply the input data by the weight parameters of the second network layer and then add the bias parameters of the second network layer to obtain the first result, that is:

[0103] rst1 = activation @ weigh + bias (7)

[0104] In formula (7), activation represents the input data of the second network layer. This input data can be the verification data in the verification dataset or the input data determined by the network layer before the second network layer based on the verification data. weigh represents the weight parameter of the second network layer, bias represents the bias parameter of the second network layer, @ represents matrix multiplication, rst1 represents the product result of the input data activation and the weight parameter weigh, plus the first result rst1 of the bias parameter bias, that is, the first result rst1 output by the second network layer without enabling the quantization simulation structure.

[0105] Synchronous, such as Figure 4 As shown in the right part, the quantization simulation structure can be enabled. First, perform a simulated quantization operation on the weight parameter of the second network layer, multiply the input data by the weight parameter after the simulated quantization operation, and then add the bias parameter of the second network layer to obtain the second result, that is:

[0106] rst2 = activation@dequant(quant(weight, quant_bit)) +bias (8)

[0107] In formula (8), rst2 represents the second result, activation represents the input data of the second network layer. This input data can be the verification data in the verification dataset or the input data determined by the network layer before the second network layer based on the verification data. weigh represents the weight parameter of the second network layer, bias represents the bias parameter of the second network layer, @ represents matrix multiplication, weight represents the weight parameter of the second network layer, quant_bit represents the third label in the quantization label used to mark the quantization bit number of the simulated quantization operation. The simulated quantization operation dequant(quant(weight,quant_bit)) can be composed of the first quantization operator quant() shown in formula (3) and the second quantization operator dequant() shown in formula (4). For details, please refer to formulas (3) and (4) above and will not be elaborated here.

[0108] After obtaining the first result rst1 and the second result rst2, the loss between the second result rst2 and the first result rst1 can be determined as the third result rst3. For example, the L2 norm of the first result rst1 and the second result rst2 can be calculated and denoted as the third result rst3, that is:

[0109] rst3 = ||rst1, rst2|| 2 =sqrt(sum((rst1-rst2) 2 )) (9)

[0110] In formula (9), rst1 represents the first result, rst2 represents the second result, rst3 represents the third result, and ||·|| 2 represents the L2 norm, sum() represents the summation function, and sqrt() represents the square root function.

[0111] After obtaining the determined third result rst3, the quantization tags of the second network layer are adaptively updated by comparing the third result rst3.

[0112] In this way, it is not necessary to load the first network layer and the second network layer simultaneously. Loading only one copy of the second network layer can adaptively update the quantization tags of the second network layer according to the obtained third result.

[0113] Figure 5 A schematic diagram showing the method for updating quantization tags according to an embodiment of the present disclosure is as follows Figure 5 As shown, the quantization tags of the second network layer include a first tag, a second tag, and a third tag. The first tag is used to mark whether the second network layer enables or disables the quantization simulation structure to perform the simulated quantization operation; the second tag is used to mark the quantization error of the simulated quantization operation; the third tag is used to mark the quantization bit number of the simulated quantization operation.

[0114] Optionally, for ease of description, the first tag can be denoted as is_quant. The value of the first tag is_quant can be a first identifier (e.g., 0) or a second identifier (e.g., 1). During the training process of the second neural network model, it is possible to determine whether each second network layer in the second neural network model starts the quantization simulation structure to perform the simulated quantization operation by judging the value of the first tag is_quant. For example, during the training process, if the value of the first tag is_quant of a certain second network layer is the first identifier (e.g., 0), the quantization simulation structure of this second network layer is disabled, and the simulated quantization operation is not performed on the weight parameters of this second network layer; if the value of the first tag is_quant of a certain second network layer is the second identifier (e.g., 1), the quantization simulation structure of this second network layer is enabled, and the simulated quantization operation is performed on the weight parameters of this second network layer.

[0115] Wherein, the first identifier and the second identifier can be different numbers, symbols, letters, etc., and the present disclosure does not limit this.

[0116] Since, during the training process, the first label is_quant of each second network layer can be updated once per round, for the same second network layer, in iterative training of different rounds (each round can include multiple steps of training), there can be some rounds in which the quantization simulation structure is enabled to perform a simulated quantization operation on the weight parameters of the second network layer, and there can be other rounds in which the quantization simulation structure is disabled and no simulated quantization operation is performed on the weight parameters of the second network layer. Similarly, for multiple second network layers in the same round of training, there can be some second network layers for which the quantization simulation structure is enabled to perform a simulated quantization operation on their respective weight parameters, and there can be other second network layers for which the quantization simulation structure is disabled and no simulated quantization operation is performed on their respective weight parameters.

[0117] Optionally, the second label can be denoted as quant_err, which can be used to mark the quantization error of the simulated quantization operation corresponding to each second network layer in each round of multi-step training. Moreover, during the process of adaptively updating the quantization label based on the validation data set in each round, the second label quant_err can be adjusted according to the third result rst3 until the second label quant_err converges to obtain the second label quant_err of the current round.

[0118] Optionally, the third label can be denoted as quant_bit. Since, during the training process, the third label quant_bit of each second network layer can be updated once per round, for the same second network layer, in iterative training of different rounds, the value of the third label quant_bit shared by each round of multi-step iterative training can be different, so that the quantization bit number for performing the quantization simulation operation on the weight parameters of the second network layer in each round can be different. Similarly, for different second network layers in the same round of training, the quantization bit numbers for performing the quantization simulation operation on their respective weight parameters can also be different.

[0119] As Figure 5 shown, the input data of the second network layer can be the output data obtained by the network layer with a prior sorting adjacent to the second network layer based on the validation data, or it can be the validation data of the validation data set, which can be set according to the specific application scenario, and the present disclosure does not limit this.

[0120] For a certain current input data, based on formula (7), the second network layer can output a first result without enabling the quantization simulation structure. Based on formula (8), the second network layer can output a second result with the quantization simulation structure enabled. Then, based on formula (9), the third result for adaptively updating the quantization label can be determined through the first result and the second result.

[0121] In a possible implementation manner, as Figure 5As shown, according to the third result, adaptively updating the quantization label of the second network layer may include: in response to the difference between the third result and the second label being greater than or equal to a first preset threshold, when the third label is the preset maximum value, setting the first label to a first identifier, and when the third label is less than the preset maximum value, expanding the third label by a first preset multiple, where the first identifier indicates that the second network layer deactivates the quantization simulation structure to perform simulated quantization operations; in response to the difference between the third result and the second label being less than or equal to a second preset threshold, shrinking the third label by a second preset multiple and setting the first label to a second identifier, where the second identifier indicates that the second network layer activates the quantization simulation structure to perform simulated quantization operations; in response to the difference between the third result and the second label being less than the first preset threshold and greater than the second preset threshold, setting the second label to the third result.

[0122] By comparing the third result, which represents the quantization error brought by the simulated quantization operation, with the previously recorded quantization error of the second label, the quantization label of the second network layer is adaptively updated, so that the quantization error can be minimized during the training process, dynamically determine whether each second network layer performs the simulated quantization operation, and dynamically adjust the quantization bits of the simulated quantization operation, improving the training accuracy and training efficiency of the second neural network model.

[0123] For example, assume that the third result is denoted as rst3, the first label is denoted as is_quant, the second label is denoted as quant_err, and the third label is denoted as quant_bit.

[0124] The third result rst3 can be compared with the second label quant_err of this second network layer. If the third result rst3 is greater than the second label quant_err by the first preset threshold (for example, including 3.0) or more, that is: rst3 - quant_err ≥ the first preset threshold. In this case, if the first label is_quant records the second identifier and the third label quant_bit is also already the preset maximum value (for example, including 16), the first label is_quant can be changed from the second identifier to the first identifier; or, if the first label is_quant records the second identifier and the third label quant_bit has not reached the preset maximum value (for example, including 16), then the third label quant_bit of this second network layer is expanded by the first preset multiple (for example, including 2 times), that is: quant_bit = quant_bit × the first preset multiple.

[0125] If the third result rst3 is less than or equal to the second preset threshold (e.g., including 0.5) than the second label quant_err, i.e., rst3 - quant_err ≤ the second preset threshold. In this case, if the first label is_quant records the first identifier, the first label is_quant can be changed from the first identifier to the second identifier, otherwise it remains unchanged; if the first label is_quant records the second identifier, the third label quant_bit can be reduced by the second preset multiple (e.g., 2) when the third label quant_bit does not reach the preset minimum value (e.g., including 4), i.e., quant_bit = quant_bit / the second preset multiple.

[0126] If the difference between the third result rst3 and the second label quant_err is less than the first preset threshold and greater than the second preset threshold, i.e., the second preset threshold < rst3 - quant_err < the first preset threshold, the second label quant_err can be updated to the third result rst3, ending the verification process of the quantization label of the second network layer in the current round and entering the next round of multi-step iterative training. The quantization label updated in the current round can be shared in the next round of training.

[0127] Among them, the preset maximum value, the preset minimum value, the first preset multiple, and the second preset multiple can be set according to the actual application scenario. The embodiments of the present disclosure do not limit the specific values of the preset maximum value, the preset minimum value, the first preset multiple, and the second preset multiple.

[0128] It should be understood that the first preset threshold and the second preset threshold are hyperparameters. When the first preset threshold is greater than the second preset threshold, different neural network models can be adjusted within a certain range as needed. The embodiments of the present disclosure do not limit the value ranges of the first preset threshold and the second preset threshold.

[0129] In addition, during the training process, in addition to verifying the second network layer with simulated quantization operations enabled in the second neural network model and adaptively updating the quantization label during the verification process, the embodiments of the present disclosure can also evaluate the accuracy of the second neural network model using a specific verification dataset, and the present disclosure does not limit this.

[0130] In step S131, the quantization label of the second network layer shared by each round of multi-step training is adaptively updated according to the verification dataset to obtain the quantization label of the second network layer shared by the current round of multi-step training. In step S132, the second model optimizer can determine the update step size of the second network layer in each step of the current round of training according to the quantization label of the second network layer shared by the current round of multi-step training and the training data used in each step of the current round of training.

[0131] In a possible implementation, step S132 includes: when the first label is the second identifier, the second model optimizer quantizes the gradient of the weight parameters of the second network layer according to the third label and the training data to obtain a quantized gradient; the second model optimizer determines an update step size of the second network layer according to the quantized gradient.

[0132] Exemplarily, when the first label is the second identifier, during backpropagation, the second model optimizer can determine the quantized gradient according to the previously updated quantized label and quantized parameters (such as the first quantization parameter and the second quantization parameter), which can be expressed as:

[0133] Q g = quant(grad, quant_bit) = clamp(grad / scale, quant_bit) + zero_point (10)

[0134] In formula (10), Q g represents the quantized gradient, quant() represents the first quantization operator, clamp() represents the clamping operation function, grad represents the gradient of the weight parameter weight of the second network layer, scale is the first quantization parameter representing the scaling ratio, quant_bit is the third label used to mark the quantization bit number for the analog quantization operation in the quantized label, and zero_point is the second quantization parameter representing the zero point position. The clamping operation result clamp(grad / scale, quant_bit) can be added to the second quantization parameter zero_point, and the quantized gradient Q can be generated by the first quantization operator quant(grad, quant_bit). g .

[0135] Calculate the approximate Hessian inverse matrix of the quantized gradient Q g to obtain H in formula (6) -1 , that is:

[0136] H -1 =1 / H d

[0137] H d =diag(H t ) (11)

[0138] H t =Q g -1

[0139] In formula (11), Q gDenote the quantization gradient as H -1 Denote the quantization gradient as Q g is the approximate Hessian inverse matrix, and diag() represents the diagonal function. For example, diag(A) is to take the diagonal elements of matrix A. Among them,

[0140] can take the diagonal matrix composed of the diagonal elements of the quantization gradient Q g , and take the derivative of this matrix. The obtained matrix is the approximate Hessian matrix. Take the inverse of this matrix (taking the reciprocal of the diagonal elements for a diagonal matrix inversion), and the approximate Hessian inverse matrix H -1 can be obtained.

[0141] The second model optimizer can use the sum of the gradient grad of the weight parameter weight of the second network layer and the approximate Hessian inverse matrix H -1 as the update step size of the second network layer.

[0142] In this way, the update step size of the weight parameter can be adaptively adjusted. In addition to being able to take into account the errors brought by the simulated quantization operation in the model training process, it can also accelerate the convergence speed of the model.

[0143] In step S132, determine the update step size of the second network layer in each step of the current training round. In step S133, according to the update step size of the second network layer in each step of the current training round, update the weight parameter of the second network layer according to formula (6) to obtain the updated second network layer;

[0144] In step S134, when the preset training end condition is met, determine the updated second network layer obtained in the current step of training as the trained second network layer.

[0145] Among them, the training end condition can be that the number of training rounds and / or steps reaches the preset end steps, or the second network layer converges, and determine the updated second network layer obtained in the current step of training as the trained second network layer.

[0146] After the training is completed, in the trained second neural network model, the second neural network model can be directly quantized according to the quantization labels of each second network layer. For example, for a certain second network layer, if its first label is the second identifier, it is quantized to the quantization bits indicated by the third label based on the quantization simulation structure.

[0147] In a possible implementation, the trained second neural network model is quantized according to the quantization tags to obtain a quantized model; the data to be processed is input into the quantized model for processing to obtain a processing result, where the quantized model is a model for performing at least one of the tasks of image classification, object detection, image segmentation, and speech recognition, and the data to be processed includes at least one of image data, speech data, and text data.

[0148] Among them, according to the quantization tags of each second network layer in the trained second neural network model, by means of formulas (1)-(4), the output result of the second quantization operator can be used as the weight parameter of the second network layer to obtain a quantized model. This quantized model can be applied to various inference tasks such as image classification, object detection, image segmentation, and speech recognition, and can efficiently and accurately determine the inference result of the inference task.

[0149] An embodiment of the present disclosure provides a training method for a neural network model, in which the first network layer in the obtained first neural network model to be trained is updated to a second network layer with a quantization simulation structure added to obtain a second neural network model, and the second neural network model is trained according to a second model optimizer to obtain a trained second neural network model; among them, the second model optimizer is determined according to the quantization simulation structure and the initial first model optimizer, and the second model optimizer is used to adjust the update step size of at least one second network layer weight parameter. The quantization simulation structure is used to perform a simulated quantization operation on the weight parameter and adjust the quantization bit number of the simulated quantization operation according to the quantization error brought by the simulated quantization operation.

[0150] By setting the quantization simulation structure, it is not necessary to load the first neural network model and the second neural network model at the same time. By loading a copy of the second neural network model, quantization-aware simulation can be performed in the training stage of the second neural network model. According to the quantization error brought by the simulated quantization operation, the quantization bit number of the simulated quantization operation can be adaptively adjusted, and the same accuracy as the original first neural network model can be maintained during the training process, and the accuracy is still not lost after quantization. Moreover, the second model optimizer can adaptively adjust the update step size of the weight parameter, and can take the error brought by the simulated quantization operation into account in the training process of the second neural network model, accelerating the convergence speed of the model. Further, the error brought by the simulated quantization operation during the training process enables the trained second neural network model to be directly quantized, making it more convenient to quantize to low bits and keeping the quantization error minimized, reducing the time and labor costs.

[0151] Embodiments of the present disclosure also propose a neural network model. The neural network model includes a second network layer, and the second network layer includes a quantization simulation structure. The neural network model is trained based on a second model optimizer; the second model optimizer is determined according to the quantization simulation structure and an initial first model optimizer. The second model optimizer is used to adjust the update step size of the weight parameters of at least one of the second network layers. The quantization simulation structure is used to perform a simulated quantization operation on the weight parameters and adjust the quantization bits of the simulated quantization operation according to the quantization error brought by the simulated quantization operation.

[0152] It can be understood that, without violating the principle logic, the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form combined embodiments. Due to space limitations, the present disclosure will not elaborate further. Those skilled in the art can understand that in the above methods of the specific implementation manner, the specific execution order of each step should be determined according to its function and possible internal logic.

[0153] In addition, the present disclosure also provides a neural network model training device, an electronic device, a computer-readable storage medium, and a program. The above can all be used to implement any one of the neural network model training methods provided by the present disclosure. The corresponding technical solutions and descriptions can be referred to the corresponding records in the method part and will not be elaborated further.

[0154] Figure 6 The block diagram showing the neural network model training device according to an embodiment of the present disclosure is as Figure 6 shown, and the device includes:

[0155] An acquisition module 61, configured to acquire a first neural network model to be trained;

[0156] An update module 62, configured to update the first network layer in the first neural network model to a second network layer to obtain a second neural network model, where the second network layer has an additional quantization simulation structure compared to the first network layer;

[0157] A training module 63, configured to train the second neural network model according to a second model optimizer to obtain a trained second neural network model;

[0158] wherein, the second model optimizer is determined according to the quantization simulation structure and an initial first model optimizer. The second model optimizer is used to adjust the update step size of the weight parameters of at least one of the second network layers. The quantization simulation structure is used to perform a simulated quantization operation on the weight parameters and adjust the quantization bits of the simulated quantization operation according to the quantization error brought by the simulated quantization operation.

[0159] In a possible implementation, the training module 63 is used to train any second network layer in the second neural network model. The training process of any second network layer in the second neural network model includes: in response to the training step reaching a preset verification step, updating the first quantization label according to the verification data set to obtain a second quantization label, where the first quantization label refers to the quantization label of the second network layer shared in the previous round of multi-step training, and the second quantization label refers to the quantization label of the second network layer shared in the current round of multi-step training; using the second model optimizer to determine the update step size of the second network layer in each step of the current round of training according to the second quantization label and the training data used in each step of the current round; updating the weight parameters of the second network layer according to the update step size of the second network layer in each step of the current round to obtain an updated second network layer; and when the preset training end condition is satisfied, determining the updated second network layer obtained in the current step of training as the trained second network layer.

[0160] In a possible implementation, updating the first quantization label according to the verification data set to obtain a second quantization label includes: determining a first result and a second result, where the first result and the second result respectively indicate verification results obtained according to the verification data in the verification data set and the weight parameters of the second network layer when the quantization simulation structure is enabled and disabled; determining the loss between the second result and the first result as a third result; and updating the quantization label of the second network layer according to the third result.

[0161] In a possible implementation, the quantization tags of the second network layer include a first tag, a second tag, and a third tag. The first tag is used to mark whether the second network layer enables or disables the quantization analog structure; the second tag is used to mark the quantization error of the analog quantization operation; the third tag is used to mark the quantization bits of the analog quantization operation. Updating the quantization tags of the second network layer according to the third result includes: in response to the difference between the third result and the second tag being greater than or equal to a first preset threshold, when the third tag is the preset maximum value, setting the first tag to a first identifier, and when the third tag is less than the preset maximum value, expanding the third tag by a first preset multiple, where the first identifier indicates that the second network layer disables the quantization analog structure to perform the analog quantization operation; in response to the difference between the third result and the second tag being less than or equal to a second preset threshold, shrinking the third tag by a second preset multiple and setting the first tag to a second identifier, where the second identifier indicates that the second network layer enables the quantization analog structure to perform the analog quantization operation; in response to the difference between the third result and the second tag being less than the first preset threshold and greater than the second preset threshold, setting the second tag to the third result.

[0162] In a possible implementation, the second model optimizer determines the update step size of the second network layer in each step of the current round of training according to the second quantization tag and the training data used in each step of the current round of training, including: when the first tag is the second identifier, the second model optimizer performs a quantization operation on the gradient of the weight parameters of the second network layer according to the third tag and the training data to obtain a quantization gradient; the second model optimizer determines the update step size of the second network layer according to the quantization gradient.

[0163] In a possible implementation, the quantization analog structure includes a first quantization parameter, a second quantization parameter, a first quantization operator, and a second quantization operator. The quantization analog structure is used to: input the weight parameters of the second network layer, the first quantization parameter, and the second quantization parameter into the first quantization operator to obtain a first quantization result; input the first quantization result, the first quantization parameter, and the second quantization parameter into the second quantization operator to obtain a second quantization result, and the second quantization result is the weight parameter after the analog quantization operation; where the first quantization parameter represents a scaling ratio, and the second quantization parameter represents a zero-point position; the first quantization operator is used to add the clamping operation result of the quotient of the weight parameters of the second network layer and the first quantization parameter to the second quantization parameter; the second quantization operator is used to multiply the difference between the first quantization result and the second quantization parameter by the first quantization parameter.

[0164] In a possible implementation, the device further includes a quantization module, configured to: quantize the trained second neural network model according to the quantization label to obtain a quantized model; input the data to be processed into the quantized model for processing to obtain a processing result, where the quantized model is a model for performing at least one task of image classification, object detection, image segmentation, speech recognition, and the data to be processed includes at least one of image data, speech data, and text data.

[0165] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0166] The embodiments of the present disclosure also propose a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above methods are implemented. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0167] The embodiments of the present disclosure also propose an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above methods.

[0168] The embodiments of the present disclosure also provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in the processor of the electronic device, the processor in the electronic device executes the above methods.

[0169] The electronic device can be provided as a terminal, a server, or other forms of devices.

[0170] Figure 7 The block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to execute the above methods.

[0171] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as the Microsoft server operating system (Windows Server TM ), the graphical user interface-based operating system launched by Apple Inc. (Mac OS X TM ), the multi-user and multi-process computer operating system (Unix TM ), the free and open-source Unix-like operating system (Linux TM ), the open-source Unix-like operating system (FreeBSD TM ) or the like.

[0172] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions, and the above computer program instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0173] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0174] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, (but is not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0175] The computer-readable program instructions described herein can be downloaded to various computing / processing devices from a computer-readable storage medium or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0176] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0177] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0178] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when the instructions are executed by the processor of the computer or other programmable data processing apparatus, an apparatus is created that implements the functions / acts specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to work in a particular manner, so that, the computer-readable medium storing the instructions comprises a manufacture, which includes instructions that implement various aspects of the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0179] The computer-readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices, so that a series of operation steps are performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other devices to implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0180] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, and the module, segment of a program, or part of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may occur in a different order than noted in the figures. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or can be implemented by a combination of dedicated hardware and computer instructions.

[0181] The computer program product can be implemented specifically in the form of hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is specifically embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0182] The descriptions of the above embodiments tend to emphasize the differences between the embodiments, and their similarities or likenesses can be referred to each other. For the sake of brevity, they will not be elaborated herein.

[0183] Those skilled in the art can understand that in the above methods of the specific embodiments, the writing order of each step does not mean a strict execution order and does not impose any limitation on the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic.

[0184] If the technical solution of this application involves personal information, before the product applying the technical solution of this application processes personal information, it has clearly informed the personal information processing rules and obtained the personal's independent consent. If the technical solution of this application involves sensitive personal information, before the product applying the technical solution of this application processes sensitive personal information, it has obtained the personal's separate consent and at the same time meets the requirements of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and personal information will be collected. If an individual voluntarily enters the collection range, it is regarded as consenting to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are informed by obvious signs / information, personal authorization is obtained through pop-up messages or by asking the individual to upload their personal information by themselves; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0185] The above has described the embodiments of the present disclosure. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein.

Claims

1. A training method for a neural network model, characterized in that: The method comprises: Obtaining a first neural network model to be trained, where the first neural network model to be trained is a neural network model for any task among image classification, image segmentation, and speech recognition; Updating the first network layer in the first neural network model to a second network layer to obtain a second neural network model, wherein the second network layer has a quantized simulation structure compared to the first network layer; According to the second model optimizer, the second neural network model is trained to obtain a trained second neural network model; The second model optimizer is determined according to the quantization simulation structure and the initial first model optimizer, the second model optimizer is used to adjust the update step size of at least one weight parameter of the second network layer, and the quantization simulation structure is used to perform a simulated quantization operation on the weight parameter, and adjust the number of quantization bits of the simulated quantization operation according to the quantization error caused by the simulated quantization operation; Among them, the quantization simulation structure includes a first quantization parameter, a second quantization parameter, a first quantization operator, and a second quantization operator. The quantization simulation structure is used to: input the weight parameter of the second network layer, the first quantization parameter, and the second quantization parameter into the first quantization operator to obtain a first quantization result; input the first quantization result, the first quantization parameter, and the second quantization parameter into the second quantization operator to obtain a second quantization result, and the second quantization result is the weight parameter after the simulated quantization operation; wherein the first quantization parameter represents the scaling ratio, and the second quantization parameter represents the zero point position; the first quantization operator is used to add the second quantization parameter to the result of the embedding operation of the weight parameter of the second network layer and the quotient of the first quantization parameter; the second quantization operator is used to multiply the difference between the first quantization result and the second quantization parameter by the first quantization parameter.

2. The method according to claim 1, characterized in that The training process of any second network layer in the second neural network model includes: In response to the number of training steps reaching a preset number of verification steps, the first quantization label is updated according to the verification data set to obtain a second quantization label; wherein the first quantization label refers to the quantization label of the second network layer shared by the previous round of multi-step training, and the second quantization label refers to the quantization label of the second network layer shared by the current round of multi-step training; Determine, using the second model optimizer according to the second quantization label and the training data used in each step of the current round of training, an update step size of the second network layer in each step of the current round of training; According to the update step size of the second network layer in each step of the current round of training, the weight parameters of the second network layer are updated to obtain an updated second network layer; When the preset training end condition is met, the updated second network layer obtained by the current step of training is determined as the trained second network layer.

3. The method according to claim 2, characterized in that The first quantitative label is updated according to the verification data set to obtain a second quantitative label, including: Determine a first result and a second result; wherein the first result and the second result respectively indicate verification results obtained according to the verification data in the verification data set and the weight parameters of the second network layer when the quantization simulation structure is enabled and disabled; determining the loss between the second result and the first result as a third result; According to the third result, the quantization label of the second network layer is updated.

4. The method according to claim 3, characterized in that The quantization tag of the second network layer includes a first tag, a second tag and a third tag, wherein the first tag is used to mark the second network layer to enable or disable the quantization simulation structure; the second tag is used to mark the quantization error of the analog quantization operation; The third tag is used to mark the number of quantization bits of the analog quantization operation; Updating the quantization label of the second network layer according to the third result includes: In response to a difference between the third result and the second label being greater than or equal to a first preset threshold, when the third label is a preset maximum value, setting the first label to a first flag, and when the third label is less than the preset maximum value, enlarging the third label by a first preset multiple, wherein the first flag indicates that the second network layer deactivates the quantization simulation structure to perform the analog quantization operation; In response to a difference between the third result and the second label being less than or equal to a second preset threshold, reducing the third label by a second preset multiple, and setting the first label to a second identifier, wherein the second identifier indicates that the second network layer enables the quantization simulation structure to perform an analog quantization operation; In response to a difference between the third result and the second label being smaller than a first preset threshold and larger than a second preset threshold, the second label is set as the third result.

5. The method according to claim 4, characterized in that Determining, using the second model optimizer according to the second quantization label and the training data used in each step of the current round of training, an update step size of the second network layer in each step of the current round of training, including: When the first label is a second identifier, the second model optimizer performs a quantization operation on the gradient of the weight parameter of the second network layer according to the third label and the training data to obtain a quantized gradient; The second model optimizer determines an update step size of the second network layer according to the quantized gradient.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: quantizing the trained second neural network model according to the quantization label to obtain a quantized model; The data to be processed is input into the quantization model for processing to obtain a processing result, wherein the quantization model is a model for performing at least one task of image classification, image segmentation, and speech recognition, and the data to be processed includes at least one of image data, speech data, and text data.

7. A neural network model device, characterized in that: The method comprises a second network layer, wherein the second network layer comprises a quantized simulation structure, wherein the neural network model is obtained by training based on a second model optimizer, and the neural network model is a neural network model for any task of image classification, image segmentation, and speech recognition; The second model optimizer is determined according to the quantization simulation structure and the initial first model optimizer, the second model optimizer is used to adjust the update step size of at least one weight parameter of the second network layer, the quantization simulation structure is used to perform a simulated quantization operation on the weight parameter, and adjust the number of quantization bits of the simulated quantization operation according to the quantization error caused by the simulated quantization operation; Among them, the quantization simulation structure includes a first quantization parameter, a second quantization parameter, a first quantization operator, and a second quantization operator. The quantization simulation structure is used to: input the weight parameter of the second network layer, the first quantization parameter, and the second quantization parameter into the first quantization operator to obtain a first quantization result; input the first quantization result, the first quantization parameter, and the second quantization parameter into the second quantization operator to obtain a second quantization result, and the second quantization result is the weight parameter after the simulated quantization operation; wherein the first quantization parameter represents the scaling ratio, and the second quantization parameter represents the zero point position; the first quantization operator is used to add the second quantization parameter to the result of the embedding operation of the weight parameter of the second network layer and the quotient of the first quantization parameter; the second quantization operator is used to multiply the difference between the first quantization result and the second quantization parameter by the first quantization parameter.

8. A training device for a neural network model, characterized in that: include: An acquisition module, used for acquiring a first neural network model to be trained, where the first neural network model to be trained is a neural network model for any task among image classification, image segmentation, and speech recognition; An updating module, used for updating the first network layer in the first neural network model to a second network layer to obtain a second neural network model, wherein the second network layer has a quantized simulation structure added compared to the first network layer; A training module, used for training the second neural network model according to the second model optimizer to obtain a trained second neural network model; The second model optimizer is determined according to the quantization simulation structure and the initial first model optimizer, the second model optimizer is used to adjust the update step size of at least one weight parameter of the second network layer, and the quantization simulation structure is used to perform a simulated quantization operation on the weight parameter, and adjust the number of quantization bits of the simulated quantization operation according to the quantization error caused by the simulated quantization operation; Among them, the quantization simulation structure includes a first quantization parameter, a second quantization parameter, a first quantization operator, and a second quantization operator. The quantization simulation structure is used to: input the weight parameter of the second network layer, the first quantization parameter, and the second quantization parameter into the first quantization operator to obtain a first quantization result; input the first quantization result, the first quantization parameter, and the second quantization parameter into the second quantization operator to obtain a second quantization result, and the second quantization result is the weight parameter after the simulated quantization operation; wherein the first quantization parameter represents the scaling ratio, and the second quantization parameter represents the zero point position; the first quantization operator is used to add the second quantization parameter to the result of the embedding operation of the weight parameter of the second network layer and the quotient of the first quantization parameter; the second quantization operator is used to multiply the difference between the first quantization result and the second quantization parameter by the first quantization parameter.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

11. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images