A method and device for determining weight parameters of a neural network model
By introducing correction values into the neural network model to correct errors and optimize weight parameters, the problems of quantization error and overfitting are solved, improving computational accuracy and training efficiency while reducing memory and energy consumption.
Patent Information
- Application Number
- CN201880092139.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-06-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2038-06-15
AI Technical Summary
Existing neural network models suffer from quantization errors and overfitting during the quantization process, leading to decreased accuracy of calculation results and the inability of gradients to effectively adjust weights during training.
By introducing correction values to correct the original error values, calculating the corrected error values, adjusting the undetermined weight parameters, and using forward and backward propagation algorithms to optimize the model weight parameters, the inference results are avoided from being dominated by some weight parameters with large values.
Reducing quantization error and avoiding overfitting improves the computational accuracy and training efficiency of the model, while reducing memory bandwidth and energy consumption.
Smart Images

Figure CN111937011B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method and device for determining weight parameters of a neural network model. Background Art
[0002] In recent years, neural network models have shown extremely superior performance in applications such as computer vision and speech processing, and have attracted widespread attention. The success of neural network models comes at the cost of introducing a large number of parameters and calculations, and the quantization technology of model parameters related to neural network models can reduce the redundancy of the accuracy of related model parameters, and achieve the purpose of model compression while reducing the adverse effects on model accuracy.
[0003] Model compression can not only reduce memory bandwidth usage and data access energy consumption, but low-precision calculations often also bring lower computing energy consumption. For some computing units that support multiple precision calculations, the number of low-precision calculations completed per unit time is higher than the number of high-precision calculations that can be completed. Summary of the invention
[0004] The embodiments of the present application provide a method and device for determining the weight parameters of a neural network model. In various data processing scenarios in which the neural network model is applied, such as image recognition, speech recognition, image super-resolution processing, etc., the method can reduce the quantization error and avoid the problem of overfitting caused by some weight parameters with large values dominating the reasoning results of the neural network by introducing appropriate correction values into the error between the output results of model training and the expected results.
[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:
[0006] In a first aspect, an embodiment of the present application provides a method for determining weight parameters of a neural network model, comprising: processing sample data based on the undetermined weight parameters of the neural network model to obtain an output result; calculating an original error value between the output result and a preset expected result, wherein the original error value is a numerical representation of the difference between the output result and the expected result; based on a correction value, correcting the original error value to obtain a corrected error value; based on the corrected error value and the undetermined weight parameters, determining the model weight parameters of the neural network model; wherein the correction value is obtained according to the following formula:
[0007] R=(w k -Q(w k ))×Q(w k )
[0008] R represents the correction value, w k represents the kth undetermined weight parameter of the neural network model, Q(wk ) represents the quantized value of the kth undetermined weight parameter, and k is a non-negative integer.
[0009] The embodiment of the present application reduces the quantization error and avoids the problem of overfitting caused by some weight parameters with large values dominating the reasoning results of the neural network by introducing appropriate correction values into the error between the output result of model training and the expected result.
[0010] In a first feasible implementation manner of the first aspect, the corrected error value is obtained according to the following formula:
[0011]
[0012] Wherein, E1 represents the corrected error value, E0 represents the original error value, α is a constant, m is the total number of undetermined weight parameters used to process the sample data, F((w k -Q(w k ))×Q(w k )) represents a function with the correction value as the independent variable, and m is a positive integer.
[0013] In a second feasible implementation manner of the first aspect, the function with the correction value as an independent variable calculates the absolute value of the correction value; correspondingly, the correction error value is obtained according to the following formula:
[0014]
[0015] Among them, |(w k -Q(w k ))×Q(w k )| indicates the calculation (w k -Q(w k ))×Q(w k )’s absolute value.
[0016] In a third feasible implementation manner of the first aspect, the neural network model includes p network layers, each of the network layers includes q undetermined weight parameters, and the kth undetermined weight parameter is the jth undetermined weight parameter of the i-th network layer in the neural network model; correspondingly, the corrected error value is obtained according to the following formula:
[0017]
[0018] Among them, p and q are positive integers, and i and j are non-negative integers.
[0019] The above-mentioned feasible implementation methods of the embodiments of the present application exemplarily give different calculation methods and forms of correction values to correct the error values of the training results, thereby achieving the effect of reducing quantization errors and avoiding the problem of overfitting caused by some weight parameters with larger values dominating the reasoning results of the neural network.
[0020] It should be understood that in some neural network models, some network layers do not have undetermined weight parameters. Obviously, the network layers without undetermined weight parameters cannot participate in the calculation of the corrected error value in the above formula.
[0021] In a fourth feasible implementation of the first aspect, the pending weight parameters based on the neural network model are used to process the sample data, including: obtaining the pending weight parameters; quantizing the obtained pending weight parameters to obtain quantized weight parameters, wherein the quantized weight parameters are quantized values of the pending weight parameters; using the quantized weight parameters as model weight parameters of the neural network model, and adopting a forward propagation algorithm to process the sample data; and obtaining the output result from the output layer of the neural network model.
[0022] It should be understood that there may be multiple forward propagation algorithms mentioned in the embodiments of the present application, and the embodiments of the present invention do not limit this. Through the forward propagation algorithm, the input sample data is deduced based on the neural network model to obtain the output result.
[0023] In a fifth feasible implementation of the first aspect, the model weight parameters of the neural network model are obtained by iterative training. When the iterative training meets the termination condition, the model weight parameters of the neural network model are determined based on the corrected error value and the pending weight parameters, including: using the quantized weight parameters as the model weight parameters of the neural network model.
[0024] The embodiment of the present application achieves compression of the neural network model by quantizing the weight parameters, reduces memory bandwidth occupancy and energy consumption, and improves the computing efficiency of the processor.
[0025] In a sixth feasible implementation of the first aspect, when the iterative training does not meet the termination condition, determining the model weight parameters of the neural network model based on the corrected error value and the pending weight parameters includes: according to the corrected error value, using a back propagation algorithm to adjust the pending weight parameters of the network layers of the neural network model layer by layer until the input layer of the neural network model to obtain the adjusted weight parameters of the neural network model.
[0026] In a seventh feasible implementation manner of the first aspect, the undetermined weight parameters of the neural network model are adjusted according to the following formula:
[0027]
[0028] Among them, w0 k represents the kth undetermined weight parameter, w1 k represents the kth adjusted weight parameter, and β is a positive constant.
[0029] It should be understood that there may be multiple back propagation algorithms, also known as back propagation algorithms, mentioned in the embodiments of the present application, which are not limited in the embodiments of the present invention. Through the back propagation algorithm, the weight parameters are trained, and the updated weight parameters further optimize the neural network model.
[0030] In an eighth feasible implementation of the first aspect, for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the termination condition includes a combination of one or more of the following conditions: the original error value in the Nth training cycle is less than a preset first threshold; the corrected error value in the Nth training cycle is less than a preset second threshold; the difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold; the difference between the corrected error value in the Nth training cycle and the corrected error value in the NMth training cycle is less than a preset fourth threshold; the difference between the pending weight parameter in the Nth training cycle and the pending weight parameter in the NMth training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold.
[0031] In a ninth feasible implementation of the first aspect, when the Nth training cycle does not meet the termination condition, a combination of one or more of the following physical quantities is stored: the original error value in the Nth training cycle; the corrected error value in the Nth training cycle; the pending weight parameter in the Nth training cycle; and the cycle number N of the Nth training cycle.
[0032] In the embodiment of the present application, by reasonably setting the end conditions of training, the efficiency of training is improved, and a balance is achieved between the training effect and the resources consumed by training.
[0033] In a tenth feasible implementation of the first aspect, obtaining the pending weight parameters includes: in the first training cycle of the iterative training, using the preset initial weight parameters as the pending weight parameters; in a training cycle other than the first training cycle of the iterative training, using the adjusted weight parameters of the neural network model as the pending weight parameters.
[0034] In an eleventh feasible implementation of the first aspect, the neural network model is used for image recognition; correspondingly, the sample data includes image samples; correspondingly, the output result includes a recognition result of the image recognition represented in a probabilistic form.
[0035] In a twelfth feasible implementation of the first aspect, the neural network model is used for sound recognition; correspondingly, the sample data includes sound samples; correspondingly, the output result includes a recognition result of the sound recognition represented in a probabilistic form.
[0036] In a thirteenth feasible implementation manner of the first aspect, the neural network model is used to acquire super-resolution images; correspondingly, the sample data includes image samples; correspondingly, the output result includes pixel values of the image after super-resolution processing.
[0037] The above-mentioned feasible implementation methods of the embodiments of the present application exemplarily give the specific application scenarios of the neural network model in the embodiments of the present application. Through the application of the neural network model, the recognition rate of image recognition and sound recognition can be improved, the image quality of image super-resolution processing can be improved, and significant beneficial effects can also be achieved in other application fields.
[0038] In a second aspect, an embodiment of the present application provides a device for determining weight parameters of a neural network model, characterized in that it includes: a forward propagation module, which is used to process sample data based on the undetermined weight parameters of the neural network model to obtain an output result; a comparison module, which is used to calculate the original error value between the output result and the preset expected result, and the original error value is a numerical representation of the difference between the output result and the expected result; a correction module, which is used to correct the original error value based on the correction value to obtain a corrected error value; a determination module, which is used to determine the model weight parameters of the neural network model based on the corrected error value and the undetermined weight parameters; wherein the correction value is obtained according to the following formula:
[0039] R=(w k -Q(w k ))×Q(w k )
[0040] R represents the correction value, w k represents the kth undetermined weight parameter of the neural network model, Q(w k ) represents the quantized value of the kth undetermined weight parameter, and k is a non-negative integer.
[0041] In a first feasible implementation manner of the second aspect, the corrected error value is obtained according to the following formula:
[0042]
[0043] Wherein, E1 represents the corrected error value, E0 represents the original error value, α is a constant, m is the total number of undetermined weight parameters used to process the sample data, F((w k -Q(w k ))×Q(w k )) represents a function with the correction value as the independent variable, and m is a positive integer.
[0044] In a second feasible implementation manner of the second aspect, the function with the correction value as an independent variable calculates the absolute value of the correction value; correspondingly, the correction error value is obtained according to the following formula:
[0045]
[0046] Among them, |(w k -Q(w k ))×Q(w k )| indicates the calculation (w k -Q(w k ))×Q(w k )’s absolute value.
[0047] In a third feasible implementation manner of the second aspect, the neural network model includes p network layers, each of the network layers includes q undetermined weight parameters, and the kth undetermined weight parameter is the jth undetermined weight parameter of the i-th network layer in the neural network model; correspondingly, the corrected error value is obtained according to the following formula:
[0048]
[0049] Among them, p and q are positive integers, and i and j are non-negative integers.
[0050] In a fourth feasible implementation of the second aspect, the forward propagation module is specifically used to: obtain the pending weight parameters; quantize the obtained pending weight parameters to obtain quantized weight parameters, and the quantized weight parameters are quantized values of the pending weight parameters; use the quantized weight parameters as model weight parameters of the neural network model, and adopt a forward propagation algorithm to process the sample data; obtain the output result from the output layer of the neural network model.
[0051] In a fifth feasible implementation of the second aspect, the model weight parameters of the neural network model are obtained by iterative training. When the iterative training meets the termination condition, the determination module is specifically used to: use the quantized weight parameters as the model weight parameters of the neural network model.
[0052] In a sixth feasible implementation of the second aspect, a back propagation module is further included. When the iterative training does not meet the termination condition, the back propagation module is specifically used to: according to the corrected error value, use a back propagation algorithm to adjust the pending weight parameters of the network layer of the neural network model layer by layer until the input layer of the neural network model, so as to obtain the adjusted weight parameters of the neural network model.
[0053] In a seventh feasible implementation manner of the second aspect, the undetermined weight parameters of the neural network model are adjusted according to the following formula:
[0054]
[0055] Among them, w0 k represents the kth undetermined weight parameter, w1 k represents the kth adjusted weight parameter, and β is a positive constant.
[0056] In an eighth feasible implementation of the second aspect, for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the termination condition includes a combination of one or more of the following conditions: the original error value in the Nth training cycle is less than a preset first threshold; the corrected error value in the Nth training cycle is less than a preset second threshold; the difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold; the difference between the corrected error value in the Nth training cycle and the corrected error value in the NMth training cycle is less than a preset fourth threshold; the difference between the pending weight parameter in the Nth training cycle and the pending weight parameter in the NMth training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold.
[0057] In a ninth feasible implementation of the second aspect, when the Nth training cycle does not meet the termination condition, a combination of one or more of the following physical quantities is stored: the original error value in the Nth training cycle; the corrected error value in the Nth training cycle; the pending weight parameter in the Nth training cycle; and the number of cycles N of the Nth training cycle.
[0058] In a tenth feasible implementation of the second aspect, the forward propagation module is specifically used to: in the first training cycle of the iterative training, use the preset initial weight parameters as the pending weight parameters; in the non-first training cycle of the iterative training, use the adjusted weight parameters of the neural network model as the pending weight parameters.
[0059] In an eleventh feasible implementation of the second aspect, the neural network model is used for image recognition; correspondingly, the sample data includes image samples; correspondingly, the output result includes a recognition result of the image recognition represented in a probabilistic form.
[0060] In a twelfth feasible implementation manner of the second aspect, the neural network model is used for sound recognition; correspondingly, the sample data includes sound samples; correspondingly, the output result includes a recognition result of the sound recognition represented in a probabilistic form.
[0061] In a thirteenth feasible implementation manner of the second aspect, the neural network model is used to acquire super-resolution images; correspondingly, the sample data includes image samples; correspondingly, the output result includes pixel values of the image after super-resolution processing.
[0062] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method for determining the weight parameters of the neural network model as described in any one of the first aspects.
[0063] In a fourth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes a data processing method as described in any one of the first aspects.
[0064] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method for determining weight parameters of a neural network model as described in any one of the first aspects.
[0065] In a sixth aspect, an embodiment of the present application provides a chip, including a processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, and when the processor executes the computer instructions, the electronic device executes a method for determining weight parameters of a neural network model as described in any one of the first aspects.
[0066] Among them, the beneficial effects of the second to sixth aspects mentioned above can be found in the description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 is a schematic diagram of an exemplary neural network structure;
[0068] Figure 2 is a schematic diagram of an exemplary structure of a neuron;
[0069] Figure 3 is another exemplary schematic diagram of a neural network structure;
[0070] FIG4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0071] Figure 5 An exemplary flow chart of a method for determining weight parameters of a neural network model provided in an embodiment of the present application;
[0072] Figure 6 An exemplary structural block diagram of a device for determining weight parameters of a neural network model provided in an embodiment of the present application;
[0073] Figure 7 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0074] Neural networks (also called deep neural networks) can be used to process various data, such as image data, audio data, etc. Neural networks can include one or more network layers (also called neural network layers), which can be convolutional layers, fully connected layers, deconvolutional layers, or recurrent layers. A typical neural network model is as follows: Figure 1 shown.
[0075] In order to facilitate understanding of the embodiments of the present invention, some concepts related to the embodiments of the present application are provided for reference by way of example.
[0076] (1) Neural network model and forward propagation: For the training sample set (x( i ), y( i ), the neural network algorithm can provide a complex and nonlinear hypothesis model h W,b (x), it has parameters W, b, which can be used to fit the data. In order to describe the neural network, we start with the simplest neural network model, which is also referred to as a neural network in this article. This neural network consists of only one "neuron". Figure 2 This is a diagram of this "neuron".
[0077] This "neuron" is an arithmetic unit with x1, x2, x3 and intercept +1 as input values, and its output is The function f is: It is called the "activation function".
[0078] A neural network is a network of neurons connected together so that the output of one neuron can be the input of another neuron. Figure 3It is a simple neural network.
[0079] Circles are used to represent the input of the neural network. The circle marked with "+1" is called the bias node, which is the intercept term. The leftmost layer of the neural network is called the input layer, and the rightmost layer is called the output layer (in this example, the output layer has only one node). The layer composed of all the nodes in the middle is called the hidden layer (in other embodiments, the hidden layer may not exist, or there may be multiple layers). At the same time, it can be seen that in the above neural network example, there are 3 input units (excluding the bias unit), 3 hidden units and one output unit.
[0080] Might as well use n l To indicate the number of layers in the network, in this case n l =3, we denote the lth layer as L l , so L1 is the input layer and the output layer is In this example, the neural network has parameters (W, b) = (W (1) , b (1) , W (2) , b (2) ),in is the connection parameter between the jth unit in the lth layer and the ith unit in the l+1th layer (actually the weight on the connection line), is the bias term of the i-th unit in the l+1th layer. No other units are connected to the bias unit (i.e., the bias unit has no input) because they always output +1. At the same time, use s l Indicates the number of nodes in the lth layer (bias units are not included). represents the activation value (output value) of the i-th unit in the l-th layer. When l = 1, That is, the i-th input value (the i-th feature of the input value). For a given parameter set W, b, the neural network can be calculated according to the function h W,b (x) to calculate the output result. The calculation steps of the neural network in this example are as follows:
[0081]
[0082]
[0083]
[0084] Might as well use represents the weighted sum of the inputs of the i-th unit in the l-th layer (including the bias unit), for example, but
[0085] The above calculation step is called forward propagation.
[0086] (2) Back propagation algorithm (also known as back propagation algorithm) and error back propagation algorithm: The back propagation algorithm mainly consists of two steps (excitation propagation and weight update) that are repeatedly iterated until the network's response to the input reaches the predetermined target range.
[0087] The learning process of the error back propagation algorithm consists of the forward propagation process and the backward propagation process. In the forward propagation process, the input information passes through the input layer and the hidden layer, and is processed layer by layer and transmitted to the output layer. If the expected output value cannot be obtained in the output layer, various representation forms of the error between the output and the expected value (for example, the sum of squares) are taken as the objective function, and the back propagation is turned to. The partial derivatives of the objective function with respect to the weights of each neuron are obtained layer by layer, and the gradient of the objective function with respect to the weight vector is formed as the basis for modifying the weights. The learning of the network is completed in the process of modifying the weights. When the error reaches the expected value, the network learning ends.
[0088] In the excitation propagation step, the propagation link in each iteration includes two steps: sending the training input into the network to obtain the excitation response (forward propagation stage); subtracting the excitation response from the target output corresponding to the training input to obtain the response error of the hidden layer and the output layer (backward propagation stage).
[0089] In the weight update step, the weights on each neuron are updated according to the following steps: multiply the input stimulus and the response error to obtain the gradient of the weight; multiply this gradient by a ratio and negate it before adding it to the weight. This ratio will affect the speed and effect of the training process, so it is called the "training factor". The direction of the gradient indicates the direction in which the error is amplified, so it needs to be negated when updating the weight to reduce the error caused by the weight.
[0090] Matt Mazur exemplarily introduces an implementation method of an error backpropagation algorithm in “A Step by Step Backpropagation Example” (which can be downloaded from https: / / mattmazur.com / 2015 / 03 / 17 / a-step-by-step-backpropagation-Example / , and the entire text is cited in this article), which can be applied to the embodiments of the present application and will not be described in detail.
[0091] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0092] The data processing device involved in the embodiment of the present application is an electronic device that uses a convolutional neural network to process image, voice and other data, for example, it can be a server or a terminal. Exemplarily, when the electronic device is a terminal, the electronic device can specifically be a desktop computer, a portable computer, a PDA (personal digital assistant), a tablet computer, an embedded device, a mobile phone, a smart peripheral (such as a smart watch, a bracelet, glasses, etc.), a TV set-top box, a surveillance camera, etc. The embodiment of the present application does not limit the specific type of electronic device.
[0093] 4 shows a schematic diagram of the hardware structure of an electronic device 400 according to an embodiment of the present application. The electronic device 400 may include at least one processor 401, a communication bus 402 and a memory 403. The electronic device 400 may also include at least one communication interface 404.
[0094] Processor 401 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a field programmable gate array (FPGA), or one or more integrated circuits used to control the execution of the program of the present application.
[0095] Communication bus 402 may include a pathway for transmitting information between the above-mentioned components.
[0096] The communication interface 404 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0097] The memory 403 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.
[0098] Among them, the memory 403 is used to store the application code for executing the solution provided in the embodiment of the present application, as well as the neural network model structure, weights, and intermediate results of the processor 401 operation using the solution provided in the embodiment of the present application, and is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 203, thereby realizing the data processing method provided in the following embodiment of the present application.
[0099] In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 4 .
[0100] In a specific implementation, as an embodiment, the electronic device 400 may include multiple processors, such as the processor 401 and the processor 407 in FIG. 4. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0101] In a specific implementation, as an embodiment, the electronic device 400 may further include an output device 405 and an input device 406. The output device 405 communicates with the processor 401 and may display information in a variety of ways. For example, the output device 405 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 406 communicates with the processor 401 and may receive user input in a variety of ways. For example, the input device 406 may be a mouse, a keyboard, a camera, a microphone, a touch screen device, or a sensor device.
[0102] As mentioned above, a neural network is composed of network layers, each of which processes its input data and passes it to the next network layer. In the network layer, different weights are used to perform convolution, multiplication and addition operations on the input data according to the different properties of the network layer (such as convolution layer, fully connected layer, etc.). The methods of these operations are determined by the properties of the network layer, but the values of the weights used for different operations are obtained through training. Different data processing results can be obtained by adjusting the weight values.
[0103] In the process of training model weight parameters, excessive emphasis on data fitting of the training data set may lead to poor generalization, which is manifested as: the model fits the data well on the training data set with high accuracy, but cannot fit the data well on data sets outside the training data set, the effect is poor, and the accuracy is seriously reduced.
[0104] The precision of neural network weight parameters is redundant. Using low-precision data formats (such as INT8, binary, etc.) to record weight parameters instead of high-precision data formats (such as FP32, FP64) can achieve compression of weight parameters.
[0105] Quantizing weight parameters is a feasible implementation method to achieve weight parameter compression. Compressing the accuracy of neural network parameters can achieve a balance between storage, energy consumption, and accuracy. The data that can be quantized include: weights, feature tensors (activations), gradients, and other parameters of the neural network model, without limitation.
[0106] In a feasible implementation, firstly, the weight coefficients of the neural network with high precision (non-quantized, generally FP32 or FP64 precision) are obtained through training, and then the weight coefficients expressed in the high-precision data format are expressed in a low-precision data format. At this time, since the low-precision data format cannot express the details of the high-precision data, it will lead to numerical differences. The differences are accumulated many times in the calculation, and eventually the accuracy of the final calculation result of the quantized model is lower than that of the original model before quantization. In order to solve the above problems, the quantized model parameters are generally trained again under the premise of ensuring their low-precision data format, and the weight values are adjusted through retraining, and finally a model accuracy close to high precision is achieved under low precision.
[0107] Specifically, during the training process, the sample data is input into the neural network, the difference with the expected data is calculated, and then the difference is used to calculate the gradient of all weights to be adjusted in the neural network (that is, the tendency of the weights to be adjusted), and the weights in the neural network are adjusted to narrow the difference, that is, to achieve higher accuracy. However, for quantized neural networks, since the gradient value of adjusting the weights during training is extremely small, there is a high probability that it is less than the minimum value of the interval that can be expressed under the weight accuracy, resulting in the gradient being unable to actually adjust the value of the weight. For example, in the INT4 data format, the value range is defined as 0 to 1, and the minimum value of the expressible interval is 2. -4 , however, the gradient is likely to be much smaller than this value, assuming it is 2 -6 At this time, any value A under INT4, after summing with the gradient, the result should be A+2 -6 However, in the INT4 data format, the minimum interval cannot express 2 -6 , the actual result is still A. Therefore, the quantized weights cannot be used directly for training.
[0108] In order to solve the above problems, a common training method is to use a high-precision reference weight to carry the unquantized weight information. The quantized weight is obtained from the reference weight, and the error calculation is carried out with the participation of the quantized weight. The gradient of the error is applied to the unquantized weight (also called the reference weight). When the reference weight is adjusted cumulatively by more than a certain amount, it can be expressed on the quantized weight. (For example, the quantized weight can only express integers from 0 to 255, and the reference weight is initially 1. During the adjustment process, the value fluctuates slightly until the reference weight is greater than 1.5, the quantized weight becomes 2, and until the reference weight is less than 0.5, the quantized weight becomes 0, otherwise the quantized weight is 1).
[0109] The current technical problems brought about by the above method include: Unlike non-quantized neural networks, quantized neural networks use quantized weights instead of reference weights in the calculation process. The difference between the reference weights and the quantized weights accumulates in the neural network calculation, which ultimately affects the calculation results and generates quantization errors. The gradient used to adjust the reference weight is derived from the inference error obtained by calculating the derivation result using the above quantized weights, which is not accurate enough and has gradient errors. The above problems will make it difficult for model training to reach the global optimal solution.
[0110] The present application embodiment provides a method for determining weight parameters of a neural network model, such as Figure 5 As shown, specifically including:
[0111] S501. Based on the undetermined weight parameters of the neural network model, the sample data is processed to obtain an output result.
[0112] Specifically, in a feasible implementation manner, this step includes:
[0113] S5011. Obtain the undetermined weight parameters.
[0114] It may be assumed that the model weight parameters of the neural network model in the embodiment of the present application are obtained by iterative training.
[0115] When entering training for the first time, that is, in the first training cycle of the iterative training, the execution of step S5011 includes obtaining default initial values, such as constants such as 0, 1, etc., and assigning them to the pending weight parameters. It is also possible to obtain predetermined values based on empirical values and assign them to the pending weight parameters, such as the stored weight parameters of a pre-trained neural network model.
[0116] When in the iterative training process, that is, when it is not the first training cycle of the iterative training, the execution of step S5011 includes obtaining the pending weight parameters (i.e., the adjusted weight parameters) updated by the back propagation algorithm in the previous training cycle as the pending weight parameters obtained in this step. The specific implementation method will be described in detail in the subsequent steps, and this step will not be repeated.
[0117] S5012: Quantize the obtained undetermined weight parameter to obtain a quantized weight parameter, where the quantized weight parameter is a quantized value of the undetermined weight parameter.
[0118] Different preset quantization schemes may be used to quantize the pending weight parameters, such as converting the high-precision expression of the weight parameters into a low-precision expression as described above, which is not limited in this step.
[0119] S5013. Use the quantized weight parameters as model weight parameters of the neural network model, and adopt a forward propagation algorithm to process the sample data.
[0120] The forward propagation algorithm was introduced in the previous article. In this step, the quantized undetermined weight parameters are used as model parameters of the neural network model, and the sample data is used as input and the forward propagation algorithm is used as the calculation criterion for calculation. It should be understood that this step does not limit the specific forward propagation algorithm.
[0121] S5014. Obtain the output result from the output layer of the neural network model.
[0122] Outputting the calculation result for the sample data from the output layer of the neural network model. Exemplarily, if the neural network model is used for image recognition, the sample data includes image samples, and the output result includes the recognition result of the image recognition represented in a probability form, specifically, the probability of judging that the sample image is the target image is 90%. If the neural network model is used for sound recognition, the sample data includes sound samples, and the output result includes the recognition result of the sound recognition represented in a probability form, specifically, the probability of judging that the sample sound is the target sound is 20%. . If the neural network model is used for obtaining super-resolution images, the sample data includes image samples, and the output result includes the pixel values of the image after super-resolution processing.
[0123] It should be understood that the neural network model can also be used in other applications related to the field of artificial intelligence. Correspondingly, the sample data used as input data and the output results used as output data can also be other types of physical quantities without limitation.
[0124] S502: Calculate an original error value between the output result and a preset expected result, where the original error value is a numerical representation of the difference between the output result and the expected result.
[0125] Corresponding to step S5014, in this step, the difference between the expected output result, i.e., the expected result, and the actual output result is calculated, and the difference is represented in a numerical form. Exemplarily, the difference can be the difference in the recognition result, for example, if the recognition result is 90% and the expected result is 100%, the original error value is 10%, or it can be the pixel difference between the original image before downsampling corresponding to the sample image and the image after super-resolution processing of the sample image, for example, it can be represented by the signal peak signal-to-noise ratio (PSNR) between the two, such as -0.2 decibels (dB), or the variance value between the pixels of the two images, etc., depending on the specific application of the neural network model, without limitation.
[0126] S503: Based on the correction value, correct the original error value to obtain a corrected error value.
[0127] In the step, a correction value is first obtained. In a feasible implementation manner, the correction value is obtained according to the following formula:
[0128] R=(w k -Q(w k ))×Q(w k )
[0129] R represents the correction value, w k represents the kth undetermined weight parameter of the neural network model, Q(w k ) represents the quantized value of the kth undetermined weight parameter, and k is a non-negative integer.
[0130] Correspondingly, the corrected error value is obtained according to the following formula:
[0131]
[0132] Wherein, E1 represents the corrected error value, E0 represents the original error value, α is a constant, m is the total number of undetermined weight parameters used to process the sample data, F((w k -Q(w k ))×Q(w k )) represents a function with the correction value as the independent variable, and m is a positive integer.
[0133] In a feasible implementation manner, the function with the correction value as an independent variable is to calculate the absolute value of the correction value;
[0134] Correspondingly, the corrected error value is obtained according to the following formula:
[0135]
[0136] Among them, |(w k -Q(w k ))×Q(w k )| indicates the calculation (w k -Q(w k ))×Q(w k )’s absolute value.
[0137] In another feasible implementation, the neural network model includes p network layers, each of the network layers includes q undetermined weight parameters, and the kth undetermined weight parameter is the jth undetermined weight parameter of the i-th network layer in the neural network model;
[0138] Correspondingly, the corrected error value is obtained according to the following formula:
[0139]
[0140] Among them, p and q are positive integers, and i and j are non-negative integers.
[0141] It should be understood that in some feasible implementations, a certain network layer of the neural network model may not contain undetermined weight parameters, that is, the q corresponding to the network layer is 0 at this time, and obviously the network layer will not be used to correct the calculation of the error value.
[0142] In the embodiment of the present application, the difference between the weight parameter and the quantized weight parameter is used as a penalty term by using a correction value (regularization function), and the non-quantized weight parameter is guided to approach its quantized weight during the training process to reduce the quantization error. At the same time, the difference as the penalty term is multiplied by the quantized weight parameter to avoid the overfitting problem caused by some weights with larger values dominating the reasoning results of the neural network.
[0143] Next, based on the corrected error value and the undetermined weight parameters, the model weight parameters of the neural network model are determined.
[0144] S504: Determine whether the iterative training meets the end condition.
[0145] It may be assumed that step S504 is performed in the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the end condition includes a combination of one or more of the following conditions:
[0146] The original error value in the Nth training cycle is less than a preset first threshold;
[0147] The corrected error value in the Nth training cycle is less than a preset second threshold;
[0148] The difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold;
[0149] The difference between the corrected error value in the Nth training cycle and the corrected error value in the NMth training cycle is less than a preset fourth threshold;
[0150] The difference between the undetermined weight parameter in the Nth training cycle and the undetermined weight parameter in the NMth training cycle is less than a preset fifth threshold; and
[0151] N is greater than a preset sixth threshold.
[0152] It should be understood that when M is 1, it means that in the relevant end condition, it is necessary to compare the difference values of the corresponding physical quantities in the two most adjacent training cycles.
[0153] It should also be understood that step S504 may be executed in each training cycle or in every M training cycles, and the embodiment of the present application does not limit the execution frequency of step S504.
[0154] A training cycle can be understood as a process of calculating and correcting the error value, adjusting the weight parameters according to the corrected error value, and then using the adjusted weight parameters to obtain new training results.
[0155] Corresponding to step S504, when the Nth training cycle does not meet the end condition, one or more combinations of the following physical quantities are stored:
[0156] The original error value in the Nth training cycle;
[0157] The corrected error value in the Nth training cycle;
[0158] the undetermined weight parameters in the Nth training cycle; and
[0159] The number of cycles N of the Nth training cycle.
[0160] The stored physical quantity will be called when step S504 is subsequently executed.
[0161] S505: When the iterative training meets the termination condition, the quantized weight parameters are used as model weight parameters of the neural network model.
[0162] Generally, the end of iterative training means that the quantized weight parameters have been optimized to the desired degree through training, that is, they can be determined as the model weight parameters of the neural network model.
[0163] In some feasible implementations, the model is trained using a training sample set A, and the model is tested using a test sample set B. After the model is trained using A for N training cycles, the model is tested on the test data set B to obtain a first test result X; after the model is continuously trained using A for M training cycles, the model is tested on the test data set B to obtain a second test result Y. When the difference between X and Y is less than a threshold, the training is terminated, otherwise A is continued to be used to train the model.
[0164] Correspondingly, in this embodiment, the termination condition includes that the difference between the first test result X and the second test result Y is less than a threshold.
[0165] S506. When the iterative training does not meet the termination condition, a back propagation algorithm is used to adjust the undetermined weight parameters of the network layers of the neural network model layer by layer according to the corrected error value until the input layer of the neural network model is reached to obtain the adjusted weight parameters of the neural network model.
[0166] The back propagation algorithm has been introduced above and will not be described here. Exemplarily, the undetermined weight parameters of the neural network model are adjusted according to the following formula:
[0167]
[0168] Among them, w0 k represents the kth undetermined weight parameter, w1 k represents the kth adjusted weight parameter, and β is a positive constant.
[0169] After the weights of the input layer are adjusted, all weights of the neural network model have been obtained, that is, the adjusted weight parameters of the neural network model are obtained.
[0170] Thereafter, the adjusted weight parameters are used as parameters to be quantified, and step S5011 is executed to continue iterative training.
[0171] The embodiments of the present application provide a method and device for determining the weight parameters of a neural network model. In various data processing scenarios in which the neural network model is applied, such as image recognition, speech recognition, image super-resolution processing, etc., the method can reduce the quantization error and avoid the problem of overfitting caused by some weight parameters with large values dominating the reasoning results of the neural network by introducing appropriate correction values into the error between the output results of model training and the expected results.
[0172] It is understandable that, in order to execute the above method, the electronic device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0173] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0174] In the case of dividing each functional module into corresponding functional modules, Figure 6 A possible schematic diagram of the composition of the electronic device involved in the above embodiment is shown.
[0175] A device 600 for determining weight parameters of a neural network model, comprising:
[0176] A forward propagation module 601 is used to process the sample data based on the undetermined weight parameters of the neural network model to obtain an output result;
[0177] A comparison module 602 is used to calculate an original error value between the output result and a preset expected result, wherein the original error value is a numerical representation of the difference between the output result and the expected result;
[0178] A correction module 603, configured to correct the original error value based on the correction value to obtain a corrected error value;
[0179] A determination module 604 is used to determine the model weight parameters of the neural network model based on the corrected error value and the undetermined weight parameters;
[0180] The correction value is obtained according to the following formula:
[0181] R=(w k -Q(w k ))×Q(w k )
[0182] R represents the correction value, w k represents the kth undetermined weight parameter of the neural network model, Q(w k ) represents the quantized value of the kth undetermined weight parameter, and k is a non-negative integer.
[0183] In a feasible implementation manner, the corrected error value is obtained according to the following formula:
[0184]
[0185] Wherein, E1 represents the corrected error value, E0 represents the original error value, α is a constant, m is the total number of undetermined weight parameters used to process the sample data, F((w k -Q(w k ))×Q(w k )) represents a function with the correction value as the independent variable, and m is a positive integer.
[0186] In a feasible implementation manner, the function with the correction value as an independent variable is to calculate the absolute value of the correction value; correspondingly, the correction error value is obtained according to the following formula:
[0187]
[0188] Among them, |(w k -Q(w k ))×Q(w k )| indicates the calculation (w k -Q(w k ))×Q(w k )’s absolute value.
[0189] In a feasible implementation manner, the neural network model includes p network layers, each of the network layers includes q undetermined weight parameters, and the kth undetermined weight parameter is the jth undetermined weight parameter of the i-th network layer in the neural network model; correspondingly, the corrected error value is obtained according to the following formula:
[0190]
[0191] Among them, p and q are positive integers, and i and j are non-negative integers.
[0192] In a feasible implementation, the forward propagation module 601 is specifically used to: obtain the pending weight parameters; quantize the obtained pending weight parameters to obtain quantized weight parameters, and the quantized weight parameters are the quantized values of the pending weight parameters; use the quantized weight parameters as model weight parameters of the neural network model, and adopt a forward propagation algorithm to process the sample data; and obtain the output result from the output layer of the neural network model.
[0193] In a feasible implementation, the model weight parameters of the neural network model are obtained by iterative training. When the iterative training meets the termination condition, the determination module 604 is specifically used to: use the quantized weight parameters as the model weight parameters of the neural network model.
[0194] In a feasible implementation, it also includes a back propagation module 605. When the iterative training does not meet the termination condition, the back propagation module 605 is specifically used to: according to the corrected error value, use a back propagation algorithm to adjust the pending weight parameters of the network layer of the neural network model layer by layer until the input layer of the neural network model, so as to obtain the adjusted weight parameters of the neural network model.
[0195] In a feasible implementation manner, the undetermined weight parameters of the neural network model are adjusted according to the following formula:
[0196]
[0197] Among them, w0 k represents the kth undetermined weight parameter, w1 krepresents the kth adjusted weight parameter, and β is a positive constant.
[0198] In a feasible implementation, for the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the termination condition includes a combination of one or more of the following conditions: the original error value in the Nth training cycle is less than a preset first threshold; the corrected error value in the Nth training cycle is less than a preset second threshold; the difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold; the difference between the corrected error value in the Nth training cycle and the corrected error value in the NMth training cycle is less than a preset fourth threshold; the difference between the pending weight parameter in the Nth training cycle and the pending weight parameter in the NMth training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold.
[0199] In a feasible implementation, when the Nth training cycle does not meet the termination condition, a combination of one or more of the following physical quantities is stored: the original error value in the Nth training cycle; the corrected error value in the Nth training cycle; the pending weight parameter in the Nth training cycle; and the cycle number N of the Nth training cycle.
[0200] In a feasible implementation, the forward propagation module 601 is specifically used to: during the first training cycle of the iterative training, use the preset initial weight parameters as the pending weight parameters; during the non-first training cycle of the iterative training, use the adjusted weight parameters of the neural network model as the pending weight parameters.
[0201] In a feasible implementation, the neural network model is used for image recognition; correspondingly, the sample data includes image samples; correspondingly, the output result includes a recognition result of the image recognition represented in a probabilistic form.
[0202] In a feasible implementation manner, the neural network model is used for sound recognition; correspondingly, the sample data includes sound samples; correspondingly, the output result includes a recognition result of the sound recognition represented in a probability form.
[0203] In a feasible implementation manner, the neural network model is used to acquire super-resolution images; correspondingly, the sample data includes image samples; correspondingly, the output result includes pixel values of the image after super-resolution processing.
[0204] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0205] Of course, the electronic device includes but is not limited to the unit modules listed above. For example, the electronic device may also include a communication unit, which may include a sending unit for sending data or signals to other devices, a receiving unit for receiving data or signals sent by other devices, etc. In addition, the specific functions that can be realized by the above functional units also include but are not limited to the functions corresponding to the method steps of the above examples. The detailed description of other units of the electronic device can refer to the detailed description of the corresponding method steps, and the embodiments of the present application will not be repeated here.
[0206] in, Figure 7 The processing unit 701 in the embodiment may be a processor or a controller, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processor (GPU), or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in combination with the contents disclosed in this application. The processor may also be a combination that implements computing functions, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The storage unit 702 may be a memory. The communication unit may be a transceiver, a radio frequency circuit or a communication interface, and the like. The processing unit 701 executes the following operations: Figure 5 The method for determining the weight parameters of the neural network model shown.
[0207] The embodiment of the present application also includes a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the following Figure 5 The method for determining the weight parameters of the neural network model shown.
[0208] The embodiment of the present application also includes a computer program product, when the computer program product is run on a computer, the computer is caused to execute the following Figure 5 The method for determining the weight parameters of the neural network model shown.
[0209] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0210] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0211] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0212] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0213] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0214] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for determining weight parameters of a neural network model, characterized in that: include: Based on the undetermined weight parameters of the neural network model, the sample data is processed to obtain the output result; Calculating an original error value between the output result and a preset expected result, wherein the original error value is a numerical representation of the difference between the output result and the expected result; Based on the correction value, correcting the original error value to obtain a corrected error value; Determining model weight parameters of the neural network model based on the corrected error value and the undetermined weight parameters; The correction value is obtained according to the following formula: R=(w k -Q(w k ))×Q(w k ) R represents the correction value, w k represents the kth undetermined weight parameter of the neural network model, Q(w k ) represents the quantized value of the kth undetermined weight parameter, where k is a non-negative integer; Wherein, the neural network model is used for image recognition; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes a recognition result of the image recognition represented in a probabilistic form; or, The neural network model is used for sound recognition; Correspondingly, the sample data includes sound samples; Correspondingly, the output result includes a recognition result of the sound recognition represented in a probabilistic form; or, The neural network model is used to acquire super-resolution images; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes pixel values of the image after super-resolution processing.
2. The method according to claim 1, characterized in that: The corrected error value is obtained according to the following formula: Wherein, E1 represents the corrected error value, E0 represents the original error value, α is a constant, m is the total number of undetermined weight parameters used to process the sample data, F((w k -Q(w k ))×Q(w k )) represents a function with the correction value as the independent variable, and m is a positive integer.
3. The method according to claim 2, characterized in that The function with the correction value as an independent variable is used to calculate the absolute value of the correction value; Correspondingly, the corrected error value is obtained according to the following formula: Among them, |(w k -Q(w k ))×Q(w k )| indicates the calculation (w k -Q(w k ))×Q(w k )’s absolute value.
4. The method according to any one of claims 1 to 3, characterized in that: The neural network model includes p network layers, each of the network layers includes q undetermined weight parameters, and the kth undetermined weight parameter is the jth undetermined weight parameter of the i-th network layer in the neural network model; Correspondingly, the corrected error value is obtained according to the following formula: Among them, p and q are positive integers, and i and j are non-negative integers.
5. The method according to any one of claims 1 to 3, characterized in that: The method of processing the sample data based on the undetermined weight parameters of the neural network model includes: Obtaining the undetermined weight parameter; quantizing the obtained undetermined weight parameter to obtain a quantized weight parameter, wherein the quantized weight parameter is a quantized value of the undetermined weight parameter; Using the quantized weight parameters as model weight parameters of the neural network model, and using a forward propagation algorithm to process the sample data; The output result is obtained from the output layer of the neural network model.
6. The method according to claim 5, characterized in that The model weight parameters of the neural network model are obtained by iterative training. When the iterative training meets the termination condition, the model weight parameters of the neural network model are determined based on the corrected error value and the undetermined weight parameters, including: The quantized weight parameters are used as model weight parameters of the neural network model.
7. The method according to claim 6, characterized in that When the iterative training does not meet the termination condition, determining the model weight parameters of the neural network model based on the corrected error value and the undetermined weight parameters includes: According to the corrected error value, a back propagation algorithm is used to adjust the undetermined weight parameters of the network layers of the neural network model layer by layer until the input layer of the neural network model to obtain the adjusted weight parameters of the neural network model.
8. The method according to claim 7, characterized in that The undetermined weight parameters of the neural network model are adjusted according to the following formula: Among them, w0 k represents the kth undetermined weight parameter, w1 k represents the kth adjusted weight parameter, and β is a positive constant.
9. The method according to any one of claims 6 to 8, characterized in that: For the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the end condition includes a combination of one or more of the following conditions: The original error value in the Nth training cycle is less than a preset first threshold; The corrected error value in the Nth training cycle is less than a preset second threshold; The difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold; The difference between the corrected error value in the Nth training cycle and the corrected error value in the NMth training cycle is less than a preset fourth threshold; The difference between the undetermined weight parameter in the Nth training cycle and the undetermined weight parameter in the NMth training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold.
10. The method according to claim 9, characterized in that When the Nth training cycle does not meet the end condition, one or more combinations of the following physical quantities are stored: The original error value in the Nth training cycle; The corrected error value in the Nth training cycle; The undetermined weight parameters in the Nth training cycle; and The number of cycles N of the Nth training cycle.
11. The method according to any one of claims 6 to 8 and 10, characterized in that: The obtaining of the undetermined weight parameter comprises: In the first training cycle of the iterative training, using the preset initial weight parameters as the pending weight parameters; In a non-first training cycle of the iterative training, the adjusted weight parameters of the neural network model are used as the pending weight parameters.
12. A device for determining weight parameters of a neural network model, characterized in that: include: The forward propagation module is used to process the sample data based on the undetermined weight parameters of the neural network model to obtain the output results; A comparison module, used to calculate an original error value between the output result and a preset expected result, wherein the original error value is a numerical representation of the difference between the output result and the expected result; A correction module, used for correcting the original error value based on the correction value to obtain a corrected error value; A determination module, used to determine the model weight parameters of the neural network model based on the corrected error value and the undetermined weight parameters; The correction value is obtained according to the following formula: R=(w k -Q(w k ))×Q(w k ) R represents the correction value, w k represents the kth undetermined weight parameter of the neural network model, Q(w k ) represents the quantized value of the kth undetermined weight parameter, where k is a non-negative integer; Wherein, the neural network model is used for image recognition; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes a recognition result of the image recognition represented in a probabilistic form; or, The neural network model is used for sound recognition; Correspondingly, the sample data includes sound samples; Correspondingly, the output result includes a recognition result of the sound recognition represented in a probabilistic form; or, The neural network model is used to acquire super-resolution images; Correspondingly, the sample data includes image samples; Correspondingly, the output result includes pixel values of the image after super-resolution processing.
13. The device according to claim 12, characterized in that The corrected error value is obtained according to the following formula: Wherein, E1 represents the corrected error value, E0 represents the original error value, α is a constant, m is the total number of undetermined weight parameters used to process the sample data, F((w k -Q(w k ))×Q(w k )) represents a function with the correction value as the independent variable, and m is a positive integer.
14. The device according to claim 13, characterized in that The function with the correction value as an independent variable is used to calculate the absolute value of the correction value; Correspondingly, the corrected error value is obtained according to the following formula: Among them, |(w k -Q(w k ))×Q(w k )| indicates the calculation (w k -Q(w k ))×Q(w k )’s absolute value.
15. The device according to any one of claims 12 to 14, characterized in that The neural network model includes p network layers, each of the network layers includes q undetermined weight parameters, and the kth undetermined weight parameter is the jth undetermined weight parameter of the i-th network layer in the neural network model; Correspondingly, the corrected error value is obtained according to the following formula: Among them, p and q are positive integers, and i and j are non-negative integers.
16. The device according to any one of claims 12 to 14, characterized in that The forward propagation module is specifically used for: Obtaining the undetermined weight parameter; quantizing the obtained undetermined weight parameter to obtain a quantized weight parameter, wherein the quantized weight parameter is a quantized value of the undetermined weight parameter; Using the quantized weight parameters as model weight parameters of the neural network model, and using a forward propagation algorithm to process the sample data; The output result is obtained from the output layer of the neural network model.
17. The device according to claim 16, characterized in that The model weight parameters of the neural network model are obtained by iterative training. When the iterative training meets the termination condition, the determination module is specifically used to: The quantized weight parameters are used as model weight parameters of the neural network model.
18. The device according to claim 17, characterized in that It also includes a back propagation module, when the iterative training does not meet the end condition, the back propagation module is specifically used to: According to the corrected error value, a back propagation algorithm is used to adjust the undetermined weight parameters of the network layers of the neural network model layer by layer until the input layer of the neural network model to obtain the adjusted weight parameters of the neural network model.
19. The device according to claim 18, characterized in that The undetermined weight parameters of the neural network model are adjusted according to the following formula: Among them, w0 k represents the kth undetermined weight parameter, w1 k represents the kth adjusted weight parameter, and β is a positive constant.
20. The device according to any one of claims 17 to 19, characterized in that For the Nth training cycle in the iterative training, N is an integer greater than 1, M is a positive integer less than N, and the end condition includes a combination of one or more of the following conditions: The original error value in the Nth training cycle is less than a preset first threshold; The corrected error value in the Nth training cycle is less than a preset second threshold; The difference between the original error value in the Nth training cycle and the original error value in the NMth training cycle is less than a preset third threshold; The difference between the corrected error value in the Nth training cycle and the corrected error value in the NMth training cycle is less than a preset fourth threshold; The difference between the undetermined weight parameter in the Nth training cycle and the undetermined weight parameter in the NMth training cycle is less than a preset fifth threshold; and N is greater than a preset sixth threshold.
21. The device according to claim 20, characterized in that When the Nth training cycle does not meet the end condition, one or more combinations of the following physical quantities are stored: The original error value in the Nth training cycle; The corrected error value in the Nth training cycle; The undetermined weight parameters in the Nth training cycle; and The number of cycles N of the Nth training cycle.
22. The device according to any one of claims 17 to 19 and 21, characterized in that: The forward propagation module is specifically used for: In the first training cycle of the iterative training, using the preset initial weight parameters as the pending weight parameters; In a non-first training cycle of the iterative training, the adjusted weight parameters of the neural network model are used as the pending weight parameters.
23. An electronic device, characterized in that: include: one or more processors and one or more memories; The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method for determining the weight parameters of the neural network model as described in any one of claims 1 to 11.
24. A computer storage medium, characterized in that It includes computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method for determining the weight parameters of a neural network model as described in any one of claims 1 to 11.
25. A computer program product, characterized in that When the computer program product runs on a computer, the computer is enabled to execute the method for determining the weight parameters of a neural network model as claimed in any one of claims 1 to 11.
