A quantitative training method, device and equipment for neural network model

Abstract

The embodiments of the present application provide a quantization training method, apparatus, and device for a neural network model, wherein the neural network model includes multiple layers. The method includes: obtaining first input data and parameters of a layer to be quantized during forward propagation; quantizing the first input data and parameters to obtain quantized first input data and quantized first parameters; performing operations on the quantized first input data and quantized first parameters to obtain first output data; determining whether the previous layer of the layer to be quantized depends on the first input data during backward propagation; and if so, saving the first input data. Implementing the embodiments of the present application and selecting different quantization methods for different scenarios can reduce the amount of computation required during neural network model training and save memory space.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, and device for quantitative training of a neural network model. Background Art

[0002] The neural network model needs to undergo multiple iterative training before it can converge. In each iterative training, it needs to go through three processes: forward propagation, backpropagation, and parameter update. First, each layer in the neural network processes the sample data to obtain the network loss function; then, the gradient of each layer is calculated based on the network loss function; secondly, the parameters of each layer are updated based on the gradient of each layer.

[0003] As neural network models become deeper, the number of parameters (weights, biases, etc.) within them also increases. Training these models places increasing demands on the computing and storage capabilities of hardware devices. Neural network models involve a large number of numerical operations, and the resulting large amounts of data consume a significant amount of memory. This makes training these models difficult, especially for hardware devices with limited memory. Summary of the Invention

[0004] Embodiments of the present application provide a method, apparatus, and device for quantizing training of a neural network model. The method helps reduce memory consumption by data generated during the training of the neural network model and increase available memory space.

[0005] In a first aspect, an embodiment of the present application discloses a quantization training method for a neural network model, characterized in that the neural network model includes multiple layers, and the method includes: in the forward propagation process, obtaining the first input data and parameters of the layer to be quantized; quantizing the first input data and the parameters respectively to obtain quantized first input data and quantized first parameters; performing operations on the quantized first input data and the quantized first parameters to obtain first output data; judging whether the previous layer of the layer to be quantized depends on the first input data during back propagation; if so, saving the first input data.

[0006] As can be seen, during the training process of the neural network model, the first input data and parameters of the quantized layer are quantized, and then calculations are performed on the quantized data to obtain the first output data. The quantized first input data and quantized parameters are fixed-point integers, while the first input data and parameters are floating-point data. Calculating on the quantized data reduces the amount of computation. By determining whether the previous layer of the quantized layer depends on the first input data during backpropagation, if so, the first input data is saved and the quantized first input data and quantized parameters are released, reducing the memory consumption of the data.

[0007] Based on the first aspect, in a possible implementation, the operating on the quantized first input data and the quantized first parameter to obtain the first output data includes: operating on the quantized first input data and the quantized first parameter to obtain the quantized first output data; and dequantizing the quantized first output data to obtain the first output data.

[0008] Based on the first aspect, in a possible implementation, after saving the first input data, the method further includes: in the back propagation process, quantizing the second input data of the layer to be quantized and the parameter respectively to obtain quantized second input data and quantized second parameters; the second input data of the layer to be quantized includes the first input data; performing operations on the quantized second input data and the quantized second parameter to obtain quantized second output data and the gradient of the quantized second parameter; performing dequantization operations on the quantized second output data and the gradient of the quantized second parameter to obtain the second output data and the gradient of the second parameter; and releasing the quantized second input data and the quantized second parameter from the memory.

[0009] It can be understood that when backpropagating to the layer to be quantized, the second input data and parameters are also quantized, and then the quantized data is operated, which reduces the amount of calculation; after the backpropagation of the layer to be quantized is completed, the unnecessary quantized second input data and quantized second parameters are released from the memory, that is, deleted, to save memory space.

[0010] Based on the first aspect, in a possible implementation, after releasing the quantized second input data and the quantized second parameters from the memory, the method further includes: inputting the second output data and the first input data into the previous layer of the layer to be quantized; the previous layer of the layer to be quantized is a layer containing an operator.

[0011] It can be understood that, because the previous layer of the layer to be quantized depends on the first input data during back propagation, the first input data needs to be input to the previous layer of the layer to be quantized.

[0012] Based on the first aspect, in a possible implementation, the method further includes: when a previous layer of the layer to be quantized does not depend on the first input data during back propagation, saving the quantized first input data and the quantized first parameter.

[0013] It can be understood that when the previous layer of the layer to be quantized does not rely on the first input data during back propagation, because the memory occupied by floating-point data is smaller than the memory of fixed-point data, the first input data is released, and only the quantized first input data and the first quantized parameter are saved, which is convenient for subsequent back propagation and does not require quantization again, saving computing time.

[0014] Based on the first aspect, in a possible implementation, the method further includes: performing a quantization operation on the third output data during back propagation to obtain quantized third output data; the third output data is the output data of the next layer of the layer to be quantized during back propagation; the quantized first input data, the quantized first parameter and the quantized third output data are input into the layer to be quantized, and operations are performed to obtain the quantized second output data and the gradient of the quantized first parameter; an inverse quantization operation is performed on the gradient of the quantized second output data and the quantized first parameter to obtain the gradient of the second output data and the first parameter; and the quantized second input data and the quantized first parameter are released from the memory.

[0015] Based on the first aspect, in a possible implementation, after releasing the second quantized input data and the first quantized parameters from the memory, the method further includes: inputting the second output data into the previous layer of the layer to be quantized; the previous layer of the layer to be quantized is a layer containing an operator.

[0016] It can be understood that the previous layer of the layer to be quantized may be a layer containing parameters (eg, weights, biases), or may be a layer containing no parameters (eg, weights, biases) but only operators.

[0017] Based on the first aspect, in a possible implementation, after the back propagation of the neural network model is completed, the method further includes: updating the parameters of each layer in the neural network model based on the parameters of each layer and the gradient of the parameters.

[0018] Based on the first aspect, in a possible implementation, the method is applicable to at least one layer in the neural network model.

[0019] It can be understood that the above-mentioned quantization training method can be applied to one or more layers of a neural network model. For example, it can be applied to layers with relatively large computational complexity in a neural network model, such as convolutional layers, deconvolutional layers, fully connected layers, or layers with matrix multiplication operations.

[0020] In a second aspect, an embodiment of the present application provides a method for testing a neural network model, characterized in that it includes: obtaining test data; testing the test data using a trained neural network model; the neural network model is trained by the method of the first aspect above or the method described in any embodiment of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a schematic diagram of a quantization training device for a neural network model, wherein the neural network model includes multiple layers, and the device includes:

[0022] An acquisition unit, configured to acquire first input data and parameters of the layer to be quantized during forward propagation;

[0023] a quantization unit, configured to quantize the first input data and the parameter respectively to obtain quantized first input data and quantized first parameter;

[0024] an operation unit, configured to operate on the quantized first input data and the quantized first parameter to obtain first output data;

[0025] a judging unit, configured to judge whether a previous layer of the layer to be quantized depends on the first input data during back propagation;

[0026] A storage unit is used to save the first input data when a previous layer of the layer to be quantized depends on the first input data during back propagation.

[0027] Based on the third aspect, in a possible implementation, the operation unit is used to: perform an operation on the quantized first input data and the quantized first parameter to obtain quantized first output data; and perform inverse quantization on the quantized first output data to obtain the first output data.

[0028] Based on the third aspect, in a possible implementation, the quantization unit is also used to, during the back-propagation process, quantize the second input data of the layer to be quantized and the parameter, respectively, to obtain the quantized second input data and the quantized second parameter; the second input data of the layer to be quantized includes the first input data; the operation unit is also used to perform operations on the quantized second input data and the quantized second parameter to obtain the quantized second output data and the gradient of the quantized second parameter; the operation unit is also used to perform dequantization operations on the quantized second output data and the gradient of the quantized second parameter to obtain the second output data and the gradient of the second parameter; the device also includes: a release unit, used to release the quantized second input data and the quantized second parameter from the memory.

[0029] Based on the third aspect, in a possible implementation, the acquisition unit is further configured to input the second output data and the first input data into a layer preceding the layer to be quantized; the layer preceding the layer to be quantized is a layer including an operator.

[0030] Based on the third aspect, in a possible implementation, the storage unit is further used to save the quantized first input data and the quantized first parameter when a previous layer of the layer to be quantized does not depend on the first input data during back propagation.

[0031] Based on the third aspect, in a possible implementation, the quantization unit is also used to perform a quantization operation on the third output data during the back propagation process to obtain quantized third output data; the third output data is the output data of the next layer of the layer to be quantized during the back propagation process; the operation unit is also used to input the quantized first input data, the quantized first parameter and the quantized third output data into the layer to be quantized, perform an operation to obtain the quantized second output data and the gradient of the quantized first parameter; the operation unit is also used to perform an inverse quantization operation on the gradient of the quantized second output data and the quantized first parameter to obtain the gradient of the second output data and the first parameter; the release unit is also used to release the quantized second input data and the quantized first parameter from the memory.

[0032] Based on the third aspect, in a possible implementation manner, the acquisition unit is further configured to input the second output data into a layer preceding the layer to be quantized; the layer preceding the layer to be quantized is a layer including an operator.

[0033] Based on the third aspect, in a possible implementation, the device further includes: a parameter updating unit, configured to update the parameters of each layer in the neural network model based on the parameters of each layer and the gradient of the parameters.

[0034] In a fourth aspect, an embodiment of the present application provides a testing device for a neural network model, comprising:

[0035] Get test data;

[0036] The test data is tested using a trained neural network model; the neural network model is trained by the method of the first aspect or the method described in any embodiment of the first aspect.

[0037] In a fifth aspect, an embodiment of the present application provides a quantitative training device for a neural network model, comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to call the instructions to execute the method of the first aspect or the method described in any embodiment of the first aspect.

[0038] In a sixth aspect, an embodiment of the present application provides a testing device for a neural network model, comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to call the instructions to execute the method described in the second aspect above.

[0039] In the seventh aspect, an embodiment of the present application provides a non-volatile storage medium for storing program instructions. When the program instructions are applied to a training device for a neural network model, they can be used to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0040] In an eighth aspect, an embodiment of the present application provides a non-volatile storage medium for storing program instructions. When the program instructions are applied to a test device for a neural network model, they can be used to implement the method described in the second aspect or any possible implementation of the second aspect.

[0041] In a ninth aspect, embodiments of the present application provide a computer program product comprising program instructions. When the computer program product is executed by a quantized training device for a neural network model, the quantized testing device for the neural network model performs the method described in the first aspect. The computer program product may be a software installation package. When the method provided by any possible design of the first aspect is required, the computer program product may be downloaded and executed on the quantized training device for the neural network model to implement the method described in the first aspect or any possible implementation of the first aspect.

[0042] In a tenth aspect, embodiments of the present application provide a computer program product comprising program instructions. When the computer program product is executed by a neural network model testing device, the neural network model testing device performs the method described in the second aspect. The computer program product may be a software installation package. When the method provided by any possible design of the second aspect is required, the computer program product may be downloaded and executed on the neural network model testing device to implement the method described in the second aspect or any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of a quantized training device for a neural network model provided in an embodiment of the present application;

[0045] Figure 2 A schematic structural diagram of a combined device provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a board structure provided in an embodiment of the present application;

[0047] Figure 4 (a) and (b) are schematic diagrams of examples of forward propagation and reverse propagation respectively;

[0048] Figure 5(a) and (b) are schematic diagrams of another example of forward propagation and backward propagation respectively;

[0049] Figure 6 A schematic diagram of an example of the fusion of forward propagation and backpropagation provided in an embodiment of the present application;

[0050] Figure 7 A flowchart of a quantized training method for a neural network model provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of forward propagation provided in an embodiment of the present application;

[0052] Figure 9 A flowchart of a quantized training method for a neural network model provided in an embodiment of the present application;

[0053] Figure 10 An example schematic diagram of back propagation provided in an embodiment of the present application;

[0054] Figure 11 An exemplary schematic diagram of another back propagation provided in an embodiment of the present application;

[0055] Figure 12 A flowchart of a method for testing a neural network model provided in an embodiment of the present application;

[0056] Figure 13 A schematic diagram of a testing device for a neural network model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] It should be noted that the terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a," "the," and "the" used in the embodiments of this application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0059] It should be noted that when used in this specification and the appended claims, the terms "comprise" and "include" and any variations thereof are intended to cover non-exclusive inclusions. For example, a system, product, or device comprising a series of units / components is not limited to the listed units / components but may optionally include units / components not listed, or other units / components inherent to the product or device.

[0060] It will also be understood that the term “if” may be interpreted as “when” or “upon” or “in response to determining” or “in response to detecting” or “in the event of” depending on the context.

[0061] It should be noted that the first and second in this application are only used to distinguish the objects in the forward propagation process and the reverse propagation process, and are not used to describe a specific order. The first input data and the first output data correspond to the input data and output data during the forward propagation, respectively, and the second input data and the second output data correspond to the input data and output data during the reverse propagation, respectively.

[0062] See also Figure 1 , Figure 1 A schematic diagram of a quantization training device 100 for a neural network model provided in an embodiment of the present application, wherein the neural network model includes multiple layers, and the device 100 includes:

[0063] An acquisition unit 101 is configured to acquire first input data and parameters of a layer to be quantized during forward propagation.

[0064] a quantization unit 102, configured to quantize the first input data and the parameter respectively to obtain quantized first input data and quantized first parameter;

[0065] an operation unit 103, configured to operate on the first quantized input data and the first quantized parameter to obtain first output data;

[0066] A judging unit 104 is configured to judge whether a previous layer of the layer to be quantized depends on the first input data during back propagation;

[0067] The storage unit 105 is configured to store the first input data when a previous layer of the layer to be quantized depends on the first input data during back propagation.

[0068] In a possible implementation, the operation unit 103 is configured to: operate on the quantized first input data and the quantized first parameter to obtain quantized first output data; and dequantize the quantized first output data to obtain the first output data.

[0069] In a possible implementation, the quantization unit 102 is further used to, during the back-propagation process, quantize the second input data of the layer to be quantized and the parameter, respectively, to obtain the quantized second input data and the quantized second parameter; the second input data of the layer to be quantized includes the first input data; the operation unit 103 is further used to perform operations on the quantized second input data and the quantized second parameter, to obtain the quantized second output data and the gradient of the quantized second parameter; the operation unit 103 is further used to perform dequantization operations on the quantized second output data and the gradient of the quantized second parameter, to obtain the second output data and the gradient of the second parameter; the device 100 also includes: a release unit 106, used to release the quantized second input data and the quantized second parameter from the memory.

[0070] In a possible implementation manner, the acquisition unit 101 is further configured to input the second output data and the first input data into a layer preceding the layer to be quantized; the layer preceding the layer to be quantized is a layer including an operator.

[0071] In a possible implementation manner, the storage unit 105 is further configured to store the quantized first input data and the quantized first parameter when a previous layer of the layer to be quantized does not depend on the first input data during back propagation.

[0072] In a possible implementation, the quantization unit 102 is further used to perform a quantization operation on the third output data during the back propagation process to obtain quantized third output data; the third output data is the output data of the next layer of the layer to be quantized during the back propagation process; the operation unit 103 is further used to input the quantized first input data, the quantized first parameter and the quantized third output data into the layer to be quantized, perform an operation to obtain the quantized second output data and the gradient of the quantized first parameter; the operation unit 103 is further used to perform an inverse quantization operation on the gradient of the quantized second output data and the quantized first parameter to obtain the gradient of the second output data and the first parameter; the release unit 106 is further used to release the quantized second input data and the quantized first parameter from the memory.

[0073] In a possible implementation manner, the acquisition unit 101 is further configured to input the second output data into a layer preceding the layer to be quantized; the layer preceding the layer to be quantized is a layer including an operator.

[0074] In a possible implementation, the apparatus 100 further includes: a parameter updating unit 107, configured to update the parameters of each layer in the neural network model based on the parameters of each layer and the gradient of the parameters.

[0075] Each functional unit of the above-mentioned device 100 can be used to implement the following Figure 7 or Figure 9 The method described in the embodiment, the specific content can be referred to Figure 7 or Figure 9 Description in the relevant content of the embodiment.

[0076] Figure 2 1 is a structural diagram showing a combined processing device 200 according to an embodiment of the present disclosure, which can be used for quantization training of a neural network model or for testing a neural network model. Figure 2 As shown in FIG, the combined processing device 200 includes a computing processing device 202, an interface device 204, other processing devices 206, and a storage device 208. According to different application scenarios, the computing processing device may include one or more computing devices 210, which may be configured to execute the following Figure 7 or attached Figure 9 The described operation.

[0077] In various embodiments, the computing and processing device of the present disclosure may be configured to perform user-specified operations. In exemplary applications, the computing and processing device may be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included in the computing and processing device may be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, the computing and processing device of the present disclosure may be considered to have a single-core structure or a homogeneous multi-core structure.

[0078] In exemplary operation, the computing processing device of the present disclosure can interact with other processing devices through interface means, to jointly complete the operation specified by the user. Depending on the difference in implementation, the other processing devices of the present disclosure may include one or more types of processors in general and / or special processors such as central processing unit (Central Processing Unit, CPU), graphics processing unit (Graphics Processing Unit, GPU), artificial intelligence processor. These processors may include but are not limited to digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As previously mentioned, only with respect to the computing processing device of the present disclosure, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, the two can be regarded as forming a heterogeneous multi-core structure.

[0079] In one or more embodiments, the other processing device may serve as an interface between the computing device disclosed herein (which may be embodied as an artificial intelligence computing device such as a neural network computing device) and external data and control, performing basic control including but not limited to data transfer, starting and / or stopping the computing device, and so on. In other embodiments, the other processing device may also collaborate with the computing device to jointly complete computing tasks.

[0080] In one or more embodiments, the interface device can be used to transmit data and control instructions between the computing and processing device and other processing devices. For example, the computing and processing device can obtain input data from other processing devices via the interface device and write it to the storage device (or memory) on the computing and processing device chip. Furthermore, the computing and processing device can obtain control instructions from other processing devices via the interface device and write them to the control cache on the computing and processing device chip. Alternatively or optionally, the interface device can also read data from the storage device of the computing and processing device and transmit it to other processing devices.

[0081] Additionally or optionally, the combined processing device of the present disclosure may further include a storage device. As shown in the figure, the storage device is connected to the computing processing device and the other processing device, respectively. In one or more embodiments, the storage device may be used to store data of the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage device of the computing processing device or other processing device.

[0082] In some embodiments, the present disclosure also discloses a chip, such as Figure 3 In one implementation, the chip is a system on chip (SoC) and integrates one or more components such as Figure 2 The chip can be connected to the external interface device (such as Figure 3 The external interface device 306 shown in the figure is connected to other related components. The related components can be, for example, a camera, a display, a mouse, a keyboard, a network card or a wifi interface. In some application scenarios, other processing units (such as video codecs) and / or interface modules (such as DRAM interfaces) can be integrated on the chip. In some embodiments, the present disclosure also discloses a chip packaging structure, which includes the above-mentioned chip. In some embodiments, the present disclosure also discloses a board card, which includes the above-mentioned chip packaging structure. The following will be combined with Figure 3 The board is described in detail.

[0083] Figure 3 FIG. 1 is a schematic diagram showing the structure of a board 300 according to an embodiment of the present disclosure. Figure 3 As shown in , the board includes a storage device 304 for storing data, which includes one or more storage units 310. The storage device can be connected to the control device 308 and the chip 302 described above and transmit data by means of, for example, a bus. Furthermore, the board also includes an external interface device 306, which is configured for data relay or transfer function between the chip (or the chip in the chip packaging structure) and the external device 312 (such as a server or computer, etc.). For example, the data to be processed can be passed from the external device to the chip through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external device via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE interface, etc.

[0084] In one or more embodiments, the control device in the disclosed board can be configured to regulate the state of the chip. To this end, in one application scenario, the control device can include a microcontroller unit (MCU) for regulating the working state of the chip.

[0085] According to the above combination Figure 2 and Figure 3 Based on the description, those skilled in the art can understand that the present disclosure also discloses an electronic device or apparatus, which may include one or more of the above-mentioned boards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0086] According to different application scenarios, the electronic devices or devices disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, Internet of Things terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, vehicles, household appliances, and / or medical equipment. The vehicles include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging (MRI), ultrasound machines and / or electrocardiographs. The electronic devices or devices disclosed herein may also be applied to the Internet, Internet of Things, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical care and other fields. Furthermore, the electronic devices or devices disclosed herein may also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, electronic devices or apparatuses with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smartphones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0087] Before describing the embodiments of the present application, the training process of the neural network model is first introduced.

[0088] Each iteration of neural network model training requires forward propagation, backpropagation, and parameter updates. Forward propagation propagates from the first layer of the neural network model to the last layer, while backpropagation propagates from the last layer to the first layer. Forward propagation calculates the network's loss function, while backpropagation calculates the gradients of each layer based on the loss function. Parameter updates update the parameters of each layer based on the gradients of each layer. After multiple iterations of training, the neural network model eventually converges, resulting in a trained neural network model. This trained neural network model can be used in fields such as image processing, speech recognition, and video.

[0089] Different neural network models are built differently for different application scenarios, and the training process for these models is naturally different. The following describes the training process for two common neural network models. This training process is applied to the processor.

[0090] In one example, see Figure 4 As shown in (a), Figure 4 (a) is a structural diagram of the forward propagation process of the neural network model provided in this application. The forward propagation process is applied to the above-mentioned device 100 or combined device 200 or board 300. In the figure, when forward propagating to the layer to be quantized, the processor inputs data X and W into the layer to be quantized, wherein X represents the input data of the layer, and W represents the parameters corresponding to the layer (weight, bias, etc.). The operation unit in the layer to be quantized calculates X and W to obtain output data Y, wherein the input data X of the layer to be quantized is also the output data of the previous layer of the layer to be quantized. The data Y is then used as the input data of the next layer of the layer to be quantized, and the operation unit in the next layer of the layer to be quantized calculates according to Y and the parameters of the next layer of the layer to be quantized, and obtains output data... until the forward propagation reaches the last layer of the neural network model and the loss function is obtained.

[0091] See also Figure 4 As shown in (b), Figure 4 (b) is a structural diagram of the back propagation process of the neural network model provided in this application. In the figure, when back propagating to the layer to be quantized, the gradients of X and Y are input into the layer to be quantized, wherein the gradients of X and Y represent input data, W represents the parameters corresponding to the layer (weights, biases, etc.), and the operation unit in the layer to be quantized calculates the gradients of X, W, and Y to obtain the gradient of the output data X. In addition, the output data of the layer to be quantized also includes the gradient of W, and the gradient of W is used in the parameter update process. The gradient of X is input into the previous layer of the layer to be quantized as the input data of the previous layer of the layer to be quantized, and the operation unit of the previous layer of the layer to be quantized calculates the gradient and parameters of X to obtain the output data... until back propagation to the first layer. After back propagation, the gradients in each layer of the network are calculated.

[0092] During the forward propagation and back propagation processes, the output data obtained by each layer of the neural network is stored as intermediate data in the storage device 208 or the storage unit 310. When the computing device 202 performs calculations, the required data is obtained from the storage device 208 or the storage unit 310.

[0093] After each round of iterative backpropagation of the neural network model, the parameters of each layer in the network are updated based on the gradients in each layer.

[0094] In this example, the neural network model has the following characteristics. For the layer to be quantized, the input data during the back propagation process includes not only the output data of the next layer, but also the input data of the layer during the normal propagation process. For example, for the layer to be quantized, during back propagation, the input data of the layer to be quantized includes the gradient of the output data Y of the next layer of the layer to be quantized, and also includes the input data X of the layer to be quantized during the forward propagation process; for the layer before the layer to be quantized, during back propagation, the input data of the layer before the layer to be quantized includes the gradient of the output data X of the layer to be quantized, and also includes the input data of the layer before the layer to be quantized during the forward propagation process. The above input data is stored in the storage device 208 or the storage unit 310.

[0095] In another example, see Figure 5 As shown, Figure 5 (a) is a schematic diagram of the structure of the forward propagation process of the neural network model provided in this application. Figure 5 (b) is a schematic diagram of the back propagation process of the neural network model provided by this application. Figure 4 The difference between the neural network model in the example is that the back propagation process of the previous layer of the to-be-quantized layer depends on the input data X of the forward propagation of the to-be-quantized layer. That is, when back propagating to the previous layer of the to-be-quantized layer, the input data includes the input data X of the forward propagation of the to-be-quantized layer. Figure 4 The forward propagation process described in (a) is similar and will not be repeated here.

[0096] In this example, the neural network model has Figure 4 In addition to the characteristics of the neural network model shown, it also has the following characteristics: for the previous layer of the layer to be quantized, the derivation process of backpropagation needs to rely on the input data of the layer to be quantized during forward propagation, that is, when backpropagating to the previous layer of the layer to be quantized, the input data of this layer includes the input data of the layer to be quantized during forward propagation.

[0097] In order to more intuitively reflect the characteristics of the neural network model in this example, this application provides a schematic diagram of neural network model training. Figure 6 ,from Figure 6 As can be seen, 1) X and W are used not only for the forward propagation of the layer to be quantized, but also for the backward propagation of the layer to be quantized; 2) the input data X of the layer to be quantized is also used for the backward propagation (derivation) of the layer before the layer to be quantized. The layer before the layer to be quantized here is the layer containing operators in the neural network model. This layer can include parameters such as weights and biases, or it can include only operators without weights and biases.

[0098] It should be noted that the above Figure 4 (a) and (b) in Figure 5 (a) and (b) in the figure only illustrate one round of iterative process in the training process of the neural network model. The neural network model needs to go through many rounds of iterative training to reach convergence. For the sake of brevity, the description will not be expanded.

[0099] During the training process of the neural network model, the data exists in the form of floating-point data. The large amount of calculations on floating-point data will occupy a large amount of memory, resulting in a large memory overhead problem for the device. Training the neural network model is especially challenging for hardware devices with small memory.

[0100] In order to solve the above problems, the present invention provides a quantization training method for a neural network model. Figure 7 , Figure 7 This is a flowchart of a quantized training method for a neural network model. This method can be applied to a computing device or processor. For example, the computing device can be a server, a terminal device, a chip, etc. Figure 8 Describe the embodiments of this application. Figure 8 A schematic diagram of the forward propagation process of the neural network model training process provided in an embodiment of the present application is provided. This method embodiment includes but is not limited to the description of the following content.

[0101] S701. During forward propagation, obtain first input data and parameters of a layer to be quantized.

[0102] See also Figure 8 As shown in the schematic diagram, during the forward propagation process, the first input data X and parameter W of the layer to be quantized are obtained, where the parameter W includes weights and may also include biases. If the layer to be quantized is the first layer in the neural network model, the first input data is the sample data input to the neural network model; if the layer to be quantized is not the first layer of the neural network model, the first input data is the first output data of the layer before the layer to be quantized. For example, Figure 8 The first input data X of the layer to be quantized may be the first output data of the previous layer of the layer to be quantized.

[0103] It should be noted that the current iteration can be any iteration process in the neural network model training, such as the first iteration, an intermediate iteration, or the last iteration.

[0104] S702: Quantize the first input data and the parameter respectively to obtain the quantized first input data and the quantized first parameter.

[0105] The first input data X and the parameter W are quantized to obtain quantized first input data X1 and quantized W1, respectively. Generally, the first input data and the parameter are floating-point data, and the quantized first input data and the quantized parameter are fixed-point integer data. For example, a commonly used floating-point data type is float32, and commonly used fixed-point data types are int8 and int16. This application does not limit the specific quantization method.

[0106] S703: Perform an operation on the quantized first input data and the quantized first parameter to obtain first output data.

[0107] Optionally, the first output data of the layer to be quantized is obtained based on the quantized first input data and the quantized parameters, including: first calculating the quantized first output data based on the quantized first input data and the quantized parameters, and then inverse quantizing the quantized first output data to obtain the first output data of the layer to be quantized.

[0108] refer to Figure 8 As shown, a calculation is performed based on the quantized first input data X1 and the quantized parameter W1 to obtain the quantized first output data Y1, where Y1 is fixed-point integer data; the fixed-point data Y1 is then dequantized to obtain the first output data Y, where Y is floating-point data. This application does not limit the specific dequantization method.

[0109] Quantization can transform data requiring large computational complexity into data requiring smaller computational complexity. This reduces the amount of computation required and increases computational speed. For example, floating-point data has many decimal places after the decimal point, while fixed-point integer data is a whole number. Therefore, operations on fixed-point integer data can be performed faster than on floating-point data, and the data takes up less memory.

[0110] Optionally, after obtaining the first output data of the layer to be quantized, the quantized first input data and the quantized parameters are released from the memory, that is, deleted. Figure 8 As shown, after obtaining the first output data Y of the layer to be quantized, the quantized first input data X1 and the quantized parameters W1 (quantized weight, quantized bias, etc.) are released from the memory.

[0111] After the forward propagation of this layer is completed, the first quantized input data and quantized parameters of this layer are released in order to increase the available memory. Especially when the memory of the hardware device is limited, it prevents a lot of intermediate data generated during the training of the neural network model from occupying too much memory and affecting the training of the neural network model.

[0112] S704: Determine whether the previous layer of the layer to be quantized depends on the first input data during back propagation.

[0113] From the above, we can see that Figure 4 In the application scenario corresponding to the neural network model, the previous layer of the quantized layer does not rely on the first input data X during back propagation, but Figure 5 In the application scenario corresponding to the neural network model, the previous layer of the quantized layer depends on the first input data X during back propagation.

[0114] During forward propagation to the layer to be quantized, it is necessary to determine whether the previous layer of the layer to be quantized depends on the first input data during backward propagation. This determination is made as follows. When obtaining the first input data X of the layer to be quantized, the first input data X carries the operation logic of the previous layer of the layer to be quantized. Based on the carried operation logic of the previous layer of the layer to be quantized, it can be determined whether the previous layer of the layer to be quantized depends on the first input data during backward propagation. For example, the operation logic of the previous layer of the layer to be quantized is: A+B+3X, where A and B can be constants or parameters. When backpropagating, the derivative of this operation logic must be taken with respect to the first input data X. Therefore, it can be determined that the previous layer of the layer to be quantized depends on the first input data during backward propagation. For another example, the operation logic of the previous layer of the layer to be quantized is: ABC, where ABC can be a constant or a parameter. When backpropagating, the derivative of this operation logic does not involve the first input data X. Therefore, it can be determined that the previous layer of the layer to be quantized does not depend on the first input data during backward propagation. It should be noted that this is merely an example to explain whether the previous layer of the layer to be quantized depends on the first input data during back propagation, and this example does not constitute a limitation to the present application.

[0115] It should be noted that this step may be placed before step S701, or before or after any step between steps S701 and S703.

[0116] S705: If yes, save the first input data.

[0117] After judgment, if the previous layer of the layer to be quantized depends on the first input data during back propagation, it means that the first input data X needs to be input into the previous layer of the layer to be quantized during back propagation. Therefore, the first input data X needs to be saved here. In order to reduce the memory consumption of intermediate data, the quantized first input data and the first quantized parameter can be released.

[0118] It should be noted that steps S701 to S705 describe the processing process when forward propagation reaches the layer to be quantized. After that, forward propagation continues, that is, the output data Y of the layer to be quantized is input into the next layer of the layer to be quantized, and the next layer of the layer to be quantized is processed according to the input data and parameters, ..., until the forward propagation reaches the last layer of the neural network model, the loss function is calculated, and the forward propagation ends.

[0119] It can be seen that during forward propagation, the first input data and parameters of the layer to be quantized are quantized, and then the quantized data is processed. Compared with performing operations on floating-point data, the amount of operations is reduced and the operation time is saved. After the forward propagation of the layer to be quantized is completed, the corresponding quantization training method is selected according to whether the previous layer of the layer to be quantized depends on the first input data during back propagation, and the first input data or the quantized data (quantized first input data and quantized parameters) is released, which reduces the memory consumption of the data. Especially for devices with a certain amount of memory, increasing the available memory space can ensure the normal training of the neural network model.

[0120] The present application also provides another quantization training method for a neural network model, see Figure 9 As shown, Figure 9 This is a flowchart of a quantized training method for a neural network model. Figure 10 and Figure 11 This embodiment is described. Figure 10 and Figure 11 are all back propagation schematic diagrams of the neural network model training process provided in the embodiments of the present application, wherein: Figure 10 The first training method is shown. Figure 11 The second training method is shown. The method includes but is not limited to the following steps, wherein steps S901 to S905 can refer to the above Figure 7 For the sake of brevity, the description of the contents of steps S701 to S705 will not be repeated here.

[0121] S901. During forward propagation, obtain first input data and parameters of the layer to be quantized.

[0122] S902 : quantize the first input data and the parameter respectively to obtain the quantized first input data and the quantized first parameter.

[0123] S903 : Perform an operation on the quantized first input data and the quantized first parameter to obtain first output data.

[0124] S904: Determine whether the previous layer of the layer to be quantized depends on the first input data during back propagation.

[0125] Determine whether the previous layer of the layer to be quantized depends on the first input data during back propagation. If it is determined that the previous layer of the layer to be quantized depends on the first input data during back propagation, execute steps S905a to S908a. If it is determined that the previous layer of the layer to be quantized does not depend on the first input data during back propagation, execute steps S905b to S908b. It should be noted that there is no logical order between steps S905a and S905b, steps S906a and S906b, steps S907a and S907b, and steps S908a and S908b.

[0126] The following first introduces the case where the previous layer of the layer to be quantized depends on the first input data during back propagation, that is, one of the branches.

[0127] S905a: When a previous layer of the layer to be quantized depends on the first input data during back propagation, save the first input data.

[0128] S906a: During the back-propagation process, the second input data and the parameters of the layer to be quantized are quantized respectively to obtain the quantized second input data and the quantized second parameters.

[0129] Backpropagation is the forward propagation from the last layer of the neural network model. The last layer calculates the output data based on the first input data and loss function during normal propagation, and continues to propagate forward until it backpropagates to the layer to be quantized.

[0130] When back-propagating to the layer to be quantized, the second input data and parameters of the layer to be quantized are quantized respectively to obtain the quantized second input data and the quantized second parameters, wherein the second input data of the layer to be quantized includes the first input data. Figure 10 As shown, the second input data of the layer to be quantized includes the gradients of X and Y, where X is the first input data of the layer to be quantized during forward propagation, W is the parameter of the layer to be quantized during forward propagation, and the gradient of Y is the output data of the next layer of the layer to be quantized, and is also the gradient of the first output data of the layer to be quantized during forward propagation. The second input data of quantization includes X1 and Y2, where X1 is the quantization result of X, and Y2 is the quantization result of the gradient of Y. The second quantized parameter W1 includes the quantization weight and may also include the quantization bias, etc.

[0131] It should be noted that, in general, the second quantized parameter (during the back propagation process) is the same as the first quantized parameter (in the forward propagation process). However, in the above steps, in order to save memory space, the first quantized parameter is released, and the first quantized parameter does not exist in the memory. This step (back propagation process) quantizes the parameter again, so it is called the second quantized parameter.

[0132] S907a: Perform an operation on the quantized second input data and the quantized second parameter to obtain second output data and a gradient of the second parameter.

[0133] Optionally, based on the quantized second input data and the quantized second parameter, the gradient of the second output data and the second parameter of the layer to be quantized is obtained, including: performing calculation based on the quantized second input data and the quantized second parameter to obtain the gradient of the quantized second output data and the quantized second parameter of the layer to be quantized, and then dequantizing the gradient of the quantized second output data and the quantized second parameter to obtain the gradient of the second output data and the second parameter of the layer to be quantized.

[0134] See also Figure 10 As shown, calculation is performed based on the quantized second input data X1 and Y2 and the quantized parameter W1 of the layer to be quantized to obtain the quantized second output data X2 and the gradient W2 of the quantized second parameter. Then, dequantization calculation is performed on X2 and W2 to obtain the gradient of X and the gradient of W. The gradient of X can be used as the input data of the previous layer of the layer to be quantized to continue backpropagation, and the gradient of W is used to calculate the optimizer in the parameter update process, and the optimizer is the parameter update method.

[0135] Optionally, after obtaining the second output data and the gradient of the second parameter of the layer, the quantized second input data and the quantized second parameter are released from the memory to reduce the memory usage of the intermediate data and increase the available memory space. For example, Figure 10 After calculating the gradient of the second output data X of the layer to be quantized and the gradient of the second parameter W, the quantized second input data X1, Y2 and the quantized second parameter W1 (quantized weight, quantized bias, etc.) are released from the memory.

[0136] S908a: Input the second output data and the first input data into a layer before the layer to be quantized.

[0137] After the back propagation of the layer to be quantized is completed, the back propagation is carried out to the layer before the layer to be quantized. Since the back propagation of the layer before the layer to be quantized depends on the first input data, the second output data and the first input data are input to the layer before the layer to be quantized for processing. Figure 10 As shown, the gradient of the second output data X and the first input data X are input to the previous layer of the layer to be quantized. In addition, the input data input to the previous layer of the layer to be quantized also includes other data, such as the first input data of the previous layer of the layer to be quantized (input data during forward propagation).

[0138] It should be noted that after this step is completed, back propagation continues until it reaches the first layer of the neural network model, and then back propagation ends.

[0139] The following describes a case where the previous layer of the layer to be quantized does not depend on the first input data during back propagation, that is, another branch.

[0140] S905b: When the previous layer of the layer to be quantized does not depend on the first input data during back propagation, save the quantized first input data and the first quantized parameter.

[0141] Upon determination, if the previous layer of the layer to be quantized does not rely on the first input data during backpropagation, the quantized first input data X1 and the quantized first parameter W1 can be saved. To reduce memory consumption by intermediate data, the first input data X (usually floating-point data) can be released. It should be noted that because the entire backpropagation process no longer relies on the first input data X, the first input data X is released. The quantized first input data X1 and the quantized first parameter W1 are required during backpropagation of the layer to be quantized. Therefore, saving the quantized first input data X1 and the quantized first parameter W1 allows direct use of X1 and W1 stored in memory during backpropagation to the layer to be quantized, eliminating the need to quantize X and W. Furthermore, the quantized first input data X1 and the quantized first parameter W1 are fixed-point data, while the first input data is floating-point data. Fixed-point data occupies less memory space than floating-point data, thus reducing the amount of computation and saving computation time as well as memory space.

[0142] After that, forward propagation continues, that is, the output data Y of the layer to be quantized is input into the next layer to be quantized, and the next layer to be quantized processes the data according to the input data and parameters, ..., until the forward propagation reaches the last layer of the neural network model, the loss function is calculated, and the forward propagation ends.

[0143] S906b: During the back-propagation process, perform a quantization operation on the third output data to obtain quantized third output data.

[0144] The third output data is the output data of the next layer after the layer to be quantized during the back propagation process. Figure 11 As shown, the third output data refers to the gradient of Y.

[0145] See also Figure 11 As shown, when backpropagating to the layer to be quantized, the second input data of the layer to be quantized includes the gradient of the third output data Y, X1, and W1. Among them, X1 and W1 are both quantized data, so the gradient of the third output data Y needs to be quantized to obtain the quantized third output data Y2.

[0146] S907b: Input the quantized first input data, the quantized first parameter, and the quantized third output data into the layer to be quantized, perform operation, and obtain the second output data and the gradient of the first parameter.

[0147] Optionally, the quantized first input data, the quantized first parameter and the quantized third output data are input into the layer to be quantized, and operations are performed to obtain the second output data and the gradient of the first parameter, including: inputting the quantized first input data, the quantized first parameter and the quantized third output data into the layer to be quantized, and operations are performed to obtain the quantized second output data and the gradient of the quantized first parameter, and then performing dequantization operations on the gradient of the quantized second output data and the quantized first parameter to obtain the gradient of the second output data and the first parameter.

[0148] See also Figure 11 As shown, a calculation is performed based on the quantized first input data X1, the quantized third output data Y2, and the quantized parameter W1 to obtain the quantized second output data X2 and the gradient W2 of the quantized first parameter. Then, a dequantization calculation is performed on X2 and W1 to obtain the gradient of X and the gradient of W. The gradient of X can be used as the input data of the previous layer of the layer to be quantized to continue backpropagation. The gradient of W is used to calculate the optimizer in the parameter update process, and the optimizer is the parameter update method.

[0149] Optionally, the quantized second input data X2 and the quantized first parameter W2 may be released from the memory to reduce the memory usage of the intermediate data and increase the available memory space.

[0150] S908b: Input the second output data into the previous layer of the layer to be quantized.

[0151] The previous layer of the layer to be quantized is a layer containing operators, that is, the previous layer of the layer to be quantized can be a layer containing parameters (weights, biases, etc.), or it can be a layer that does not contain parameters (weights, biases, etc.) and only contains simple operators or formulas.

[0152] After the back propagation of the layer to be quantized is completed, back propagation is performed to the layer before the layer to be quantized, and the gradient of the second output data X is directly input into the layer before the quantized layer. Of course, other data can also be used as input data, such as the first input data of the layer before the layer to be quantized (input data during forward propagation).

[0153] It should be noted that after this step is completed, back propagation continues until it reaches the first layer of the neural network model, and then back propagation ends.

[0154] S909: Update the parameters of each layer in the neural network model based on the parameters of each layer and the gradient of the parameters.

[0155] After the back propagation process of S905a to S908a or the back propagation process of S905b to S908b is completed, the gradient of the parameters (weights, biases, etc.) of each layer in the neural network model is obtained. The parameters of each layer in the neural network model are updated based on the parameters, the gradient of the parameters, and the optimizer. For example, for a layer in the neural network model, a new weight is calculated based on the weight of the layer, the gradient of the weight, and the optimizer (calculated by the back propagation process), and the new weight is used to replace the original weight to achieve the weight update.

[0156] It should be noted that the steps in this embodiment describe a certain iteration process in the neural network model training process. In actual training, the neural network model needs to be trained many times before convergence.

[0157] It should also be noted that the quantized training method for the neural network model described in this application can be applied to one or more layers in the neural network model, and this application does not limit it. Generally speaking, it can be applied to layers with relatively large computational load, such as layers involving convolution operations and matrix multiplication operations, such as convolution layers, deconvolution layers, depthwise convolution layers, etc., and fully connected layers.

[0158] It can be seen that when the previous layer of the layer to be quantized depends on the first input data during back propagation, the first input data is saved, and the quantized first input data and quantized parameters of the intermediate data are released, thereby reducing the memory consumption of the intermediate data and increasing the available memory space; when the previous layer of the layer to be quantized does not depend on the first input data during back propagation, the first input data is released, and the quantized first input data and quantized parameters are saved. When back propagating to the previous layer of the layer to be quantized, it is convenient to directly use the quantized first input data and quantized parameters saved in the memory without the need for re-quantization, thereby reducing the amount of calculation and also reducing the memory consumption of the intermediate data generated by the calculation, thereby ensuring the normal training of the neural network model.

[0159] The present invention provides a method for testing a neural network model. Figure 12 The flowchart of the testing method of the neural network model shown is shown, and the method includes but is not limited to the description of the following content.

[0160] S1201. Obtain test data.

[0161] Acquire test data. For example, the test data may be an image or an image frame, or may be speech, etc.

[0162] S1202: Use the trained neural network model to test the test data.

[0163] use Figure 7 or Figure 9 In the method embodiment, the trained neural network model is tested on test data to obtain test results. The neural network model is obtained using a quantization training method, and during the training process, different quantization methods can be selected based on whether the previous layer of the quantized layer depends on the first input data of the quantized layer during backpropagation. The quantized layer generally refers to a layer with a large amount of computation, such as a convolutional layer, a deconvolutional layer, a fully connected layer, etc.

[0164] The present application also provides a test device for a neural network model. Figure 13 As shown in the schematic diagram, the testing device 1300 of the neural network model includes:

[0165] An acquisition unit 1301 is used to acquire test data;

[0166] The testing unit 1302 is used to test the test data using the trained neural network model. Figure 7 or Figure 9 The embodiment method shown is trained.

[0167] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teachings of the present disclosure, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present disclosure. In addition, depending on the different schemes, the description of some embodiments of the present disclosure also has different emphases. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present disclosure may also refer to the relevant descriptions of other embodiments.

[0168] In terms of specific implementation, based on the disclosure and teachings of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this document divides them based on the consideration of logical functions, and there may be other ways of division in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0169] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solution described in the embodiments of this disclosure. In addition, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically separately.

[0170] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program code, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0171] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0172] The foregoing content can be better understood in accordance with the following terms:

[0173] Item A1. A quantization training method for a neural network model, wherein the neural network model includes multiple layers, and the method includes: during forward propagation, obtaining first input data and parameters of the layer to be quantized; quantizing the first input data and the parameters respectively to obtain quantized first input data and quantized first parameters; performing operations on the quantized first input data and the quantized first parameters to obtain first output data; determining whether the previous layer of the layer to be quantized depends on the first input data during backward propagation; and if so, saving the first input data.

[0174] Clause A2. According to the method described in Clause A1, the operation on the quantized first input data and the quantized first parameter to obtain the first output data includes: operating on the quantized first input data and the quantized first parameter to obtain the quantized first output data; and dequantizing the quantized first output data to obtain the first output data.

[0175] Clause A3. According to the method described in Clause A1 or Clause A2, after saving the first input data, the method further includes: in the back propagation process, quantizing the second input data and the parameters of the layer to be quantized respectively to obtain quantized second input data and quantized second parameters; the second input data of the layer to be quantized includes the first input data; performing operations on the quantized second input data and the quantized second parameters to obtain quantized second output data and the gradient of the quantized second parameters; performing dequantization operations on the quantized second output data and the gradient of the quantized second parameters to obtain the gradient of the second output data and the second parameter; releasing the quantized second input data and the quantized second parameters from the memory.

[0176] Clause A4. According to the method described in Clause A3, after releasing the second quantized input data and the second quantized parameters from the memory, the method further includes: inputting the second output data and the first input data into the previous layer of the layer to be quantized; the previous layer of the layer to be quantized is a layer containing an operator.

[0177] Clause A5. According to the method described in Clause A1, the method also includes: when the previous layer of the layer to be quantized does not depend on the first input data during back propagation, saving the quantized first input data and the first quantized parameters.

[0178] Clause A6. According to the method described in Clause A5, the method also includes: performing a quantization operation on the third output data during the back propagation process to obtain quantized third output data; the third output data is the output data of the next layer of the layer to be quantized during the back propagation process; the quantized first input data, the quantized first parameter and the quantized third output data are input into the layer to be quantized, and operations are performed to obtain the quantized second output data and the gradient of the quantized first parameter; dequantization operations are performed on the gradient of the quantized second output data and the quantized first parameter to obtain the gradient of the second output data and the first parameter; and the quantized second input data and the quantized first parameter are released from the memory.

[0179] Clause A7. According to the method described in Clause A6, after releasing the second input data of the quantization and the first parameter of the quantization from the memory, the method further includes: inputting the second output data into the previous layer of the layer to be quantized; the previous layer of the layer to be quantized is a layer containing an operator.

[0180] Clause A8. According to the method described in Clause A3 or Clause A6, after the back propagation of the neural network model is completed, the method also includes: updating the parameters of each layer in the neural network model based on the parameters of each layer and the gradient of the parameters.

[0181] Item A9. The method according to any one of items A1-A8 is characterized in that the method is applicable to at least one layer in the neural network model.

[0182] Clause A10. A method for testing a neural network model, comprising: obtaining test data; testing the test data using a trained neural network model; the neural network model is trained using the method described in any one of clauses A1-A9.

[0183] Item A11. A quantization training device for a neural network model, wherein the neural network model includes multiple layers, and the device includes: an acquisition unit for acquiring first input data and parameters of the layer to be quantized during forward propagation; a quantization unit for quantizing the first input data and the parameters respectively to obtain quantized first input data and quantized first parameters; an operation unit for operating on the quantized first input data and the quantized first parameters to obtain first output data; a judgment unit for judging whether the previous layer of the layer to be quantized depends on the first input data during back propagation; and a storage unit for saving the first input data if the previous layer of the layer to be quantized depends on the first input data during back propagation.

[0184] Item A12, a neural network model quantitative training device, characterized in that it includes a memory and a processor, the memory is used to store instructions, and the processor is used to call the instructions to execute any one of the methods described in Items A1-A9.

[0185] Item A13. A computer storage medium, characterized in that it includes program instructions, which, when the program instructions are run on a computer, enable the computer to execute the method described in any one of Items A1-A9.

[0186] Although a plurality of embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may conceive of many modifications, changes, and alternatives without departing from the ideas and spirit of the present disclosure. It should be understood that in practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The appended claims are intended to define the scope of protection of the present disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A quantized training method for a neural network model, characterized in that: The neural network model includes multiple layers, and the method includes: During the forward propagation process, the first input data and parameters of the layer to be quantized are obtained; quantizing the first input data and the parameter respectively to obtain quantized first input data and quantized first parameter; performing an operation on the quantized first input data and the quantized first parameter to obtain first output data; Determining whether a previous layer of the layer to be quantized depends on the first input data during back propagation; If so, save the first input data; After saving the first input data, the method further includes: During the back propagation process, the second input data of the layer to be quantized and the parameter are quantized respectively to obtain the quantized second input data and the quantized second parameter; the second input data of the layer to be quantized includes the first input data; performing an operation on the quantized second input data and the quantized second parameter to obtain quantized second output data and a gradient of the quantized second parameter; performing a dequantization operation on the quantized second output data and the gradient of the quantized second parameter to obtain the second output data and the gradient of the second parameter; After obtaining the second output data of the layer to be quantized and the gradient of the second parameter, the quantized second input data and the quantized second parameter are released from the memory.

2. The method according to claim 1, characterized in that The performing operation on the quantized first input data and the quantized first parameter to obtain first output data includes: An operation is performed on the quantized first input data and the quantized first parameter to obtain quantized first output data; and dequantization is performed on the quantized first output data to obtain the first output data.

3. The method according to claim 1, characterized in that After releasing the quantized second input data and the quantized second parameter from the memory, the method further includes: The second output data and the first input data are input into a layer before the layer to be quantized; the layer before the layer to be quantized is a layer including an operator.

4. The method according to claim 1, characterized in that The method further comprises: When a previous layer of the layer to be quantized does not depend on the first input data during back propagation, the quantized first input data and the quantized first parameter are saved.

5. The method according to claim 4, characterized in that: The method further comprises: During the back propagation process, a quantization operation is performed on the third output data to obtain quantized third output data; the third output data is output data of a subsequent layer of the layer to be quantized during the back propagation process; Inputting the quantized first input data, the quantized first parameter, and the quantized third output data into the to-be-quantized layer, performing operations to obtain the quantized second output data and the gradient of the quantized first parameter; performing a dequantization operation on the quantized second output data and the gradient of the quantized first parameter to obtain the second output data and the gradient of the first parameter; After calculating the gradient of the second output data and the gradient of the first parameter, the quantized second input data and the quantized first parameter are released from the memory.

6. The method according to claim 5, characterized in that After releasing the quantized second input data and the quantized first parameter from the memory, the method further includes: The second output data is input into a layer before the layer to be quantized; the layer before the layer to be quantized is a layer including an operator.

7. The method according to claim 1 or 5, characterized in that After the back propagation of the neural network model is completed, the method further includes: updating the parameters of each layer in the neural network model based on the parameters of each layer and the gradient of the parameters.

8. The method according to claim 7, characterized in that: The method is applicable to at least one layer in the neural network model.

9. A method for testing a neural network model, characterized in that: include: Get test data; Testing the test data using the trained neural network model; The neural network model is obtained by training through the method according to any one of claims 1 to 8.

10. A quantized training device for a neural network model, characterized in that: The neural network model includes multiple layers, and the device includes: An acquisition unit, configured to acquire first input data and parameters of the layer to be quantized during forward propagation; a quantization unit, configured to quantize the first input data and the parameter respectively to obtain quantized first input data and quantized first parameter; an operation unit, configured to operate on the quantized first input data and the quantized first parameter to obtain first output data; a judging unit, configured to judge whether a previous layer of the layer to be quantized depends on the first input data during back propagation; A storage unit, configured to store the first input data when a previous layer of the layer to be quantized depends on the first input data during back propagation; The quantization unit is further configured to, during a back-propagation process, quantize the second input data of the layer to be quantized and the parameter, respectively, to obtain quantized second input data and quantized second parameters; the second input data of the layer to be quantized includes the first input data; The operation unit is further configured to perform an operation on the quantized second input data and the quantized second parameter to obtain quantized second output data and a gradient of the quantized second parameter; and perform an inverse quantization operation on the quantized second output data and the gradient of the quantized second parameter to obtain the second output data and the gradient of the second parameter. The quantization training device further comprises: a releasing unit (106) for releasing the quantized second input data and the quantized second parameter from the memory after calculating the gradient of the second output data and the gradient of the second parameter.

11. A neural network model quantization training device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store instructions, and the processor is used to call the instructions to execute the method according to any one of claims 1 to 8.

12. A computer storage medium, characterized in that The method comprises program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 8.

Images