Neural network training and data processing method and device, medium and computer equipment

Abstract

The present disclosure discloses a neural network training and data processing method and device, medium and computer equipment, the method comprises: inputting sample data into an original neural network, obtaining sample output parameters of each network layer in the original neural network; a loss function is established based on the difference between the sample output parameters of each network layer and the parameter boundary value corresponding to the preset quantization bit width of the network layer; the original neural network is trained based on the loss function to obtain a target neural network.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, medium, and computer equipment for neural network training and data processing. Background Technology

[0002] In recent years, neural networks have been widely used in artificial intelligence fields such as computer vision. As the performance requirements for neural networks become increasingly demanding, their structural designs tend to be deeper and wider. While this has led to improved accuracy in practical applications, it has also increased the demand for computing resources, such as memory and bandwidth, and has posed challenges to computing speed.

[0003] The scenarios for deploying neural networks in practical applications are becoming increasingly diverse, and the requirements for various devices, especially mobile devices, are correspondingly increasing in some scenarios. Currently, an efficient approach is to deploy neural networks on artificial intelligence chips, but due to limitations such as chip area and heat dissipation, the neural networks need to be compressed.

[0004] One efficient compression method is to quantize the neural network. A typical quantization approach maps the input and weight parameters of each layer of the neural network to a lower-bit representation space. However, the output parameters obtained after calculation using these mapped input and weight parameters may exceed the representation range of the mapped lower-bit space. Therefore, it is necessary to store the output parameters in a higher-bit space. Higher-bit spaces require higher-bit accumulators for computation, thus increasing bandwidth consumption and complicating hardware design. Summary of the Invention

[0005] In a first aspect, embodiments of this disclosure provide a neural network training method, the method comprising: inputting sample data into an original neural network to obtain sample output parameters of each network layer in the original neural network; establishing a loss function based on the difference between the sample output parameters of each network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer; and training the original neural network based on the loss function to obtain a target neural network.

[0006] Optionally, a truncated layer is provided between two adjacent network layers of the original neural network. Each truncated layer includes an activation function, which is used to truncate the sample output parameters of the previous network layer in the two adjacent network layers, and input the truncated sample output parameters into the next network layer in the two adjacent network layers.

[0007] Optionally, the sample output parameters of a network layer in the original neural network are obtained based on the quantized sample input parameters and the quantized weight parameters of the network layer; the quantized sample input parameters of the network layer are obtained by quantizing the sample input parameters of the network layer based on the preset quantization bit width of the network layer; the quantized weight parameters of the network layer are obtained by quantizing the weight parameters of the network layer based on the preset quantization bit width of the network layer; the sample output parameters of each network layer of the original neural network are truncated to the parameter boundary value corresponding to the preset quantization bit width of the network layer and used as the sample input parameters of the next network layer.

[0008] Optionally, the method further includes: inputting the data to be processed into the target neural network for forward propagation to obtain the target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer; and performing inverse mapping on the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network; wherein, the output scaling factor of the output layer of the target neural network is obtained by using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network, and performing forward propagation in the target neural network.

[0009] Optionally, when the number of input parameters of the target network layer of the target neural network is equal to 1, the method further includes: determining the output scale factor of the target network layer based on the input scale factor of the target network layer and the weight scale factor of the target network layer; wherein the input scale factor of the target network layer is equal to the output scale factor of the network layer preceding the target network layer.

[0010] Optionally, the method further includes: when the number of input parameters of the target network layer of the target neural network is equal to 1, a scaling factor layer is provided between the target network layer and the next network layer of the target network layer, including the output scaling factor of the network layer.

[0011] Optionally, if the number of input parameters of the target network layer of the target neural network is greater than 1, the output scaling factor of the previous network layer of the target network layer is used as the output scaling factor of the target network layer.

[0012] Optionally, when the number of input parameters of the target network layer of the target neural network is greater than 1, each input parameter corresponds to an input scale factor, and the method further includes: normalizing each input scale factor of the target network layer to obtain a normalized input scale factor corresponding to each input scale factor of the target network layer; and mapping each corresponding input parameter through each normalized scale factor.

[0013] Optionally, the step of normalizing each input scaling factor of the target network layer to obtain the normalized input scaling factor corresponding to each input scaling factor of the target network layer includes: obtaining the minimum value of each input scaling factor of the target network layer; and determining the ratio of each input scaling factor of the target network layer to the minimum value as the normalized input scaling factor corresponding to each input scaling factor of the target network layer.

[0014] Optionally, the method further includes: for any target network layer in the target neural network, inputting the input parameters of the target network layer and the weight parameters of the network layer into the multiplier corresponding to the network layer respectively, to obtain the product of the input parameters of the target network layer and the weight parameters of the target network layer; inputting the product and the bias parameters of the target network layer into the adder corresponding to the target network layer respectively, to obtain the sum of the product and the bias parameters of the target network layer; the bit width of the multiplier and the adder corresponding to the target network layer are both equal to the preset quantization bit width of the target network layer.

[0015] Secondly, embodiments of this disclosure provide a data processing method, the method comprising: inputting data to be processed into a target neural network for forward propagation to obtain target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer; performing inverse mapping on the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network; wherein, the output scaling factor of the output layer of the target neural network is obtained by forward propagation in the target neural network using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network; the target neural network is trained based on the difference between the output parameters of each network layer of the original neural network and the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding network layer.

[0016] Thirdly, embodiments of this disclosure provide a neural network training apparatus, the apparatus comprising: an acquisition module, configured to input sample data into an original neural network and acquire sample output parameters of each network layer in the original neural network; a loss function establishment module, configured to establish a loss function based on the difference between the sample output parameters of each network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer; and a training module, configured to train the original neural network based on the loss function to obtain a target neural network.

[0017] Optionally, a truncated layer is provided between two adjacent network layers of the original neural network. Each truncated layer includes an activation function, which is used to truncate the sample output parameters of the previous network layer in the two adjacent network layers, and input the truncated sample output parameters into the next network layer in the two adjacent network layers.

[0018] Optionally, the sample output parameters of a network layer in the original neural network are obtained based on the quantized sample input parameters and the quantized weight parameters of the network layer; the quantized sample input parameters of the network layer are obtained by quantizing the sample input parameters of the network layer based on the preset quantization bit width of the network layer; the quantized weight parameters of the network layer are obtained by quantizing the weight parameters of the network layer based on the preset quantization bit width of the network layer; the sample output parameters of each network layer of the original neural network are truncated to the parameter boundary value corresponding to the preset quantization bit width of the network layer and used as the sample input parameters of the next network layer.

[0019] Optionally, the apparatus further includes: a first input module, configured to input the data to be processed into the target neural network for forward propagation to obtain the target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer; and an inverse mapping module, configured to inverse map the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network; wherein the output scaling factor of the output layer of the target neural network is obtained by using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network, and performing forward propagation in the target neural network.

[0020] Optionally, when the number of input parameters of the target network layer of the target neural network is equal to 1, the device further includes: an output scale factor determination module, used to determine the output scale factor of the target network layer based on the input scale factor of the target network layer and the weight scale factor of the target network layer; wherein, the input scale factor of the target network layer is equal to the output scale factor of the network layer preceding the target network layer.

[0021] Optionally, when the number of input parameters of the target network layer of the target neural network is equal to 1, a scaling factor layer is provided between the target network layer and the next network layer of the target network layer, including the output scaling factor of the network layer.

[0022] Optionally, if the number of input parameters of the target network layer of the target neural network is greater than 1, the output scaling factor of the previous network layer of the target network layer is used as the output scaling factor of the target network layer.

[0023] Optionally, when the number of input parameters of the target network layer of the target neural network is greater than 1, each input parameter corresponds to an input scale factor. The device further includes: a normalization module, used to normalize each input scale factor of the target network layer to obtain a normalized input scale factor corresponding to each input scale factor of the target network layer; and a mapping module, used to map each corresponding input parameter through each normalized scale factor.

[0024] Optionally, the normalization module is used to: obtain the minimum value of each input scale factor of the target network layer; and determine the ratio of each input scale factor of the target network layer to the minimum value as the normalized input scale factor corresponding to each input scale factor of the target network layer.

[0025] Optionally, the device further includes: a second input module, configured to input the input parameters of the target network layer and the weight parameters of the network layer into the multiplier corresponding to the network layer for any target network layer in the target neural network, respectively, to obtain the product of the input parameters of the target network layer and the weight parameters of the target network layer; a third input module, configured to input the product and the bias parameters of the target network layer into the adder corresponding to the target network layer, respectively, to obtain the sum of the product and the bias parameters of the target network layer; the bit width of the multiplier and the adder corresponding to the target network layer are both equal to the preset quantization bit width of the target network layer.

[0026] Fourthly, embodiments of this disclosure provide a data processing apparatus, the apparatus comprising: an input module, configured to input data to be processed into a target neural network for forward propagation to obtain target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer; and an inverse mapping module, configured to inverse map the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network; wherein, the output scaling factor of the output layer of the target neural network is obtained by forward propagation in the target neural network using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network; and the target neural network is trained based on the difference between the output parameters of each network layer of the original neural network and the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding network layer.

[0027] Fifthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described in any of the embodiments.

[0028] In a sixth aspect, embodiments of this disclosure provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any embodiment.

[0029] This embodiment of the disclosure determines the output parameters of the corresponding network layer based on the quantized input parameters and quantized weight parameters of each network layer in the original neural network. Since the output parameters may exceed the parameter boundary values ​​corresponding to the preset quantization bit width, a loss function is established based on the difference between the output parameters of each network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer to train the original neural network. This reduces the dependence of the original neural network on parameters outside the range during training. In this way, it can be ensured that the output parameters of each network layer do not exceed the parameter boundary values ​​corresponding to the preset quantization bit width of that network layer, thus eliminating the need for high-bit accumulators, while also ensuring that the neural network achieves high performance.

[0030] This disclosure trains a target neural network based on the difference between the output parameters of each layer of the original neural network and the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding layer. This ensures that the output parameters of each layer of the target neural network do not exceed the parameter boundary values ​​corresponding to the preset quantization bit width of that layer. Therefore, during the forward propagation of the data to be processed, there is no need to reverse map the output parameters of each layer. Only the output parameters of the output layer of the target neural network need to be reverse mapped, which reduces the number of reverse mappings and thus improves the data processing efficiency. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0032] Figure 1 This is a flowchart of a neural network training method according to an embodiment of the present disclosure.

[0033] Figure 2A This is a schematic diagram of the network structure according to an embodiment of the present disclosure.

[0034] Figure 2B This is a schematic diagram of the parameters before and after truncation in an embodiment of this disclosure.

[0035] Figure 3A This is a schematic diagram of the mapping and inverse mapping processes in the quantization process of related technologies.

[0036] Figure 3B This is a schematic diagram of the mapping process according to an embodiment of the present disclosure.

[0037] Figure 4 This is a flowchart of a data processing method according to an embodiment of the present disclosure.

[0038] Figure 5 This is a block diagram of a neural network training apparatus according to an embodiment of the present disclosure.

[0039] Figure 6 This is a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0040] Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art should fall within the scope of protection of this disclosure.

[0042] Neural networks (e.g., convolutional neural networks) can extract robust and rich high-dimensional features, significantly improving cognitive abilities and finding wide application in artificial intelligence fields such as computer vision. To reduce the resource consumption of neural networks, quantization processing is often required. This quantization processing includes at least one of the following: quantizing the input parameters of the neural network to obtain quantized input parameters; quantizing the weight parameters of the neural network to obtain quantized weight parameters; and quantizing the output parameters of the neural network to obtain quantized output parameters.

[0043] A common quantization method maps the input and weight parameters of each layer of a neural network to a lower bit space. However, the output parameters obtained after calculation using these mapped input and weight parameters may exceed the representation range of the mapped lower bit space. For example, 8-bit quantization maps the input and weight parameters to values ​​between 0 and 255; however, the output parameters may have values ​​greater than 255. This necessitates using more bits to represent the quantized output parameters, increasing bandwidth consumption and complicating hardware design.

[0044] To address the aforementioned problems, this disclosure proposes a neural network training method. See [link to relevant documentation]. Figure 1 The method includes the following steps: Step 101: Input the sample data into the original neural network and obtain the sample output parameters of each network layer in the original neural network.

[0045] Step 102: Establish a loss function based on the difference between the sample output parameters of each network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer.

[0046] Step 103: Train the original neural network based on the loss function to obtain the target neural network.

[0047] The neural network training method of this disclosure can be used in the neural network training stage to train an untrained neural network using sample data and the aforementioned loss function. It can also be used after the neural network pre-training stage to fine-tune a pre-trained neural network using sample data and the aforementioned loss function. Therefore, in step 101 above, the original neural network can be either a pre-trained neural network or an untrained neural network. The pre-training and fine-tuning processes of the neural network can use different sample data.

[0048] In some embodiments, the sample data is a sample image, and the original neural network can extract features from the sample image. The sample output parameters of each network layer in the original neural network are output feature maps. In image application scenarios, embodiments of this disclosure can input the sample image into the original neural network to obtain the output feature maps of each network layer in the original neural network. A loss function is established based on the difference between the pixel values ​​of the output feature maps of each network layer and the pixel value boundary values ​​corresponding to the preset quantization bit width of the network layer. The original neural network is trained based on the loss function to obtain the target neural network.

[0049] The original neural network can be a convolutional neural network or other types of neural networks, without limitation. The neural network may include multiple network layers, including an input layer, intermediate layers, and an output layer. The input layer is the first network layer of the neural network, the output layer is the last network layer, and the intermediate layers are the network layers between the input layer and the intermediate layers. It should be noted that, in this disclosure, during the training phase, the input parameters and output parameters of each network layer of the original neural network are respectively the sample input parameters and sample output parameters, where the sample input parameters of the input layer of the original neural network are the sample data. After training to obtain the target neural network, the input parameters and output parameters of each network layer of the target neural network are respectively the target input parameters and target output parameters, where the target input parameters of the input layer of the target neural network are the data to be processed (e.g., the image to be processed). For ease of description, sometimes the sample input parameters and target input parameters are collectively referred to as input parameters, and the sample output parameters and target output parameters are collectively referred to as output parameters.

[0050] In the original neural network, the output parameters of each network layer are obtained based on the quantized input parameters and quantized weight parameters of that layer. Specifically, the quantized input parameters of each network layer are obtained by quantizing the input parameters of that layer based on its preset quantization bit width, and the quantized weight parameters of each network layer are obtained by quantizing the weight parameters of that layer based on its preset quantization bit width. The preset quantization bit widths of different network layers can be the same or different.

[0051] Taking the case where each network layer has a preset quantization bit width of 8 bits as an example, assuming the input parameter of a network layer is 252.02 and the weight parameter is 2.03, we can round the input parameter and weight parameter to the nearest integer, resulting in a quantized input parameter of 252 and a quantized weight parameter of 2. Of course, the quantization method is not limited to rounding; different quantization bit widths can be assigned to the integer and fractional parts respectively, which is not restricted here. Then, the product of the quantized input parameter and the quantized weight parameter, plus the preset bias (assuming 5), can be used to determine the output parameter, resulting in an output parameter of 509. In related technologies, since the value 509 exceeds the range (0~255) that can be represented by an 8-bit quantization bit width, a quantization bit width greater than 8 bits is required to represent the output parameter.

[0052] In this embodiment, the output parameters of each network layer can be truncated to the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer. The parameter boundary values ​​corresponding to the preset quantization bit width include an upper boundary and a lower boundary. Output parameters exceeding the upper boundary can be truncated to the upper boundary, and output parameters exceeding the lower boundary can be truncated to the lower boundary. In the example above, the value 509 exceeds the upper boundary corresponding to the 8-bit quantization bit width; therefore, 509 is truncated to the upper boundary 255. Simultaneously, 255 is used as the input parameter for the next network layer.

[0053] However, directly performing the above truncation process may lead to a significant decrease in the performance of the neural network. Therefore, to address this performance degradation, in step 102, a loss function can be established based on the difference between the output parameters of the network layer before truncation and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer. This loss function can be added as a penalty term to other loss functions, and the final loss function used to train the neural network is the sum of the other loss functions and the penalty term. Alternatively, other loss functions can be used for pre-training to obtain the original neural network, and then the original neural network can be retrained using this loss function.

[0054] The other loss functions can be determined based on the function performed by the neural network. For example, if the function of the neural network is classification, the other loss functions can be used to characterize the classification loss of the neural network. Optionally, the loss function characterizing the classification loss can be the cross-entropy loss function. If the function of the neural network is object detection, the other loss functions may include a loss function characterizing the classification loss of the neural network and a loss function characterizing the deviation between the output position of the neural network and the true position of the target object (e.g., the cross-union loss function). By adding a penalty term during training, the value of the output parameters can be used as a factor in the training of the neural network, so that the output parameters of each network layer of the target neural network obtained after training do not exceed the parameter boundary value corresponding to the preset quantization bit width of that network layer as much as possible.

[0055] like Figure 2A As shown, a truncating layer can be placed between two adjacent network layers of the original neural network to truncate the output parameters of the previous network layer in the two adjacent network layers, and input the truncated output parameters into the next network layer in the two adjacent network layers. (See figure) n 1. n 2. n 3. n N-1 , n N This indicates the number of the network layers in the original neural network, which includes a total of [number of layers]. N Each network layer, r 1. r 2. r 3. r N-1 Representing network layers n 1. n 2. n 3. n N-1 The subsequent cutoff layers are numbered. Each cutoff layer may include an activation function, which in some embodiments may be expressed in the following form: ; Among them, S 上 and S 下 These represent the upper and lower bounds of the parameters corresponding to the preset quantization bit width, respectively. Different truncated layers can use different activation functions, which are determined based on the preset quantization bit width of the preceding network layer to which the truncated layer is connected. For example, the activation function of the truncated layer numbered r1 is determined based on the preset quantization bit width of the network layer numbered n1, and the activation function of the truncated layer numbered r2 is determined based on the preset quantization bit width of the network layer numbered n2.

[0056] like Figure 2B The image shows a comparison of the output parameters before and after truncation when the preset quantization bit width is 8. It can be seen that output parameters greater than 255 are all truncated to 255, while output parameters not greater than 255 remain the same. Similarly, output parameters less than 0 are truncated to 0, while output parameters not less than 0 remain the same. After the training phase ends and the target neural network is obtained, the truncated layer can be removed.

[0057] This embodiment determines the output parameters of each network layer based on the quantized input parameters and quantized weight parameters of each network layer in the original neural network. Since the output parameters may exceed the parameter boundary values ​​corresponding to the preset quantization bit width, the output parameters are truncated before being used as the input to the next network layer. Furthermore, a loss function is established based on the difference between the output parameters of each network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of that network layer to train the original neural network. This reduces the original neural network's dependence on out-of-range parameters during training. In this way, it is ensured that the output parameters of each network layer do not exceed the parameter boundary values ​​corresponding to the preset quantization bit width of that network layer, thus eliminating the need for high-bit accumulators, while also ensuring high performance of the neural network.

[0058] After obtaining the target neural network, the data to be processed can be processed through the target neural network. In related technologies, it is necessary to map the input parameters of each layer of the target neural network through an input scaling factor, map the weight parameters of each layer of the target neural network through a weight scaling factor, and inversely map the output parameters of each layer of the target neural network through an output scaling factor. The function of each scaling factor is to map the corresponding parameters to the parameter boundary values ​​corresponding to a preset quantization bit width, so that each parameter can be quantized using the preset quantization bit width. In some embodiments, the ratio of the upper boundary corresponding to the preset quantization bit width of a network layer to the input parameter with the largest value in that network layer can be determined as the input scaling factor of that network layer. The method of determining the weight scaling factor and the output scaling factor is similar to the method of determining the input scaling factor.

[0059] The input scaling factor, weight scaling factor, and output scaling factor satisfy a certain relationship. Taking a convolutional layer as an example, such as... Figure 3A As shown, x 1. w 1 and y 1 represents the number as n The input parameters, weight parameters, and output parameters of a network layer of 1. express x The input scaling factor corresponding to 1, express w The corresponding weight scaling factor is 1. express y The output scaling factor corresponding to 1. First, the input scaling factor is used. For input parameters x 1. Perform mapping (i.e., map) Multiply x 1), and use weighted scaling factors. For weight parameters w 1. Perform mapping (i.e., map) Multiply w 1). The quantized convolution process can be represented as: ; in, x Generally refers to the input parameters of any network layer. w for x The weight parameters of the network layer, y for x The output parameters of the network layer in which it is located S x This represents the input scale factor corresponding to the input parameter x. S w Represents weight parameters w The corresponding weighting scaling factor, S y This represents the output scaling factor corresponding to the output parameter y.

[0060] According to the formula above, the number is... n The output of a network layer of size 1 is the product of the output parameters and the output scale factor, i.e. Therefore, it is necessary to output the results. Divide by the output scaling factor of this network layer This process, known as inverse mapping, yields the output parameters of the network layer. Then, the output parameters of this network layer are used as the parameters for the next network layer (numbered as follows). n The input parameters of the network layer 2 are then processed, and the above process is repeated until each network layer of the target neural network has been processed.

[0061] As can be seen, the above process requires continuous mapping and demapping of each network layer, which is very cumbersome. To solve the above problem, in this embodiment, the data to be processed is input into the target neural network for forward propagation to obtain the target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer; the target output parameters of the output layer of the target neural network are demapped based on the output scaling factor of the output layer of the target neural network; wherein, the output scaling factor of the output layer of the target neural network is obtained by using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network, and performing forward propagation in the target neural network.

[0062] In some embodiments, when the number of input parameters of the target network layer of the target neural network is equal to 1, the output scale factor of the target network layer can be determined based on the input scale factor and the weight scale factor of the target network layer; wherein, the input scale factor of the target network layer is equal to the output scale factor of the network layer preceding the target network layer.

[0063] Based on the above formula, we can deduce that the input parameters and output parameters of a network layer satisfy the following relationship: ; Therefore, the output scaling factor of this network layer is: .

[0064] The scaling factor input to the first layer of the network is used as the network input for forward propagation (i.e., extracting the scaling factor from the original quantization forward process) until the output of the target neural network's output layer is obtained. The output of the last layer of the network is used as the new output factor. S o The new output factor is equivalent to the scale factor of the final output layer in the original quantization process.

[0065] like Figure 3B As shown, through the above processing, in each network layer of the target neural network except the last one, there is no need to map the input parameters or reverse-map the output parameters. Instead, the input parameters are directly propagated forward, and the resulting output parameters are directly used as the input to the next network layer, until the output of the last network layer is obtained. The output of the last network layer is divided by... S oThen you can get the final output result.

[0066] It should be noted that during the forward propagation of the data to be processed, the input parameters and weight parameters of each network layer of the target neural network still need to be mapped through the corresponding scaling factor before quantization. However, after processing using the method of this embodiment, it is not necessary to perform inverse mapping on the output parameters of each network layer; instead, only the output parameters of the output layer of the target neural network need to be inverse mapped. This reduces the number of inverse mapping operations, thereby improving processing efficiency. It should be noted that because the neural network is trained using the above-described training method in this embodiment, the neural network can keep the output parameters of each network layer as close as possible to the parameter boundaries corresponding to the preset quantization bit width. Therefore, it is not necessary to perform inverse mapping on the output parameters of each layer; only the output layer of the target neural network needs to be inverse mapped once.

[0067] In some embodiments, when the number of input parameters of the target network layer of the target neural network is equal to 1, a scale layer can be set between the target network layer and the next network layer of the target network layer. The scale layer connected to the output of each network layer may include the output scale factor of the network layer, and the output scale factor of each scale layer can participate in the forward propagation of the scale factor.

[0068] In other embodiments, when the number of input parameters of the target network layer of the target neural network is greater than 1, the output scaling factor of the previous network layer of the target network layer can be used as the output scaling factor of the target network layer.

[0069] For example, suppose the first i If the number of input parameters to a network layer is greater than 1, then during scale factor forward propagation, the first... i The network layer does not participate in the forward propagation of the scale factor, but directly transmits the data to the first layer. i -1 is the output scaling factor of the network layer, which is then input to the... i +1 network layer, thus making the first i -1 is the output scaling factor of the network layer as the first i +1 layer input scaling factor.

[0070] Furthermore, when the number of input parameters of the target network layer of the target neural network is greater than 1, since forward propagation of the input parameters of the target network layer is still required, each input scale factor of the target network layer can be normalized to obtain the normalized input scale factor corresponding to each input scale factor of the target network layer.

[0071] In some embodiments, the minimum value of each input scaling factor of the target network layer can be obtained; the ratio of each input scaling factor of the target network layer to the minimum value is determined as the normalized input parameter corresponding to each input parameter of the target network layer.

[0072] For example, suppose the multiple input scale factors of the i-th network layer are respectively S 1, S 2, ..., S k Then it can be determined S 1, S 2, ..., S k The minimum value is denoted as S min and will S 1, S 2, ..., S k Normalized to S 1 / S min , S 2 / S min , ..., S k / S min .

[0073] In other embodiments, normalization can be performed in other ways. For example, the maximum value of each input scale factor of the target network layer can be obtained; the ratio of each input scale factor of the target network layer to the maximum value can be determined as the normalized input scale factor corresponding to each input scale factor of the target network layer.

[0074] In some embodiments, for any target network layer in the target neural network, the input parameters and weight parameters of the target network layer can be respectively input into the multiplier corresponding to the network layer to obtain the product of the input parameters and weight parameters of the target network layer; the product and the bias parameter of the target network layer can be respectively input into the adder corresponding to the target network layer to obtain the sum of the product and the bias parameter of the target network layer; the bit width of the multiplier and the adder corresponding to the target network layer are both equal to the preset quantization bit width of the target network layer.

[0075] In related technologies, since the output parameters of a network layer often require a larger bit width than the input parameters, the bit width of the multipliers and adders corresponding to that network layer is often greater than the preset quantization bit width of the network layer. However, in the embodiments of this disclosure, since the output parameters of the neural network have been constrained within the parameter boundary values ​​corresponding to the preset quantization bit width through training, multipliers and adders with a bit width equal to the preset quantization bit width of the network layer can be used to process the input parameters and weight parameters of the network layer. For example, if the preset quantization bit width used for the input parameters and weight parameters of a network layer is 8, then the bit width of the multipliers and adders corresponding to that network layer is also equal to 8, thereby improving computational efficiency and reducing hardware bandwidth consumption.

[0076] The overall flow of the embodiments of this disclosure will be described below through a specific example: Taking 8-bit quantization and convolutional layers as an example, during quantization, the input parameters and weight parameters need to be limited to the range of 0-255. First, the neural network is trained using sample data. Specifically, an activation function is added to the output of each layer. This activation function truncates parameters outside the 0-255 range, resulting in numbers within that range. Simultaneously, a penalty term is added to the loss function during training. This penalty term represents the magnitude of each layer's output exceeding the boundary value of 0 or 255, causing the neural network to lose its dependence on parameters outside the 0-255 range during training.

[0077] Based on the above operations, the output parameters of the neural network can be directly saved as 8 bits during quantization, reducing the impact of the overflow of the neural network output parameters. Furthermore, the bias term can also be quantized to 8 bits during training to adapt to a smaller bit width, so that an 8-bit accumulator can be used during deployment.

[0078] To eliminate the need for de-mapping each network layer during quantization of the neural network trained in the above manner, a forward propagation of the scale factor is performed before quantization. The input scale factor of the first layer of the neural network is used as the input of the neural network for forward propagation. Finally, the output of the last layer of the neural network is obtained and used as the target output factor. Based on these operations, quantization only requires mapping each network layer and de-mapping the output layer to obtain the original quantized result of the neural network, reducing the de-mapping operation and saving computational resources.

[0079] When quantizing the neural network, the input parameters and weight parameters of the neural network are input into an 8-bit wide multiplier, and the output and bias term of the multiplier are input into an 8-bit wide adder. The quantization bit width of the entire neural network is consistent, and the bit width of the accumulator and / or multiplier is consistent, which results in high computational efficiency and reduced bandwidth consumption.

[0080] See Figure 4 This disclosure also provides a data processing method, which may include: Step 401: Input the data to be processed into the target neural network for forward propagation to obtain the target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer. Step 402: Perform inverse mapping on the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network; The output scaling factor of the output layer of the target neural network is obtained by forward propagation in the target neural network using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network. The target neural network is trained based on the difference between the output parameters of each network layer of the original neural network and the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding network layer.

[0081] In this embodiment, the input parameters, weight parameters, and output parameters of each network layer of the target neural network can be quantized using a preset quantization bit width for that network layer. The quantized output parameters are then used as the input parameters for the next layer. Specifically, the input parameters are mapped to the parameter boundary value range corresponding to the preset quantization bit width using an input scaling factor before quantization. Similarly, the weight parameters are mapped to the parameter boundary value range corresponding to the preset quantization bit width using a weight scaling factor before quantization. In related technologies, the output parameters of each network layer undergo inverse mapping.

[0082] In this disclosure, the target neural network is trained based on the difference between the output parameters of each layer of the original neural network and the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding layer. This ensures that the output parameters of each layer of the target neural network do not exceed the parameter boundary values ​​corresponding to the preset quantization bit width of that layer. Therefore, during the forward propagation of the data to be processed, there is no need to reverse map the output parameters of each layer; only the output parameters of the output layer of the target neural network need to be reverse mapped. This reduces the number of reverse mapping operations and thus improves data processing efficiency.

[0083] The target neural network of this disclosure embodiment can be trained using the neural network training method in any of the foregoing embodiments. For specific details, please refer to the embodiments of the foregoing training methods, which will not be repeated here.

[0084] Corresponding to the above method embodiments, this disclosure also provides a neural network training device, see [link to relevant documentation]. Figure 5 As shown, the device includes: The acquisition module 501 is used to input sample data into the original neural network and acquire the sample output parameters of each network layer in the original neural network. The loss function establishment module 502 is used to establish a loss function based on the difference between the sample output parameters of each network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer. The training module 503 is used to train the original neural network based on the loss function to obtain the target neural network.

[0085] Optionally, the loss function establishment module is used to: sum the absolute values ​​of the differences between the sample output parameters of each network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer to obtain the loss function.

[0086] Optionally, different network layers in the original neural network correspond to different preset quantization bit widths; the loss function establishment module is used to: for each network layer in the original neural network, sum the absolute values ​​of the differences between the output parameters of each sample of the network layer and the parameter boundary values ​​corresponding to the preset quantization bit width of the network layer to obtain the total difference of the network layer; sum the total differences of each network layer of the original neural network to obtain the loss function.

[0087] Optionally, a truncated layer is provided between two adjacent network layers of the original neural network. Each truncated layer includes an activation function, which is used to truncate the sample output parameters of the previous network layer in the two adjacent network layers, and input the truncated sample output parameters into the next network layer in the two adjacent network layers.

[0088] Optionally, the sample output parameters of a network layer in the original neural network are obtained based on the quantized sample input parameters and the quantized weight parameters of the network layer; the quantized sample input parameters of the network layer are obtained by quantizing the sample input parameters of the network layer based on the preset quantization bit width of the network layer; the quantized weight parameters of the network layer are obtained by quantizing the weight parameters of the network layer based on the preset quantization bit width of the network layer; the sample output parameters of each network layer of the original neural network are truncated to the parameter boundary value corresponding to the preset quantization bit width of the network layer and used as the sample input parameters of the next network layer.

[0089] Optionally, the apparatus further includes: a first input module, configured to input the data to be processed into the target neural network for forward propagation to obtain the target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer; and an inverse mapping module, configured to inverse map the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network; wherein the output scaling factor of the output layer of the target neural network is obtained by using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network, and performing forward propagation in the target neural network.

[0090] Optionally, when the number of input parameters of the target network layer of the target neural network is equal to 1, the device further includes: an output scale factor determination module, used to determine the output scale factor of the target network layer based on the input scale factor of the target network layer and the weight scale factor of the target network layer; wherein, the input scale factor of the target network layer is equal to the output scale factor of the network layer preceding the target network layer.

[0091] Optionally, when the number of input parameters of the target network layer of the target neural network is equal to 1, a scaling factor layer is provided between the target network layer and the next network layer of the target network layer, including the output scaling factor of the network layer.

[0092] Optionally, if the number of input parameters of the target network layer of the target neural network is greater than 1, the output scaling factor of the previous network layer of the target network layer is used as the output scaling factor of the target network layer.

[0093] Optionally, if the number of input parameters of the target network layer of the target neural network is greater than 1, the device further includes: a normalization module, used to normalize each input scale factor of the target network layer to obtain a normalized input scale factor corresponding to each input scale factor of the target network layer; and a mapping module, used to map each input parameter of the target network layer through the normalized scale factor of the target network layer.

[0094] Optionally, the normalization module is used to: obtain the minimum value of each input scale factor of the target network layer; and determine the ratio of each input scale factor of the target network layer to the minimum value as the normalized input scale factor corresponding to each input scale factor of the target network layer.

[0095] Optionally, the device further includes: a second input module, configured to input the input parameters of the target network layer and the weight parameters of the network layer into the multiplier corresponding to the network layer for any target network layer in the target neural network, respectively, to obtain the product of the input parameters of the target network layer and the weight parameters of the target network layer; a third input module, configured to input the product and the bias parameters of the target network layer into the adder corresponding to the target network layer, respectively, to obtain the sum of the product and the bias parameters of the target network layer; the bit width of the multiplier and the adder corresponding to the target network layer are both equal to the preset quantization bit width of the target network layer.

[0096] See Figure 6 This disclosure also provides a data processing apparatus, the apparatus comprising: The input module 601 is used to input the data to be processed into the target neural network for forward propagation, and obtain the target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer. The inverse mapping module 602 is used to inverse map the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network; wherein, the output scaling factor of the output layer of the target neural network is obtained by forward propagation in the target neural network using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network; the target neural network is trained based on the difference between the output parameters of each network layer of the original neural network and the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding network layer.

[0097] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0098] This specification also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the foregoing embodiments.

[0099] Figure 7 This diagram illustrates a more specific hardware structure of a computing device provided in an embodiment of this specification. The device may include a processor 701, a memory 702, an input / output interface 703, a communication interface 704, and a bus 705. The processor 701, memory 702, input / output interface 703, and communication interface 704 are interconnected internally via the bus 705.

[0100] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor 701 may also include a graphics card, such as an Nvidia Titan X graphics card or a 1080Ti graphics card.

[0101] The memory 702 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701.

[0102] The input / output interface 703 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0103] The communication interface 704 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0104] Bus 705 includes a pathway for transmitting information between various components of the device, such as processor 701, memory 702, input / output interface 703, and communication interface 704.

[0105] It should be noted that although the above-described device only shows the processor 701, memory 702, input / output interface 703, communication interface 704, and bus 705, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0106] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0107] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0108] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this specification, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this specification.

[0109] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0110] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0111] The above description is merely a specific implementation of the embodiments of this specification. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this specification, and these improvements and modifications should also be considered within the protection scope of the embodiments of this specification.

Claims

1. A neural network training method, characterized in that, The method includes: Sample data is input into the original neural network to obtain the sample output parameters of each network layer in the original neural network. A truncated layer is set between two adjacent network layers in the original neural network. Each truncated layer includes an activation function to truncate the sample output parameters of the previous network layer in the two adjacent network layers, and inputs the truncated sample output parameters into the next network layer in the two adjacent network layers. The truncated sample output parameters are within the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding network layer. The activation function of the truncated layer is determined based on the preset quantization bit width of the previous network layer connected to the truncated layer. The truncated layer is removed from the neural network after training. A loss function is established based on the difference between the sample output parameters of each network layer and the parameter boundary value corresponding to the preset quantization bit width of the network layer; the loss function includes a penalty term, which is used to ensure that the output parameters of each network layer of the trained target neural network do not exceed the parameter boundary value corresponding to the preset quantization bit width of the network layer. The original neural network is trained based on the loss function to obtain the target neural network; The sample data is a sample image. The original neural network is used to extract features from the sample image. The sample output parameters of each network layer in the original neural network are output feature maps. The loss function is established based on the difference between the pixel value of the output feature map of each network layer and the pixel value boundary value corresponding to the preset quantization bit width of the network layer.

2. The method according to claim 1, characterized in that, The sample output parameters of a network layer in the original neural network are obtained based on the quantized sample input parameters and the quantized weight parameters of the network layer. The quantized sample input parameters of the network layer are obtained by quantizing the sample input parameters of the network layer based on the preset quantization bit width of the network layer; The quantization weight parameters of the network layer are obtained by quantizing the weight parameters of the network layer based on the preset quantization bit width of the network layer; The sample output parameters of each network layer of the original neural network are truncated to the parameter boundary value corresponding to the preset quantization bit width of the network layer, and then used as the sample input parameters of the next network layer.

3. The method according to claim 1, characterized in that, The method further includes: The data to be processed is input into the target neural network for forward propagation to obtain the target output parameters of the output layer of the target neural network; during the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer. The target output parameters of the output layer of the target neural network are inversely mapped based on the output scaling factor of the output layer of the target neural network. The output scaling factor of the output layer of the target neural network is obtained by using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network, and then performing forward propagation in the target neural network.

4. The method according to claim 3, characterized in that, When the number of input parameters of the target network layer of the target neural network is equal to 1, the method further includes: The output scaling factor of the target network layer is determined based on the input scaling factor and the weight scaling factor of the target network layer. Wherein, the input scaling factor of the target network layer is equal to the output scaling factor of the network layer preceding the target network layer.

5. The method according to claim 3, characterized in that, When the number of input parameters of the target network layer of the target neural network is equal to 1, a scaling factor layer is provided between the target network layer and the next network layer of the target network layer, including the output scaling factor of the network layer.

6. The method according to claim 3, characterized in that, If the number of input parameters of the target network layer of the target neural network is greater than 1, the output scaling factor of the network layer preceding the target network layer is used as the output scaling factor of the target network layer.

7. The method according to claim 3, characterized in that, When the number of input parameters of the target network layer of the target neural network is greater than 1, each input parameter corresponds to an input scale factor, and the method further includes: Normalize each input scale factor of the target network layer to obtain the normalized input scale factor corresponding to each input scale factor of the target network layer; Each input parameter is mapped to its corresponding normalized input scale factor.

8. The method according to claim 7, characterized in that, The normalization process for each input scaling factor of the target network layer to obtain the normalized input scaling factor corresponding to each input scaling factor of the target network layer includes: Obtain the minimum value of each input scale factor of the target network layer; The ratio of each input scaling factor of the target network layer to the minimum value is determined as the normalized input scaling factor corresponding to each input scaling factor of the target network layer.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: For any target network layer in the target neural network, the input parameters of the target network layer and the weight parameters of the network layer are respectively input into the multiplier corresponding to the network layer to obtain the product of the input parameters of the target network layer and the weight parameters of the target network layer; The product and the bias parameter of the target network layer are respectively input into the adder corresponding to the target network layer to obtain the sum of the product and the bias parameter of the target network layer; The bit width of both the multiplier and the adder corresponding to the target network layer is equal to the preset quantization bit width of the target network layer.

10. A data processing method, characterized in that, The method includes: The data to be processed is input into the target neural network for forward propagation to obtain the target output parameters of the output layer of the target neural network. During the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer. The target output parameters of the output layer of the target neural network are inversely mapped based on the output scaling factor of the output layer of the target neural network. The output scaling factor of the output layer of the target neural network is obtained by forward propagation in the target neural network using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network. The target neural network is trained based on the method described in any one of claims 1 to 9.

11. A neural network training device, characterized in that, The device includes: An acquisition module is used to input sample data into the original neural network and acquire the sample output parameters of each network layer in the original neural network. A truncated layer is set between two adjacent network layers of the original neural network. Each truncated layer includes an activation function, which is used to truncate the sample output parameters of the previous network layer in the two adjacent network layers, and input the truncated sample output parameters into the next network layer in the two adjacent network layers. The truncated sample output parameters are within the parameter boundary values ​​corresponding to the preset quantization bit width of the corresponding network layer. The activation function of the truncated layer is determined based on the preset quantization bit width of the previous network layer connected to the truncated layer. The truncated layer is removed from the neural network after training. The loss function establishment module is used to establish a loss function based on the difference between the sample output parameters of each network layer and the parameter boundary value corresponding to the preset quantization bit width of the network layer; the loss function includes a penalty term, which is used to ensure that the output parameters of each network layer of the trained target neural network do not exceed the parameter boundary value corresponding to the preset quantization bit width of the network layer. The training module is used to train the original neural network based on the loss function to obtain the target neural network; The sample data is a sample image. The original neural network is used to extract features from the sample image. The sample output parameters of each network layer in the original neural network are output feature maps. The loss function is established based on the difference between the pixel value of the output feature map of each network layer and the pixel value boundary value corresponding to the preset quantization bit width of the network layer.

12. A data processing apparatus, characterized in that, The device includes: The input module is used to input the data to be processed into the target neural network for forward propagation and obtain the target output parameters of the output layer of the target neural network. During the forward propagation of the data to be processed, the input parameters of each network layer in the target neural network are mapped by the input scaling factor of the network layer, and the weight parameters of each network layer in the target neural network are quantized by the weight scaling factor of the network layer. The inverse mapping module is used to inverse map the target output parameters of the output layer of the target neural network based on the output scaling factor of the output layer of the target neural network. The output scaling factor of the output layer of the target neural network is obtained by forward propagation in the target neural network using the input scaling factor of the input layer of the target neural network as the input parameter of the input layer of the target neural network. The target neural network is trained based on the training device described in claim 11.

13. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-10.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 10.

Images