Quantization method and device of deep neural network, equipment and medium

Abstract

The application provides a quantization method and device of a deep neural network, equipment and a medium. The method comprises: obtaining a full floating point deep neural network training model, the storage length of at least one parameter of the full floating point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits. According to the relationship between each operation in the operation set of the full floating point deep neural network training model, the operation set of the full floating point deep neural network training model is compressed to obtain the operation set of a deep neural network inference model corresponding to the full floating point deep neural network training model. The deep neural network inference model is sent to a terminal. Thus, the influence on the performance of the terminal can be reduced, such as the influence on the ROM, RAM, CPU, inference time and other performances of the terminal.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a method, apparatus, device, and medium for quantizing deep neural networks. Background Technology

[0002] The intelligentization and smartification of terminals are major future trends. In fields such as Automatic Speech Recognition (ASR), Natural Language Processing (NLP), and Optical Character Recognition (OCR), with the increasing application of Artificial Intelligence (AI) year by year, and because different application scenarios typically use different Deep Neural Networks (DNN) models, users' requirements for terminal configurations are also getting higher and higher. However, it is not realistic for users to replace their terminals in order to use AI applications.

[0003] Typically, the training phase of a DNN model is performed by the server, while the inference phase in terminal applications is performed by the terminal. After the terminal completes the inference phase of the DNN model, users expect that various performance metrics of the terminal will be minimally affected, such as read-only memory (ROM) performance, random access memory (RAM) performance, central processing unit (CPU) performance, and inference time. Therefore, in order to minimize the impact on various performance metrics of the terminal, how to process the DNN model during the inference phase without changing the terminal configuration is a technical problem that this application urgently needs to solve. On the other hand, without changing the user's terminal configuration, users can also significantly reduce memory usage and improve performance. Summary of the Invention

[0004] This application provides a quantization method, apparatus, device, and medium for deep neural networks. This can reduce the impact on terminal performance, such as the performance of the terminal's ROM, CPU, and inference time.

[0005] In a first aspect, this application provides a method for processing deep neural networks, comprising: obtaining a fully floating-point deep neural network training model, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits; compressing the operation set of the fully floating-point deep neural network training model according to the relationship between the operations in the operation set of the fully floating-point deep neural network training model to obtain the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model; and sending the deep neural network inference model to a terminal.

[0006] Optionally, the operation set of the fully floating-point deep neural network training model is compressed according to the relationship between the operations in the operation set of the fully floating-point deep neural network training model, including: compressing multiple operations in the operation set of the fully floating-point deep neural network training model into one operation according to the relationship between the operations in the operation set of the fully floating-point deep neural network training model.

[0007] Because network devices compress the operation set of the full floating-point deep neural network training model to obtain the operation set of the deep neural network inference model, when the terminal executes the deep neural network inference model, the impact on the terminal's performance can be reduced, such as the impact on the terminal's ROM, CPU, inference time, and other performance aspects.

[0008] Optionally, before sending the deep neural network inference model to the terminal, the method further includes: quantizing the parameters involved in the operation set of the deep neural network inference model.

[0009] In the embodiments of this application, the network device can quantify the parameters involved in the operation set of the deep neural network inference model. When the terminal executes the deep neural network inference model, the impact on the terminal's performance can be further reduced, such as the impact on the terminal's ROM, CPU, inference time, and other performance parameters.

[0010] Optionally, the parameters involved in the operation set of the deep neural network inference model are floating-point parameters. Correspondingly, quantizing the parameters involved in the operation set of the deep neural network inference model includes: quantizing the parameters corresponding to the operation set of the deep neural network inference model into 1-bit parameters.

[0011] Optionally, the deep neural network inference model is used by the server to process the operations that need to be processed from the operation set of the fully floating-point deep neural network training model.

[0012] The following will provide a method, apparatus, device, and storage medium for processing deep neural networks.

[0013] Secondly, this application provides a method for processing a deep neural network, comprising: receiving a deep neural network inference model sent by a network device, wherein the operation set of the deep neural network inference model is obtained by compressing the operation set of a fully floating-point deep neural network training model, and the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits; and executing the deep neural network inference model.

[0014] Optionally, after executing the deep neural network inference model, the process further includes: acquiring terminal performance data. The terminal performance data is sent to a control device to determine the operations that need to be processed within the operation set of the fully floating-point deep neural network training model.

[0015] Thirdly, this application provides a method for processing deep neural networks, comprising: receiving performance data from a terminal, wherein the performance data is obtained by the terminal through executing a deep neural network inference model, the operation set of the deep neural network inference model is obtained by compressing the operation set of a fully floating-point deep neural network training model, and the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits; processing the performance data of the terminal to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model; and sending an instruction message to a server, the instruction message being used to instruct the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

[0016] Fourthly, this application provides a method for processing a deep neural network, comprising: receiving an instruction message; processing operations required to be processed in the operation set of a fully floating-point deep neural network training model according to the instruction message, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits; and sending the processed fully floating-point deep neural network training model to a network device.

[0017] Fifthly, this application provides a deep neural network processing apparatus, comprising:

[0018] The acquisition module is used to acquire a fully floating-point deep neural network training model, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0019] The compression module is used to compress the operation set of the fully floating-point deep neural network training model according to the relationship between the operations in the operation set, so as to obtain the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model.

[0020] The sending module is used to send the deep neural network inference model to the terminal.

[0021] Sixthly, this application provides a deep neural network processing apparatus, comprising:

[0022] The receiving module is used to receive the deep neural network inference model sent by the network device. The operation set of the deep neural network inference model is obtained by compressing the operation set of the fully floating-point deep neural network training model. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0023] The execution module is used to execute deep neural network inference models.

[0024] In a seventh aspect, this application provides a deep neural network processing apparatus, comprising:

[0025] The receiving module is used to receive the performance data of the terminal. The performance data is obtained by the terminal through the execution of a deep neural network inference model. The operation set of the deep neural network inference model is obtained by compressing the operation set of the fully floating-point deep neural network training model. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0026] The processing module is used to process the terminal's performance data to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

[0027] The sending module is used to send instruction messages to the server, which instruct the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

[0028] Eighthly, this application provides a deep neural network processing apparatus, comprising:

[0029] The receiving module is used to receive instruction messages.

[0030] The processing module is used to process the operations required by the operation set of the fully floating-point deep neural network training model according to the instruction message, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0031] The sending module is used to send the processed full-floating-point deep neural network training model to the network device.

[0032] Ninthly, this application provides a network device, comprising:

[0033] processor.

[0034] Memory for storing executable instructions of a processor, so that the processor performs a deep neural network processing method as described in the first aspect or an alternative to the first aspect.

[0035] In a tenth aspect, this application provides a terminal, comprising:

[0036] processor.

[0037] Memory for storing executable instructions of a processor, so that the processor performs a deep neural network processing method as described in the second aspect or an alternative to the second aspect.

[0038] In one aspect, this application provides a control device, comprising:

[0039] processor.

[0040] Memory for storing executable instructions of a processor, so that the processor performs a deep neural network processing method as described in the third aspect or an alternative method of the third aspect.

[0041] In a twelfth aspect, this application provides a server, comprising:

[0042] processor.

[0043] Memory for storing executable instructions of a processor, so that the processor performs a deep neural network processing method as described in the fourth aspect or an alternative to the fourth aspect.

[0044] In a thirteenth aspect, this application provides a storage medium comprising: executable instructions for implementing the above-described deep neural network processing method.

[0045] In a fourteenth aspect, this application provides a computer program product, comprising: executable instructions for implementing the above-described deep neural network processing method. Attached Figure Description

[0046] Figure 1 An interactive flowchart illustrating a deep neural network processing method provided in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram illustrating the compression of the operation set of a fully floating-point deep neural network training model according to an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of a terminal interface provided in an embodiment of this application;

[0049] Figure 4 An interactive flowchart of a deep neural network processing method provided in another embodiment of this application;

[0050] Figure 5 An interactive flowchart of a deep neural network processing method provided in another embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the fully floating-point deep neural network training model before and after operation processing, as provided in Example 1 of this application.

[0052] Figure 7 This is a schematic diagram of the fully floating-point deep neural network training model before and after operation processing, as provided in Example 2 of this application.

[0053] Figure 8 This is a schematic diagram of the fully floating-point deep neural network training model before and after operation processing, as provided in Example 3 of this application.

[0054] Figure 9 This is a schematic diagram of the fully floating-point deep neural network training model before and after operation processing, as provided in Example 4 of this application.

[0055] Figure 10 A schematic diagram of a deep neural network processing device 1000 provided in an embodiment of this application;

[0056] Figure 11 A schematic diagram of a deep neural network processing device 1100 provided in an embodiment of this application;

[0057] Figure 12 A schematic diagram of a deep neural network processing device 1200 provided in an embodiment of this application;

[0058] Figure 13 A schematic diagram of a deep neural network processing device 1300 provided in an embodiment of this application;

[0059] Figure 14 A schematic diagram of a network device 1400 provided in an embodiment of this application;

[0060] Figure 15 A schematic diagram of a terminal 1500 provided in an embodiment of this application;

[0061] Figure 16 A schematic diagram of a control device 1600 provided in an embodiment of this application;

[0062] Figure 17This is a schematic diagram of a server 1700 provided in one embodiment of this application. Detailed Implementation

[0063] As mentioned above, with the increasing number of AI applications year by year, and because different application scenarios typically use different DNN models, users' requirements for terminal configurations are also becoming increasingly stringent. However, it is not practical for users to change their terminals in order to use AI applications. Typically, the training phase of the DNN model is executed by the server, while the inference phase is executed by the terminal. After the terminal completes the inference phase of the DNN model, users expect that various performance indicators of the terminal, such as ROM performance, RAM performance, CPU performance, and inference time, should be as unaffected as possible. Therefore, in order to ensure that the various performance indicators of the terminal are as unaffected as possible, and without changing the terminal configuration, and while keeping the user's terminal configuration unchanged, users should also be able to significantly reduce memory usage and improve performance. How to process the DNN model during the inference phase is a technical problem that this application urgently needs to solve.

[0064] To address the aforementioned technical problems, this application provides a method, apparatus, device, and medium for quantizing deep neural networks. Before introducing the technical solution of this application, the following terms will be explained:

[0065] Deep neural network training model: This refers to the deep neural network model involved in the training phase.

[0066] Deep neural network inference model: This refers to the deep neural network model involved in the inference stage.

[0067] Full-precision deep neural network training model: This refers to the full-precision deep neural network model involved in the training phase. The parameters and data of this full-precision deep neural network model are all full-precision data, with the storage length of the parameters and data both greater than or equal to 8 bits. For example, the parameters of this full-precision deep neural network model might be 14.

[0068] A fully floating-point deep neural network training model refers to a deep neural network training model where the storage length of each parameter and / or each data point is less than 8 bits. For example, a parameter in this fully floating-point deep neural network training model might be +1.0, -1.0, etc. Since the deep neural network training model includes multiple operations, each operation can be understood as a deep neural network layer, or a deep neural network layer includes at least one operation. Each operation involves parameters, input data, and output data. Therefore, the parameters mentioned above refer to the parameters involved in the operation, and the data mentioned above refer to the input data and / or output data involved in the operation.

[0069] Figure 1This application provides an interactive flowchart of a deep neural network processing method according to an embodiment of the present application. The network elements involved in this method include: network devices and terminals. The network devices can be some or all of intelligent devices such as network devices, tablet computers, laptops, and servers (which store full-precision deep neural network training models). The terminals can be mobile phones, network devices, digital broadcasting terminals, message sending and receiving devices, game consoles, tablet devices, medical devices, fitness equipment, personal digital assistants, etc. Figure 1 As shown, the method includes the following steps:

[0070] Step S101: The network device acquires the training model of the full floating-point deep neural network.

[0071] Step S102: The network device compresses the operation set of the fully floating-point deep neural network training model according to the relationship between the operations in the operation set, so as to obtain the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model.

[0072] Step S103: The network device sends the deep neural network inference model to the terminal.

[0073] The following explanation is provided for step S101:

[0074] In one possible design: Assume the network device is not a server storing a full-precision deep neural network training model, and that the server quantizes at least one parameter and / or at least one piece of data in the full-precision deep neural network training model such that the storage length of the quantized at least one parameter is less than 8 bits and / or the storage length of at least one piece of data is less than 8 bits. That is, the server processes the full-precision deep neural network training model to obtain a fully floating-point deep neural network training model. Then the network device can obtain the fully floating-point deep neural network training model from this server. Alternatively, the network device could also be part of a server storing the full-precision deep neural network training model.

[0075] In another possible design: assuming the network device is not a server storing the full-precision deep neural network training model, the network device can obtain the full-precision deep neural network training model from the server and quantize at least one parameter and / or at least one piece of data in the full-precision deep neural network training model, such that the storage length of the quantized at least one parameter is less than 8 bits and / or the storage length of at least one piece of data is less than 8 bits. That is, the network device processes the full-precision deep neural network training model to obtain a fully floating-point deep neural network training model. The network device can use existing technologies to process the full-precision deep neural network training model, which will not be described further in this application.

[0076] The following explanation is provided for step S102:

[0077] Before executing step S102, the network device uses the same fully floating-point deep neural network training model and fully floating-point deep neural network inference model. As the names suggest, the operation set of the training model and the operation set of the inference model are also the same. As mentioned above, the configuration of a typical terminal is often insufficient to meet the needs of various AI applications. Therefore, in this application, the network device compresses the operation set of the fully floating-point deep neural network training model. Specifically, the network device compresses the operation set of the fully floating-point deep neural network training model based on the relationships between the operations within the model's operation set, thereby obtaining the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model.

[0078] For example: Figure 2 This is a schematic diagram illustrating the compression of the operation set of a fully floating-point deep neural network training model according to an embodiment of this application. Figure 2 The batch normalization (Bn), scaling (Scale), binary activation (BinAct), and binary tangent (BinTanh) layers shown are four different operations. Bn, Scale, and BinAct are some of the operations performed by the network device before operation set compression during the training of the fully floating-point deep neural network model. BinTanh is the operation obtained after the network device compresses Bn, Scale, and BinAct.

[0079] Bn represents y=alpha*(x-mu / sqrt(delta+epsilon))+beta;

[0080] Scale means thresh=sqrt(delta+epsilon)*(-beta)+alpha*mu;

[0081] BinAct means y >= 0.

[0082] Where y is the output data of Bn, x is the input data of Bn, and alpha, mu, delta, epsilon, and beta are the parameters involved in Bn.

[0083] Thresh is the output data of Scale, and delta, epsilon, beta, alpha, and mu are the parameters involved in Thresh.

[0084] y is the input data for BinAct.

[0085] "*" represents multiplication, " / " represents division, and sqrt represents the square root function.

[0086] It should be emphasized that this application does not limit the meaning of the above-mentioned data and parameters such as alpha, mu, delta, epsilon, beta, Thresh, x and y. This application focuses on the relationship between Bn, Scale and BinAct.

[0087] From the relationship between Bn, Scale, and BinAct, we can see that network devices can integrate y = alpha*(x - mu / sqrt(delta + epsilon)) + beta, thresh = sqrt(delta + epsilon)*(-beta) + alpha*mu, and y >= 0 into alpha*x >= thresh. Therefore, Figure 2 The BinTanh on the right represents alpha*x >= thresh. Based on this, the network device achieves the goal of compressing the three operations Bn, Scale, and BinAct into a single operation BinTanh.

[0088] It should be noted that the training model of a fully floating-point deep neural network may actually include more than the three operations mentioned above, and may also include other operations. As long as there is a certain relationship between the operations, the network device can compress these operations according to the relationship between them to obtain the compressed operation.

[0089] Furthermore, the above example of compressing three operations into one operation can actually be used to compress multiple operations in the operation set of a fully floating-point deep neural network training model into one operation, such as compressing two operations into one operation, or compressing four operations into one operation. This application does not limit this.

[0090] For example: Suppose an operation is used to calculate i*x i *y i Let i be any integer. Another operation is used to implement: for all i, i*x i *y i Summation; based on this, network devices can compress these two operations into one operation, which is represented as N-2*bitcount(xnor(x)). i y i In this context, bitcount is a bit-by-bit counting function, and xnor is an XNOR gate function.

[0091] The following explanation is provided for step S103:

[0092] The network device sends a deep neural network inference model to the terminal. The operation set in this deep neural network inference model includes compressed operations. After receiving the deep neural network inference model, the terminal executes it. If the deep neural network inference model is related to AI applications, the terminal can obtain AI data by executing the model. On the other hand, the terminal can also obtain its own performance data, such as ROM performance data, RAM performance data, and CPU performance data. For example: Figure 3 This is a schematic diagram of a terminal interface provided in an embodiment of this application, such as... Figure 3 As shown, when a user clicks on an application (APP) on the terminal desktop, the terminal displays an interface including the ROM usage rate.

[0093] In summary, this application provides a method for processing deep neural networks. The method includes: a network device acquiring a fully floating-point deep neural network training model; the network device compressing the operation set of the fully floating-point deep neural network training model based on the relationships between operations within the model's operation set to obtain an operation set for the corresponding deep neural network inference model; and the network device sending the deep neural network inference model to a terminal. Because the network device compresses the operation set of the fully floating-point deep neural network training model to obtain the operation set of the deep neural network inference model, the impact on terminal performance can be reduced when the terminal executes the deep neural network inference model, such as reducing the impact on ROM, CPU, and inference time.

[0094] The above describes compressing the operation set of a deep neural network inference model to reduce its impact on terminal performance. In addition, this application also considers quantizing the parameters involved in the operation set of the deep neural network inference model to further reduce its impact on terminal performance. Figure 4 This application provides an interactive flowchart of a deep neural network processing method according to another embodiment of the present application. The network elements involved in this method include: network devices and terminals, such as... Figure 4 As shown, prior to step S103 above, the method further includes the following steps:

[0095] Step S104: The network device quantifies the parameters involved in the operation set of the deep neural network inference model.

[0096] Example 1: Suppose the parameters involved in the operation set include: the weights involved in a certain operation, assuming there are 32 weights, for example, these 32 weights are as follows:

[0097] +1-1+1+1-1+1-1-1+1+1+1-1-1+1+1+1+1-1+1+1-1+1+1-1+1-1-1+1+1+1+1-1-1+1+1+1-1-1+1+1+1.

[0098] Assuming each weight is stored as a float, the 32 weights would occupy 32*4 bytes before the network device quantizes them. Based on this, the network device can quantize each weight into 1 bit; for example, +1 is quantized as 1, -1 as 0, and the 32 weights are quantized as 0110100111001111011010011100111. In this way, the storage length of the 32 weights is compressed by a factor of 32.

[0099] Example 2: Suppose the operation set involves a floating-point parameter 'a', where 'a' takes the values ​​0, 1, and 2, and each of these values ​​has a storage length of 4 bytes. In this case, the network device can quantize the parameter 'a', that is, represent 'a' using 2 bits. For example, 00 represents 0, 01 represents 1, and 10 represents 2. This is equivalent to quantizing the 4-byte parameter into 2 bits, and the storage length of 'a' is compressed by a factor of 16.

[0100] In summary, in the embodiments of this application, the network device can quantify the parameters involved in the operation set of the deep neural network inference model. When the terminal executes the deep neural network inference model, the impact on the terminal's performance can be further reduced, such as the impact on the terminal's ROM, CPU, inference time, and other performance parameters.

[0101] Based on any of the above embodiments, further, after receiving the deep neural network inference model, the terminal executes the deep neural network inference model, obtains performance data, and sends the performance data to the control device. The control device determines the operations that need to be processed in the operation set of the full-floating-point deep neural network training model and sends an instruction message to the server, so that the server processes the operations that need to be processed in the operation set of the full-floating-point deep neural network training model according to the instruction message. Specifically, based on the second embodiment, the deep neural network processing method will be further described. Specifically, it is assumed that the following network device is not a server storing the full-precision deep neural network training model. The method will be described below. Figure 5 This application provides an interactive flowchart of a deep neural network processing method according to another embodiment. The network elements involved in this method include: network devices, terminals, control devices, and servers, such as... Figure 5 As shown, after step S103 above, the method further includes the following steps:

[0102] Step S105: The terminal executes the deep neural network inference model.

[0103] Step S106: The terminal obtains the terminal's performance data.

[0104] Step S107: The terminal sends its performance data to the control device.

[0105] Step S108: Control the device to process the terminal's performance data to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

[0106] Step S109: The control device sends an instruction message to the server.

[0107] Step S110: The server processes the operations required in the operation set of the fully floating-point deep neural network training model according to the instruction message.

[0108] The following explanation is provided for steps S105-S107:

[0109] The terminal executes the deep neural network inference model to obtain AI data, and also acquires its own performance data, such as ROM performance data, RAM performance data, CPU performance data, and inference time. The terminal then sends this performance data to the control device.

[0110] The following explanation is provided for steps S108-S110:

[0111] In one possible design: the control device stores the correspondence between the terminal's performance data and the operations (including deletion, update, etc.) that need to be processed in the operation set of the fully floating-point deep neural network training model, as shown in Table 1:

[0112] Table 1

[0113]

[0114] Table 1 merely illustrates the correspondence between the terminal's ROM performance data and the operations (including deletion, update, etc.) that need to be processed in the operation set of the full floating-point deep neural network training model. In reality, the control device can obtain the correspondence between at least one of the four data—ROM performance data, RAM performance data, CPU performance data, and inference time—and the operations that need to be processed in the operation set of the full floating-point deep neural network training model, and determine the operations that need to be processed in the operation set of the full floating-point deep neural network training model based on the correspondence.

[0115] Optionally, the instruction message is used to instruct the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model. After receiving the instruction message, the server updates or deletes the operations that need to be processed in the operation set of the fully floating-point deep neural network training model according to the instruction message.

[0116] The following examples illustrate the processing performed by the control device on the operations required for training a fully floating-point deep neural network model:

[0117] Example 1: Figure 6 This is a schematic diagram of the fully floating-point deep neural network training model before and after operation processing, as provided in Example 1 of this application. The BN-Scale-BinAct operation set in the training model is as follows: Figure 2 As shown, network devices convert these into BinTanh operations. Experiments have shown that this saves ROM storage space and improves inference time performance. However, when the terminal obtains its performance data and sends it to the control device, the control device determines the operations that need to be processed in the operation set of the fully floating-point deep neural network training model based on the performance data and sends an instruction message to the server. The server processes the operations that need to be processed in the operation set of the fully floating-point deep neural network training model according to the instruction message. This feedback mechanism determines how to handle such operations. Figure 6 The BinAct operation in the shown fully floating-point deep neural network training model is updated to ReLU. Both ROM and inference time performance metrics are within acceptable ranges, and the accuracy of the fully floating-point deep neural network training model can be improved by 10%. Based on this, this feedback mechanism can achieve joint optimization of the accuracy of the fully floating-point deep neural network training model and the terminal performance.

[0118] Example 2: Figure 7 The diagrams provided in Example 2 of this application illustrate the fully floating-point deep neural network training model before and after operational processing. Figure 7 As shown, even after compressing and quantizing the operation sets of BinConv, Pool, and BinAct, the terminal's performance still shows a significant decrease. However, when the terminal acquires its performance data and sends it to the control device, the control device determines the operations that need to be processed in the operation set of the fully floating-point deep neural network training model based on this data and sends an instruction message to the server. The server processes the operations that need to be processed in the operation set of the fully floating-point deep neural network training model according to the instruction message. Through this feedback mechanism, it is determined that if the Pool operation is deleted and merged into BinConv withStride, the terminal's inference time is reduced by 10% without loss of accuracy of the fully floating-point deep neural network training model.

[0119] Example 3: Figure 8The diagrams provided in Example 3 of this application illustrate the fully floating-point deep neural network training model before and after operational processing. Figure 8 As shown, when the terminal acquires its performance data and sends it to the control device, the control device determines the operations that need to be processed in the operation set of the fully floating-point deep neural network training model based on the performance data, and sends an instruction message to the server. The server processes the operations that need to be processed in the operation set of the fully floating-point deep neural network training model according to the instruction message. After updating BinConv in BinConv and BinAct to Conv through this feedback mechanism, although the ROM performance of the terminal is reduced, the accuracy of the fully floating-point deep neural network training model is significantly improved. In this case, when the ROM performance reduction is within the user's acceptable range, it is permissible to update BinConv to Conv.

[0120] Example 4: Figure 9 The diagrams provided in Example 4 of this application illustrate the fully floating-point deep neural network training model before and after operational processing. Figure 9 As shown, when the terminal acquires its performance data and sends it to the control device, the control device determines the operations that need to be processed in the operation set of the full floating-point deep neural network training model based on the performance data and sends an instruction message to the server. The server processes the operations that need to be processed in the operation set of the full floating-point deep neural network training model according to the instruction message. Through this feedback mechanism, after updating the ReLU in the biased binary convolutional layer (BinConv with Bias), Pool, and ReLU to BinAct, specifically, because ReLU prevents the convolution (Bias) addition operation in BinConv with Bias from being optimized away, it can only be performed as a full-precision Conv operation. By changing the ReLU operation to BinAct, although the precision of the full floating-point deep neural network training model is reduced, inference performance and ROM performance can be improved. In this case, when the reduction in the precision of the full floating-point deep neural network training model is within the user's acceptable range, updating ReLU to BinAct is allowed.

[0121] In summary, in this example, when the terminal acquires its performance data and sends it to the control device, the control device determines the operations that need to be processed in the operation set of the fully floating-point deep neural network training model based on the performance data, and sends an instruction message to the server. The server processes the operations that need to be processed in the operation set of the fully floating-point deep neural network training model according to the instruction message. This feedback mechanism can achieve joint optimization or trade-off optimization of the accuracy of the fully floating-point deep neural network training model and the terminal performance.

[0122] Figure 10 A schematic diagram of a deep neural network processing device 1000 provided in an embodiment of this application is shown below. Figure 10 As shown, the device can be part or all of a network device such as a computer, laptop, or server (which stores a full-precision deep neural network training model), and the device includes:

[0123] The acquisition module 1001 is used to acquire a fully floating-point deep neural network training model, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0124] The compression module 1002 is used to compress the operation set of the fully floating-point deep neural network training model according to the relationship between the operations in the operation set of the fully floating-point deep neural network training model, so as to obtain the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model.

[0125] The sending module 1003 is used to send the deep neural network inference model to the terminal.

[0126] In one possible design, the compression module 1002 is specifically used to: compress multiple operations in the operation set of the fully floating-point deep neural network training model into one operation based on the relationship between the operations in the operation set of the fully floating-point deep neural network training model.

[0127] In one possible design, the device further includes a quantization module 1004 for quantizing the parameters involved in the operation set of the deep neural network inference model before sending the deep neural network inference model to the terminal.

[0128] In one possible design, the operation set of the deep neural network inference model involves floating-point parameters.

[0129] Accordingly, the quantization module 1004 is specifically used to quantize the parameters corresponding to the operation set of the deep neural network inference model into 1-bit parameters.

[0130] In one possible design, the deep neural network inference model is used by the server to process the operations that need to be processed from the operation set of the fully floating-point deep neural network training model.

[0131] The deep neural network processing apparatus provided in this application embodiment can be used to execute the deep neural network processing method executed on the network device side. Its content and effects can be referred to the method section, and will not be repeated here.

[0132] Figure 11A schematic diagram of a deep neural network processing device 1100 provided in an embodiment of this application is shown below. Figure 11 As shown, the device can be part or all of a terminal such as a mobile phone or tablet computer, and the device includes:

[0133] The receiving module 1101 is used to receive the deep neural network inference model sent by the network device. The operation set of the terminal device's deep neural network inference model is obtained by compressing the operation set of the full floating-point deep neural network training model. The storage length of at least one parameter of the terminal device's full floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0134] Execution module 1102 is used to execute the deep neural network inference model of the terminal device.

[0135] In one possible design, the device also includes:

[0136] The acquisition module 1103 is used to acquire the performance data of the terminal after executing the deep neural network inference model of the terminal device.

[0137] The sending module 1104 is used to send the performance data of the terminal device to the control device. The performance data of the terminal device is used to determine the operations that need to be processed in the operation set of the terminal device's full floating-point deep neural network training model.

[0138] The deep neural network processing apparatus provided in this application embodiment can be used to execute the deep neural network processing method executed on the terminal side as described above. Its content and effects can be referred to in the method section, and will not be repeated here.

[0139] Figure 12 A schematic diagram of a deep neural network processing device 1200 provided in an embodiment of this application is shown below. Figure 12 As shown, the device may be part or all of the control equipment, and the device includes:

[0140] The receiving module 1201 is used to receive the performance data of the terminal. The performance data of the terminal device is obtained by the terminal device through the execution of a deep neural network inference model. The operation set of the deep neural network inference model of the terminal device is obtained by compressing the operation set of the full floating-point deep neural network training model. The storage length of at least one parameter of the full floating-point deep neural network training model of the terminal device is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0141] The processing module 1202 is used to process the performance data of the terminal device in order to determine the operations that need to be processed in the operation set of the terminal device's full floating-point deep neural network training model.

[0142] The sending module 1203 is used to send an instruction message to the server. The terminal device instruction message is used to instruct the terminal device server to process the operations that need to be processed in the operation set of the terminal device's full floating-point deep neural network training model.

[0143] The deep neural network processing apparatus provided in this application embodiment can be used to execute the deep neural network processing method executed by the control device side. Its content and effects can be referred to the method section, and will not be repeated here.

[0144] Figure 13 A schematic diagram of a deep neural network processing device 1300 provided in an embodiment of this application is shown below. Figure 13 As shown, the device may be part or all of the server, and the device includes:

[0145] The receiving module 1301 is used to receive indication messages.

[0146] The processing module 1302 is used to process the operations that need to be processed in the operation set of the full floating-point deep neural network training model according to the terminal device instruction message, wherein the storage length of at least one parameter of the terminal device's full floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0147] The sending module 1303 is used to send the processed full floating-point deep neural network training model to the network device.

[0148] The deep neural network processing apparatus provided in this application embodiment can be used to execute the deep neural network processing method executed on the server side as described above. Its content and effects can be referred to in the method section, and will not be repeated here.

[0149] Figure 14 This is a schematic diagram of a network device 1400 provided in an embodiment of this application, as shown below. Figure 14 As shown, the network device includes a memory 1401, a processor 1402, and a transceiver 1403, wherein the memory 1401 is used to store computer instructions so that the processor 1402 executes the instructions to implement the processing method of a deep neural network.

[0150] The processor 1402 is used to: acquire a fully floating-point deep neural network training model, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits; and compress the operation set of the fully floating-point deep neural network training model according to the relationship between the operations in the operation set of the fully floating-point deep neural network training model to obtain the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model.

[0151] Transceiver 1403 is used to send deep neural network inference models to the terminal.

[0152] Optionally, the processor 1402 is specifically used to: compress multiple operations in the operation set of the fully floating-point deep neural network training model into one operation based on the relationship between the operations in the operation set of the fully floating-point deep neural network training model.

[0153] Optionally, the processor 1402 is also used to: quantize the parameters involved in the operation set of the deep neural network inference model.

[0154] Optionally, the parameters involved in the operation set of the deep neural network inference model are floating-point parameters. Accordingly, the processor 1402 is specifically used to: quantize the parameters corresponding to the operation set of the deep neural network inference model into 1-bit parameters.

[0155] Optionally, the deep neural network inference model is used by the server to process the operations that need to be processed from the operation set of the fully floating-point deep neural network training model.

[0156] The network device provided in this application embodiment can be used to execute the deep neural network processing method executed on the network device side. Its content and effects can be referred to the method section, and will not be repeated here.

[0157] Figure 15 This is a schematic diagram of a terminal 1500 provided in one embodiment of this application, as shown below. Figure 15 As shown, the terminal includes: a memory 1501, a processor 1502, and a transceiver 1503, wherein the memory 1501 is used to store computer instructions so that the processor 1502 executes the instructions to implement the processing method of a deep neural network.

[0158] Transceiver 1503 is used to receive a deep neural network inference model sent by a network device. The operation set of the deep neural network inference model is obtained by compressing the operation set of the fully floating-point deep neural network training model. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0159] Processor 1502 is used to execute deep neural network inference models.

[0160] Optionally, the processor 1502 is also used to acquire terminal performance data. The transceiver 1503 is also used to send the terminal performance data to the control device, the terminal performance data being used to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

[0161] The terminal provided in this application embodiment can be used to execute the deep neural network processing method executed on the terminal side as described above. Its content and effects can be referred to in the method section, and will not be repeated here.

[0162] Figure 16 This is a schematic diagram of a control device 1600 provided in an embodiment of this application, as shown below. Figure 16 As shown, the control device includes a memory 1601, a processor 1602, and a transceiver 1603, wherein the memory 1601 is used to store computer instructions so that the processor 1602 executes the instructions to implement the deep neural network processing method.

[0163] Transceiver 1603 is used to receive performance data from the terminal. The performance data is obtained by the terminal through executing a deep neural network inference model. The operation set of the deep neural network inference model is obtained by compressing the operation set of the fully floating-point deep neural network training model. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0164] Processor 1602 is used to process the terminal's performance data to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

[0165] Transceiver 1603 is also used to send instruction messages to the server, which instruct the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

[0166] The control device provided in this application embodiment can be used to execute the deep neural network processing method executed on the control device side. Its content and effects can be referred to the method section, and will not be repeated here.

[0167] Figure 17 A schematic diagram of a server 1700 provided in one embodiment of this application is shown below. Figure 17 As shown, the server includes: a memory 1701, a processor 1702, and a transceiver 1703, wherein the memory 1701 is used to store computer instructions so that the processor 1702 executes the instructions to implement the processing method of deep neural networks.

[0168] Transceiver 1703 is used to receive instruction messages.

[0169] The processor 1702 is used to process operations required by the operation set of the fully floating-point deep neural network training model according to the instruction message, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits.

[0170] Transceiver 1703 is also used to send processed full-floating-point deep neural network training models to network devices.

[0171] The server provided in this application embodiment can be used to execute the deep neural network processing method executed on the server side as described above. Its content and effects can be found in the method section, and will not be repeated here.

[0172] This application also provides a computer storage medium including computer instructions for implementing the above-described deep neural network processing method. The content and effects of these instructions can be found in the method section and will not be repeated here.

[0173] This application also provides a computer program product, which includes computer instructions for implementing the above-described deep neural network processing method. The content and effects of these instructions can be found in the method section, and will not be repeated here.

Claims

1. A method for processing deep neural networks, characterized in that, include: Obtain a fully floating-point deep neural network training model, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits; Based on the relationship between the operations in the operation set of the fully floating-point deep neural network training model, multiple operations in the operation set of the fully floating-point deep neural network training model are compressed into a preset operation to obtain the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model. The deep neural network inference model is used for speech recognition, natural language processing or optical character recognition. The preset operation is different from any of the multiple operations. The deep neural network inference model is sent to the terminal.

2. The method according to claim 1, characterized in that, Before sending the deep neural network inference model to the terminal, the method further includes: The parameters involved in the operation set of the deep neural network inference model are quantified.

3. The method according to claim 2, characterized in that, The operation set of the deep neural network inference model involves floating-point parameters; Accordingly, the parameters involved in the operation set for quantizing the deep neural network inference model include: The parameters corresponding to the operation set of the deep neural network inference model are quantized into 1-bit parameters.

4. The method according to any one of claims 1-3, characterized in that, The deep neural network inference model is used by the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

5. A method for processing deep neural networks, characterized in that, include: The system receives a deep neural network inference model sent by a network device. The operation set of the deep neural network inference model is obtained by compressing multiple operations in the operation set of the fully floating-point deep neural network training model into a preset operation. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits. The deep neural network inference model is used for speech recognition, natural language processing, or optical character recognition. The preset operation is different from any of the multiple operations. Execute the deep neural network inference model.

6. The method according to claim 5, characterized in that, After executing the deep neural network inference model, the method further includes: Obtain terminal performance data; The terminal's performance data is sent to the control device, and the terminal's performance data is used to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

7. A method for processing deep neural networks, characterized in that, include: The receiving terminal receives performance data, which is obtained by the terminal through executing a deep neural network inference model. The operation set of the deep neural network inference model is obtained by compressing multiple operations in the operation set of the fully floating-point deep neural network training model into a preset operation. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits. The deep neural network inference model is used for speech recognition, natural language processing, or optical character recognition. The preset operation is different from any of the multiple operations. Process the performance data of the terminal to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model; Send an instruction message to the server, the instruction message being used to instruct the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

8. A method for processing deep neural networks, characterized in that, include: Receive instruction message; The operation to be processed according to the operation set of the full floating-point deep neural network training model is processed according to the instruction message, wherein the storage length of at least one parameter of the full floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits, and the deep neural network inference model is used for speech recognition, natural language processing or optical character recognition. Send the processed full-floating-point deep neural network training model to the network device.

9. A deep neural network processing device, characterized in that, include: The acquisition module is used to acquire a fully floating-point deep neural network training model, wherein the storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits. A compression module is used to compress multiple operations in the operation set of the fully floating-point deep neural network training model into a preset operation based on the relationship between the operations in the operation set of the fully floating-point deep neural network training model, so as to obtain the operation set of the deep neural network inference model corresponding to the fully floating-point deep neural network training model. The deep neural network inference model is used for speech recognition, natural language processing or optical character recognition. The preset operation is different from any of the multiple operations. The sending module is used to send the deep neural network inference model to the terminal.

10. The apparatus according to claim 9, characterized in that, Also includes: The quantization module is used to quantize the parameters involved in the operation set of the deep neural network inference model before sending the deep neural network inference model to the terminal.

11. The apparatus according to claim 10, characterized in that, The operation set of the deep neural network inference model involves floating-point parameters; Accordingly, the quantization module is specifically used for: The parameters corresponding to the operation set of the deep neural network inference model are quantized into 1-bit parameters.

12. The apparatus according to any one of claims 9-11, characterized in that, The deep neural network inference model is used by the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

13. A deep neural network processing device, characterized in that, include: A receiving module is used to receive a deep neural network inference model sent by a network device. The operation set of the deep neural network inference model is obtained by compressing multiple operations in the operation set of the fully floating-point deep neural network training model into a preset operation. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits. The deep neural network inference model is used for speech recognition, natural language processing or optical character recognition. The preset operation is different from any of the multiple operations. An execution module is used to execute the deep neural network inference model.

14. The apparatus according to claim 13, characterized in that, Also includes: The acquisition module is used to acquire the terminal's performance data after executing the deep neural network inference model; The sending module is used to send the performance data of the terminal to the control device. The performance data of the terminal is used to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

15. A processing device for a deep neural network, characterized in that, include: A receiving module is used to receive performance data from a terminal. The performance data is obtained by the terminal through executing a deep neural network inference model. The operation set of the deep neural network inference model is obtained by compressing multiple operations in the operation set of the fully floating-point deep neural network training model into a preset operation. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits. The deep neural network inference model is used for speech recognition, natural language processing, or optical character recognition. The preset operation is different from any of the multiple operations. The processing module is used to process the performance data of the terminal to determine the operations that need to be processed in the operation set of the fully floating-point deep neural network training model; The sending module is used to send an instruction message to the server, the instruction message being used to instruct the server to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model.

16. A deep neural network processing device, characterized in that, include: The receiving module is used to receive indication messages; The processing module is used to process the operations that need to be processed in the operation set of the fully floating-point deep neural network training model according to the instruction message. The storage length of at least one parameter of the fully floating-point deep neural network training model is less than 8 bits and / or the storage length of at least one data is less than 8 bits. The deep neural network inference model is used for speech recognition, natural language processing or optical character recognition. The sending module is used to send the processed full-floating-point deep neural network training model to the network device.

17. A network device, characterized in that, include: processor; A memory for storing executable instructions of the processor, so that the processor performs the deep neural network processing method as described in any one of claims 1-4.

18. A terminal, characterized in that, include: processor; A memory for storing executable instructions of the processor, so that the processor performs the deep neural network processing method as described in claim 5 or 6.

19. A control device, characterized in that, include: processor; A memory for storing executable instructions of the processor, so that the processor performs the deep neural network processing method as described in claim 7.

20. A server, characterized in that, include: processor; A memory for storing executable instructions of the processor, so that the processor performs the deep neural network processing method as described in claim 8.

21. A storage medium, characterized in that, include: Executable instructions for implementing the deep neural network processing method as described in any one of claims 1-8.

22. A computer program product, characterized in that, include: Executable instructions for implementing the deep neural network processing method as described in any one of claims 1-8.

Images