Mixed precision neural network quantization method, apparatus and device

Abstract

The application discloses a mixed-precision neural network quantization method, device and equipment. The method combines reinforcement learning and verification data generation technology to determine a mixed-precision quantization strategy, so that the distribution gap between a mixed-precision network and the output of a network to be quantized is determined based on a self-generated verification data set, which is used as a standard to judge the effect of the quantization strategy. With this processing method, the user does not need to provide verification data for testing the accuracy of the quantized network, and the time-consuming for verifying the accuracy of the quantized network is reduced. The user does not need to fine-tune the quantized network determined through reinforcement learning, and can obtain a quantized network with a smaller precision loss and a larger calculation amount reduction, and the network quantization process is simplified, and the user is not aware of the model quantization strategy search. Therefore, under the premise of ensuring a small model precision loss, the model quantization efficiency can be effectively improved, the data privacy of the user is protected, and the artificial cost is reduced.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, specifically to mixed-precision neural network quantization methods, apparatus and systems, and electronic devices. Background Technology

[0002] With the development of deep learning, neural network-based machine learning models have been widely applied in various fields. While improving model performance, this has also introduced a huge number of parameters and computational burdens. Mixed-precision quantization quantizes the weights of each layer of a neural network into different precisions. For example, it converts floating-point calculations of some network layers into low-precision point-to-point calculations, while other network layers still use floating-point calculations, thereby obtaining a smaller model.

[0003] Mixed-precision quantization can effectively reduce model computational intensity, parameter size, and memory consumption, but it often leads to significant accuracy loss. Therefore, how to search for mixed-precision quantization strategies to achieve a good trade-off between network computational cost and accuracy has been extensively studied. A typical approach to searching for mixed-precision quantization strategies involves searching for quantization strategies at each layer of the network, fine-tuning the quantized network after each search to more accurately evaluate the quantization strategy, and then performing a final fine-tuning training after the overall strategy search is complete. This method uses a user-provided validation dataset to test the accuracy of various possible quantized networks, selecting the quantized network that achieves a good trade-off between computational cost and accuracy.

[0004] However, in the process of realizing this invention, the inventors discovered that existing solutions have at least the following problems: 1) Fine-tuning training leads to a complex network quantization process, such as the need to determine training hyperparameters; 2) Inference on a large validation set provided by the user to obtain the accuracy of the quantized network is very time-consuming; 3) Quantization calibration and validation both require user-provided data, which may lead to various issues such as privacy. In summary, how to achieve user-insensitive model quantization, quickly determine a high-precision mixed-precision network, and protect sensitive user privacy data has become an urgent problem for developers in this field. Summary of the Invention

[0005] This application provides a hybrid precision neural network quantization method to address the problems of existing technologies, such as user-perceived model quantization, low quantization efficiency, significant model accuracy loss, and inability to protect user data privacy. This application also provides a hybrid precision neural network quantization device and system, and electronic equipment.

[0006] This application provides a mixed-precision neural network quantization method, including:

[0007] Obtain the network to be quantized;

[0008] Generate a validation dataset based on the network to be quantized;

[0009] Using a reinforcement learning algorithm, the target mixed-precision network of the network to be quantized is determined based on the validation dataset.

[0010] Optionally, determining the target mixed-precision network of the network to be quantized based on the validation dataset using a reinforcement learning algorithm includes:

[0011] Based on the resource usage threshold, determine multiple mixed-precision networks for the network to be quantized;

[0012] Based on the validation dataset, determine the loss data of the hybrid precision network relative to the quantized network;

[0013] Based on the loss data, a target mixed-precision network is determined from multiple mixed-precision networks.

[0014] Optionally, determining the loss data of the mixed-precision network relative to the quantized network based on the validation dataset includes:

[0015] Based on the validation dataset, determine the first output data of the network to be quantized and the second output data of the mixed precision network;

[0016] The loss data is determined based on the first output data and the second output data.

[0017] Optionally, determining the target mixed-precision network from multiple mixed-precision networks based on the loss data includes:

[0018] The mean of the loss data at each point is used as the exponent of the exponential function. Based on the exponential function value, the target mixed-precision network is determined from multiple mixed-precision networks.

[0019] Optionally, determining the target mixed-precision network from multiple mixed-precision networks based on the loss data includes:

[0020] Based on the loss data and the resource usage of the mixed-precision network, a target mixed-precision network is determined from multiple mixed-precision networks.

[0021] Optionally, determining the target mixed-precision network of the network to be quantized based on the validation dataset using a reinforcement learning algorithm further includes:

[0022] The mixed-precision network is calibrated based on the validation dataset.

[0023] Optionally, determining multiple mixed-precision networks of the network to be quantized based on a resource usage threshold includes:

[0024] If the resource usage of the hybrid precision network exceeds a resource usage threshold, the precision of some layers of the hybrid precision network is reduced.

[0025] Optionally, reducing the accuracy of certain layers of the hybrid precision network includes:

[0026] The partial layers are determined in a random manner.

[0027] Optionally, the observation features of the reinforcement learning algorithm include: the number of bits of computational precision of multiple network layers of the mixed precision network, the computational cost of the multiple network layers, and the identifier of the currently quantized network layer.

[0028] Optional, also includes:

[0029] Based on the device resource data of the target device in the network to be quantized, determine the resource usage threshold of the network to be quantized;

[0030] The target mixed-precision network is determined using a reinforcement learning algorithm based on the validation dataset and the resource usage threshold.

[0031] Deploy the target mixed-precision network to the target device.

[0032] This application also provides a mixed-precision neural network quantization device, comprising:

[0033] The network acquisition unit is used to acquire the network to be quantized.

[0034] The data generation unit is used to generate a validation dataset based on the network to be quantized.

[0035] A reinforcement learning unit is used to determine the target mixed-precision network of the network to be quantized based on the validation dataset using a reinforcement learning algorithm.

[0036] Optionally, the reinforcement learning unit includes:

[0037] The network quantization subunit is used to determine multiple mixed-precision networks of the network to be quantized based on a resource usage threshold.

[0038] The loss determination subunit is used to determine the loss data of the mixed precision network relative to the quantized network based on the validation dataset.

[0039] A network selection sub-unit is used to determine a target mixed-precision network from multiple mixed-precision networks based on the loss data.

[0040] This application also provides an electronic device, including:

[0041] Processor; and

[0042] The memory stores the program for implementing the above-described mixed-precision neural network quantization method. The terminal is powered on and runs the program of the method through the processor.

[0043] This application also provides a mixed-precision neural network processing system, including:

[0044] Network construction device, the aforementioned hybrid precision neural network quantization device, and network deployment device.

[0045] This application also provides an electronic device, including:

[0046] Processor and memory; memory for storing a program that implements the above method, the device being powered on and the program of the method being run by the processor.

[0047] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the various methods described above.

[0048] This application also provides a computer program product including instructions that, when run on a computer, cause the computer to perform the various methods described above.

[0049] Compared with the prior art, this application has the following advantages:

[0050] The mixed-precision neural network quantization method provided in this application combines reinforcement learning and validation data generation techniques to determine the mixed-precision quantization strategy. This allows the distribution difference between the outputs of the mixed-precision network and the network to be quantized to be determined based on an automatically generated validation dataset, serving as a standard for evaluating the effectiveness of the quantization strategy. This approach eliminates the need for users to provide validation data to verify the accuracy of the quantized network, reducing the time spent on validating the network's accuracy. It also eliminates the need for users to fine-tune the quantized network determined through reinforcement learning, achieving a quantized network with a smaller loss of precision but a larger reduction in computational cost. Furthermore, it simplifies the network quantization process, enabling seamless model quantization strategy search for the user. Therefore, while ensuring minimal loss of model accuracy, it effectively improves model quantization efficiency, protects user data privacy, and reduces labor costs. In addition, this approach allows the converted network layers using fixed-point computation to achieve truly low-bit convolutional computation, achieving a larger reduction in computational cost with a smaller loss of precision, thus effectively reducing model accuracy loss. Attached Figure Description

[0051] Figure 1 This application provides an illustration of an application scenario for the network quantization method.

[0052] Figure 2 A flowchart illustrating an embodiment of the network quantization method provided in this application;

[0053] Figure 3 A detailed flowchart illustrating an embodiment of the network quantization method provided in this application;

[0054] Figure 4 Schematic diagrams of different precision data mappings in embodiments of the network quantization method provided in this application;

[0055] Figure 5 A schematic diagram of the reinforcement learning process of an embodiment of the network quantization method provided in this application. Detailed Implementation

[0056] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0057] This application provides a method, apparatus, and system for mixed-precision neural network quantization, as well as an electronic device. The various solutions are described in detail in the following embodiments.

[0058] Please refer to Figure 1 This is a schematic diagram illustrating an application scenario of the hybrid precision neural network quantization method provided in this application. In this embodiment, terminal devices (such as smartphones, smart TVs, smart air conditioners, etc.) can perform model prediction processing through locally deployed neural network models. For example, a smartphone can use a locally deployed 'tap-to-click' recognition model to determine whether a tap-to-click action has occurred based on the state data of the phone's motion detection device (such as a gyroscope). If it is determined that a tap-to-click action has occurred, application processing related to the tap-to-click action can be executed, such as adding friends in an instant messaging application. Due to the limited computing resources of terminal devices, hybrid precision networks are usually deployed. Meanwhile, with the continuous development of GPU chip technology, the computational speed of low-precision fixed-point computation (such as Int8, Int4) has increased several times compared to floating-point computation (such as FP32, FP16). Therefore, using as much fixed-point computation as possible on terminal devices greatly helps with both model size and speed.

[0059] In practical applications, the network builder's users can first train a neural network-based prediction model using a network building server, and then submit this model as the network to be quantized to the network quantization processing department. Next, the network quantization team's users can quantize the original network using a network quantization server. Finally, the quantized network can be deployed to the terminal device for prediction processing.

[0060] It should be noted that the methods provided in the embodiments of this application are not limited to those described above. Figure 1 The application scenarios shown are as follows: Figure 1 The application scenario shown is only one possible one. In specific implementations, the network builder and the network quantizer can be as follows: Figure 1 The separate design shown can also be an integrated approach, where the network builder is responsible for model quantization. Furthermore, the quantization network can be deployed on terminal devices or on servers and other equipment.

[0061] First Embodiment

[0062] Please refer to Figure 2 This is a flowchart illustrating an embodiment of the mixed-precision neural network quantization method of this application. In this embodiment, the method may include the following steps:

[0063] Step S201: Obtain the network to be quantized.

[0064] The network to be quantized refers to a machine learning model based on neural networks, such as an object detection model, a touch-to-action recognition model, or a speech recognition model. The number of parameters in the network to be quantized is typically large. Taking the commonly used AlexNet as an example, the number of parameters for each network layer is shown in the table below:

[0065]

[0066]

[0067] The table shows that AlexNet has a total of 62,369,155 parameters and consumes 238MB of memory. For example, the 16-layer VGG network has a total of 138,357,544 + 138,348,355 parameters and consumes 528MB of memory.

[0068] The parameters of each layer of the network to be quantized can be calculated using floating-point numbers (such as FP32 or FP16). Due to the large number of parameters, the computational cost of using the network to be quantized is also enormous. To obtain a smaller model, the method provided in this application embodiment can be used to quantize the parameters (weights) of each layer of the neural network to different precisions. For example, the floating-point calculations of some network layers can be converted to low-precision fixed-point calculations, while other network layers still use floating-point calculations. In this way, a large reduction in computational cost can be achieved with a small loss of precision.

[0069] Step S203: Generate a validation dataset based on the network to be quantized.

[0070] In the process of network quantization, it is necessary to determine the accuracy of each network layer and to verify the accuracy of the quantized network using a validation dataset. The method provided in this application generates a validation dataset based on the network to be quantized, which is used to verify the accuracy of the quantized network.

[0071] The parameters of the network to be quantized are learned from the training dataset, and the validation data automatically generated based on the network to be quantized is data with certain characteristics similar to the training data. In this embodiment, the validation dataset can be generated in a self-supervised manner by utilizing the characteristics of network Batch Normalization (BN).

[0072] In the field of machine learning, it is assumed that training, validation, and test data follow the same distribution. This is a fundamental guarantee that a model obtained from training data can achieve good results on the test set. BatchNorm is an algorithm that ensures that the input to each layer of a deep neural network maintains the same distribution during training, thereby accelerating neural network training, convergence speed, and stability.

[0073] In one example, the input data to the network to be quantized is image data, and the validation dataset automatically generated based on the network to be quantized is a validation image set. Specifically, ZeroQ image generation technology can be used, leveraging the characteristics of the validation data at the batchnorm layer to generate the validation data based on the difference between the mean and variance of the network to be quantized. Since generating a validation dataset from the network to be quantized is a relatively mature existing technology, it will not be elaborated upon here.

[0074] It's important to note that the validation data automatically generated based on the network to be quantized differs from user-provided validation data in two ways: 1) User-provided validation data includes private data, while the validation data automatically generated based on the network to be quantized does not. User-provided validation data is typically a subset of the training dataset. This subset is not used for training; its primary purpose is to assess the model's training effectiveness, periodically checking the model's accuracy on the validation dataset to prevent undertraining or overtraining. 2) Each user-provided validation dataset contains only a limited number of features, requiring validation on a large dataset (e.g., tens of thousands of images), leading to more time-consuming multiple validations. In contrast, each validation dataset automatically generated based on the network to be quantized includes more features (e.g., a single image contains tens of thousands of features), allowing for better accuracy even on a small dataset (e.g., 64 images). This accelerates the quantization strategy search process in feature matching scenarios.

[0075] In summary, the method provided in this application determines the distribution difference between the outputs of the mixed-precision network and the network to be quantized based on an automatically generated validation dataset, using this as a standard to evaluate the effectiveness of the quantization strategy. This approach eliminates the need for users to provide validation data to verify the accuracy of the quantized network, reduces the time spent validating the accuracy of the quantized network, and enables seamless model quantization strategy search for the user. Therefore, it can effectively improve model quantization efficiency and protect user data privacy.

[0076] Step S205: Using a reinforcement learning algorithm, determine the target mixed-precision network of the network to be quantized based on the validation dataset.

[0077] The method provided in this application embodiment can use a reinforcement learning (RL) agent to search for quantization strategies of each network layer in the network to be quantized, and verify the accuracy of the mixed precision network (hereinafter referred to as the quantized network) based on the verification dataset, so as to obtain a target mixed precision network with high accuracy.

[0078] The target mixed-precision network can be a mixed-precision neural network. In a mixed-precision neural network, the parameters of some network layers are of floating-point data type and are used for floating-point operations; the parameters of some network layers are of integer data type and are used for fixed-point operations. The network layers using floating-point precision may include FP32 precision network layers, FP16 precision network layers, etc. The network layers using fixed-point precision may include Int8 precision network layers, Int4 precision network layers, and Int2 precision network layers.

[0079] The target mixed-precision network meets the computational requirements, meaning its resource usage is less than or equal to a resource usage threshold. In this embodiment, the target mixed-precision network is determined using a reinforcement learning algorithm based on the validation dataset and the resource usage threshold.

[0080] The resource usage threshold is the upper limit of device resources that the target hybrid precision network can consume, including at least a computational resource threshold, and may also include a storage resource threshold. The resource usage threshold can be determined according to application requirements. In practice, the resource usage threshold can be determined manually based on experience.

[0081] In one example, the method may further include the following step: determining a resource usage threshold for the network to be quantized based on the device resource data of the target device. Thus, after determining the target mixed-precision network, it is deployed to the target device for operation. The target device includes, but is not limited to, mobile terminals such as smartphones and tablets, and may also include servers, personal computers, smart TVs, smart speakers, smart refrigerators, and self-driving cars.

[0082] Reinforcement learning, also known as reward learning, evaluative learning, or enhancement learning, describes and solves the problem of how an agent learns strategies to maximize rewards or achieve specific goals during interactions with its environment. Intuitively, reinforcement learning aims to teach an agent how to act in an environment to obtain the maximum total reward. This reward is associated with the agent's defined task objective. The main learning content for the agent includes an action policy. The goal of learning the action policy is to optimize the policy—that is, to use a policy that maximizes the agent's reward in a specific environment, thereby achieving its task objective.

[0083] The method provided in this application applies reinforcement learning to the quantization scenario of mixed-precision neural networks. This involves enabling the agent to learn the quantization policies (i.e., actions) of each network layer in the network to be quantized, thereby obtaining a mixed-precision network with high accuracy. Specifically, Deep Deterministic Policy Gradient (DDPG) or Twin Delayed DDPG (TD3) can be used as the reinforcement learning agent.

[0084] like Figure 5 As shown, this embodiment uses TD3 as the agent. TD3 simultaneously learns both the policy network and the value network (resulting in a pair of twin value networks), making it suitable for continuous action spaces. The value network Q(s,a) primarily models the value of taking an action a in a given state (also known as an observation) s, and it is trained using the self-consistency of the Bellman equation. The policy network Pl(s) primarily models which action a in a given state s will yield the maximum reward. If the value network Q exists, its objective is easily obtained. Experiments show that TD3 has significantly improved robustness and convergence compared to DDPG.

[0085] In this embodiment, the reward function (reward function) of the reinforcement learning algorithm can be determined based on the loss data of the mixed-precision network relative to the network to be quantized, in order to measure the precision loss of the mixed-precision network. Specifically, the verification data can be used as input data for both the mixed-precision network and the network to be quantized, and the difference between the output data of the two networks can be used as the loss data. In this case, step S205 may include the following sub-steps:

[0086] Step S2051: Determine multiple mixed-precision networks of the network to be quantized based on the resource usage threshold.

[0087] Reinforcement learning agents can generate various hybrid precision networks that meet resource usage requirements through policy networks. For example... Figure 5 As shown, the generated mixed-precision network is as follows: the first layer uses floating-point arithmetic (FP), the Nth layer uses fixed-point arithmetic (8-bit), and the last layer uses floating-point arithmetic (FP).

[0088] In this embodiment, step S2051 can be implemented as follows: if the resource usage of the mixed-precision network exceeds a resource usage threshold, the precision of some layers of the mixed-precision network is reduced. The task of quantization strategy search is to search for quantization strategies within a given computing power (flops) limit. The computing power used by the quantized network during operation cannot exceed the given computing power limit. If a certain exploration path causes the computing power of the quantized network to exceed the limit, some layers of the quantized network can be converted to low-bit computation, such as randomly forcing some layers from floating-point computation to fixed-point (e.g., 8-bit) computation quantization. This processing method ensures that the quantized network meets the resource usage requirements. The computing power usage of the neural network can be calculated using relatively mature existing technologies, which will not be elaborated here.

[0089] In one example, the observation features of the reinforcement learning algorithm may include: the number of bits for computational precision in multiple network layers of the mixed-precision network, the computational cost of the multiple network layers, and the identifier of the currently quantized network layer. This is a globally observable observation space, where the observation feature of each network layer is the number of quantized bits (int2, 4, 8, fp16, fp32, etc.) and the computational cost (flops) of that layer, forming a 2D feature. If the network has N layers that need quantization, the observation space is the superposition of these N 2D features. Additionally, there is another dimension: which layer's policy is being set. This global observation feature input allows the reinforcement learning agent to obtain the current quantization information of the entire network, which is more beneficial for modeling inter-layer relationships and improving robustness compared to local observation features.

[0090] Step S2053: Based on the verification dataset, determine the loss data of the hybrid precision network relative to the quantization network.

[0091] In this embodiment, based on the validation data, the first output data of the network to be quantized and the second output data of the mixed-precision network are determined; based on the first and second output data, the loss data corresponding to the validation data is determined. The validation data is used as input data for the network to be quantized and the mixed-precision network, and the first and second output data are obtained through these two networks. The validation dataset includes multiple validation data sets, and the loss data corresponding to each validation data set forms a data distribution loss. In this embodiment, the distribution difference between the outputs of the two networks is used as the evaluation criterion for the effectiveness of this quantization strategy.

[0092] In one example, the mixed-precision network determined by the policy network is an unlabeled quantized network. In this case, step S205 may further include the following sub-step: calibrating the mixed-precision network based on the validation dataset.

[0093] like Figure 4 As shown, taking a network layer (such as a convolutional layer or a fully connected layer) in a network to be quantized as an example, there are generally two places that need quantization: the network input and the network parameters. Here, we take the network parameters as an example. The data type of the network parameters is floating-point, assuming the range of floating-point data is [a, b]. If this layer is quantized to fixed-point computation, such as the data type of the network parameters being Int8, the range of Int8 fixed-point data is generally (-127, 127). It can be seen that a linear mapping is required from the network to be quantized to the quantized network, which can be expressed by the formula y = scale * x + bias. Here, x represents the floating-point parameter value, y represents the fixed-point parameter value, scale represents the scaling ratio of the data mapping (referred to as the mapping ratio), and bias represents the offset of the data mapping (referred to as the mapping offset). The process of calibrating a mixed-precision network is to determine the mapping ratio scale and the mapping offset bias.

[0094] In practice, the calibration process for the mixed-precision network based on the verification dataset is as follows: the verification data is used as the input data for the uncalibrated mixed-precision network, and the mapping ratio and mapping offset are determined based on the output data of the uncalibrated mixed-precision network.

[0095] The method provided in this embodiment calibrates the mixed-precision network based on the verification data to obtain a calibrated quantization network. The verification data is then input into the calibrated quantization network to obtain the second output data.

[0096] Step S2055: Based on the loss data, determine the target mixed precision network from multiple mixed precision networks.

[0097] When evaluating the quality of a quantization strategy using reinforcement learning algorithms, a reward function can be used for evaluation, such as directly using the mean of the loss data corresponding to each validation data point to describe the quality of the quantization network.

[0098] In one example, step S2055 can be implemented as follows: the mean of the loss data at each point is used as the exponent of the exponential function, and the target mixed-precision network is determined from multiple mixed-precision networks based on the exponential function value. Specifically, the value of the mixed-precision network can be determined based on the exponential function value; and the target mixed-precision network can be determined based on the value. In this way, by introducing a non-linearly changing exponential function, the reward function is transformed into a convex function, such as e^abs(), which better matches the changes in the accuracy of the quantified network.

[0099] In one example, step S2055 can be implemented as follows: A target mixed-precision network is determined from multiple mixed-precision networks based on the loss data and the resource usage of the mixed-precision network. Specifically, the value of the mixed-precision network can be determined based on the loss data and the resource usage of the mixed-precision network; the target mixed-precision network is then determined based on the value. In a further implementation, weights can be set for the loss data and the resource usage, comprehensively considering both the precision loss and resource usage of the quantization network to obtain a target quantization network that meets the user's needs.

[0100] Experiments have shown that the method provided in this application embodiment can converge and determine the target quantized network for a lightweight convolutional neural network with a search space of approximately 2^53 power only a few hundred explorations are required.

[0101] As can be seen from the above embodiments, the mixed-precision neural network quantization method provided in this application determines the mixed-precision quantization strategy by combining reinforcement learning and validation data generation techniques. This allows the distribution difference between the outputs of the mixed-precision network and the network to be quantized to be determined based on an automatically generated validation dataset, which serves as a standard for evaluating the effectiveness of the quantization strategy. This approach eliminates the need for users to provide validation data to verify the accuracy of the quantized network, reducing the time spent verifying the accuracy of the quantized network. It also eliminates the need for users to fine-tune the quantized network determined through reinforcement learning, enabling a quantized network that achieves a significant reduction in computational cost with a smaller loss of precision. Furthermore, it simplifies the network quantization process, achieving seamless model quantization strategy search for the user. Therefore, while ensuring minimal loss of model precision, it effectively improves model quantization efficiency, protects user data privacy, and reduces labor costs. In addition, this approach enables the converted network layers using fixed-point computation to achieve truly low-bit convolutional computation, achieving a significant reduction in computational cost with a smaller loss of precision, thus effectively reducing model precision loss.

[0102] Second Embodiment

[0103] In the above embodiments, a mixed-precision neural network quantization method is provided. Correspondingly, this application also provides a mixed-precision neural network quantization apparatus. This apparatus corresponds to the embodiments of the above method. Since the apparatus embodiments are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described below are merely illustrative.

[0104] This application also provides a mixed-precision neural network quantization device, comprising:

[0105] The network acquisition unit is used to acquire the network to be quantized.

[0106] The data generation unit is used to generate a validation dataset based on the network to be quantized.

[0107] A reinforcement learning unit is used to determine the target mixed-precision network of the network to be quantized based on the validation dataset using a reinforcement learning algorithm.

[0108] Optionally, the reinforcement learning unit includes:

[0109] The network quantization subunit is used to determine multiple mixed-precision networks of the network to be quantized based on a resource usage threshold.

[0110] The loss determination subunit is used to determine the loss data of the mixed precision network relative to the quantized network based on the validation dataset.

[0111] A network selection sub-unit is used to determine a target mixed-precision network from multiple mixed-precision networks based on the loss data.

[0112] Optionally, the loss determination subunit includes:

[0113] The prediction subunit is used to determine the first output data of the network to be quantized and the second output data of the mixed precision network based on the validation dataset.

[0114] A calculation subunit is used to determine the loss data based on the first output data and the second output data.

[0115] Optionally, the network selection sub-unit is specifically used to take the mean of the loss data at each point as the exponent value of the exponential function, and determine the target mixed precision network from multiple mixed precision networks based on the exponential function value.

[0116] Optionally, the network selection sub-unit is specifically used to determine a target mixed-precision network from multiple mixed-precision networks based on the loss data and the resource usage of the mixed-precision network.

[0117] Optionally, the reinforcement learning unit further includes:

[0118] The calibration subunit is used to calibrate the mixed-precision network based on the validation dataset.

[0119] Optionally, the network quantization subunit is specifically used to reduce the precision of some layers of the mixed precision network if the resource usage of the mixed precision network is greater than the resource usage threshold.

[0120] Optionally, the network quantization subunit is specifically used to determine the partial layers in a random manner if the resource usage of the mixed precision network is greater than the resource usage threshold.

[0121] Optionally, the observation features of the reinforcement learning algorithm include: the number of bits of computational precision of multiple network layers of the mixed precision network, the computational cost of the multiple network layers, and the identifier of the currently quantized network layer.

[0122] Optional, also includes:

[0123] The threshold determination unit is used to determine the resource usage threshold of the network to be quantized based on the device resource data of the target device of the network to be quantized.

[0124] The reinforcement learning unit is specifically used to determine the target mixed precision network based on the verification dataset and the resource usage threshold using a reinforcement learning algorithm.

[0125] A network deployment unit is used to deploy the target mixed-precision network to the target device.

[0126] Third Embodiment

[0127] In the above embodiments, a mixed-precision neural network quantization method is provided. Correspondingly, this application also provides an electronic device. This device corresponds to the embodiments of the above method. Since the device embodiments are basically similar to the method embodiments, the description is relatively simple, and relevant details can be found in the description of the method embodiments. The device embodiments described below are merely illustrative.

[0128] This application also provides an electronic device, including a processor and a memory. The memory stores a program for implementing the above-described mixed-precision neural network quantization method, and the terminal, when powered on, runs the program of the method through the processor.

[0129] Fourth embodiment

[0130] In the above embodiments, a mixed-precision neural network quantization method is provided. Correspondingly, this application also provides a mixed-precision neural network quantization system. This system corresponds to the embodiments of the above method. Since the system embodiments are basically similar to the method embodiments, the description is relatively simple; relevant details can be found in the descriptions of the method embodiments. The system embodiments described below are merely illustrative.

[0131] This application also provides a hybrid precision neural network quantization system, including: a network construction device, the hybrid precision neural network quantization device described in the above embodiments, and a network deployment device.

[0132] The network construction device is used to learn a machine learning model based on a neural network; the network deployment device is used to deploy the target mixed-precision neural network obtained by the mixed-precision neural network quantization device to the device side for operation.

[0133] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0134] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0135] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0136] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0137] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A mixed-precision neural network quantization method, characterized in that, include: Obtain a neural network-based prediction model as the network to be quantized; Based on the network to be quantized, a validation dataset for the prediction model is generated, including: when the input data of the network to be quantized is image data, the validation dataset generated based on the network to be quantized is a validation image set; the parameters of the network to be quantized are learned from the training dataset provided by the user, and the validation data generated based on the network to be quantized includes data with characteristics similar to the training data provided by the user; the training data and the validation data are used to detect the effect of model training. Using a reinforcement learning algorithm, a target mixed-precision network for the network to be quantized is determined based on the validation dataset. This includes: determining multiple mixed-precision networks for the network to be quantized based on a resource usage threshold; determining the loss data of the mixed-precision networks relative to the network to be quantized based on the validation dataset; and determining the target mixed-precision network from the multiple mixed-precision networks based on the loss data.

2. The method according to claim 1, characterized in that, The step of determining the loss data of the mixed-precision network relative to the quantized network based on the validation dataset includes: Based on the validation dataset, determine the first output data of the network to be quantized and the second output data of the mixed precision network; The loss data is determined based on the first output data and the second output data.

3. The method according to claim 1, characterized in that, The step of determining the target mixed-precision network from multiple mixed-precision networks based on the loss data includes: The mean of the loss data at each point is used as the exponent of the exponential function. Based on the exponential function value, the target mixed-precision network is determined from multiple mixed-precision networks.

4. The method according to claim 1, characterized in that, The step of determining the target mixed-precision network from multiple mixed-precision networks based on the loss data includes: Based on the loss data and the resource usage of the mixed-precision network, a target mixed-precision network is determined from multiple mixed-precision networks.

5. The method according to claim 1, characterized in that, The step of determining the target mixed-precision network of the network to be quantized based on the validation dataset using a reinforcement learning algorithm further includes: The mixed-precision network is calibrated based on the validation dataset.

6. The method according to claim 1, characterized in that, The step of determining multiple mixed-precision networks of the network to be quantized based on a resource usage threshold includes: If the resource usage of the hybrid precision network exceeds a resource usage threshold, the precision of some layers of the hybrid precision network is reduced.

7. The method according to claim 6, characterized in that, The reduction of the accuracy of certain layers in the hybrid precision network includes: The partial layers are determined in a random manner.

8. The method according to claim 1, characterized in that, The observation features of the reinforcement learning algorithm include: the number of bits of computational precision in multiple network layers of the mixed precision network, the computational cost of the multiple network layers, and the identifier of the currently quantized network layer.

9. The method according to claim 1, characterized in that, Also includes: Based on the device resource data of the target device in the network to be quantized, determine the resource usage threshold of the network to be quantized; The target mixed-precision network is determined using a reinforcement learning algorithm based on the validation dataset and the resource usage threshold. Deploy the target mixed-precision network to the target device.

10. A hybrid precision neural network quantization device, characterized in that, include: The network acquisition unit is used to acquire a neural network-based prediction model as the network to be quantized. A data generation unit is configured to generate a validation dataset for the prediction model based on the network to be quantized, including: when the input data of the network to be quantized is image data, the validation dataset generated based on the network to be quantized is a validation image set; the parameters of the network to be quantized are learned from the training dataset provided by the user, and the validation data generated based on the network to be quantized includes data with characteristics similar to the training data provided by the user; the training data and the validation data are used to detect the effect of model training. A reinforcement learning unit is used to determine a target mixed-precision network of the network to be quantized based on the validation dataset using a reinforcement learning algorithm. The reinforcement learning unit includes: a network quantization subunit, used to determine multiple mixed-precision networks of the network to be quantized based on a resource usage threshold; a loss determination subunit, used to determine the loss data of the mixed-precision network relative to the network to be quantized based on the validation dataset; and a network selection subunit, used to determine the target mixed-precision network from the multiple mixed-precision networks based on the loss data.

11. An electronic device, characterized in that, include: processor; as well as A memory for storing a program for implementing the mixed-precision neural network quantization method according to any one of claims 1-9, wherein the electronic device is powered on and the program of the method is run by the processor.

12. A hybrid precision neural network processing system, characterized in that, include: A network construction apparatus, a hybrid precision neural network quantization apparatus according to claim 10, and a network deployment apparatus.

Images