Neural network model training method, neural network model application method, neural network model training device, neural network model application device, storage medium and computer program product

Abstract

The invention provides a training method, an application method, a training device, an application device, a storage medium and a computer program product of a neural network model. The training method comprises a pre-training step of pre-training the neural network model so that the neural network model comprises at least one quantization unit, and each quantization unit comprises a plurality of different quantization bit widths; in the calculation step, the sensitivity of the quantization units is calculated, the quantization bit width of each quantization unit is updated based on the calculated sensitivity, quantization parameters are updated, and therefore a neural network model with mixed precision is generated, and the sensitivity represents the influence degree of the quantization bit width of the quantization units on network output; and a retraining step: retraining the generated neural network model with the mixed precision.

Description

Technical Field

[0001] The present invention relates to the field of modeling of deep neural network models (Deep Neural Networks, DNN). Background Art

[0002] A deep neural network model is a model with a complex network architecture in the field of artificial intelligence and is one of the most widely used architectures. Common neural network models include the Convolutional Neural Network (CNN) model. Deep neural network models are widely used in the fields of computer vision, computer hearing, and natural language processing, such as image classification, object recognition and tracking, image segmentation, speech recognition, and so on. Deep neural network models have a large number of learnable parameters. The linear processing units and nonlinear processing units within the deep neural network model can be interlaced, their topological relationships can be complex, and they have the ability to represent arbitrarily complex functions. After a specific learning process, deep neural network models can have powerful recognition and generalization capabilities.

[0003] On the other hand, running deep neural network models requires a large amount of memory overhead and abundant processor resources. Although deep neural network models can achieve good performance targets on GPU-based workstations or servers, they are generally not suitable for running on resource-constrained embedded devices such as smartphones, tablets, and various handheld devices.

[0004] In order to solve the above problems, the following solutions can usually be used to optimize the model:

[0005] Pruning / sparseness: During network training, unimportant connections are pruned, reducing most of the network weights to zero and storing the model in a sparse manner. Pruning can be performed at different levels, such as weight level, channel level, layer level, etc., depending on the task.

[0006] Low-rank factorization: Use structured matrices for low-rank factorization, so that the original dense full-rank matrix can be represented as a combination of several low-rank matrices, and the low-rank matrix can be decomposed into the product of small-scale matrices.

[0007] Quantization: Use a lower bit width (1 bit, 2 bits, or 8 bits) to represent 32-bit or higher-precision floating-point numbers, thereby mapping continuous real values ​​in network parameters and feature maps to discrete integer values, significantly reducing parameter storage space and memory usage, speeding up calculations, and reducing device power consumption.

[0008] Knowledge distillation: Unlike pruning and quantization in model compression, knowledge distillation builds a lightweight, smaller model and uses the supervisory information from a larger, higher-performing model to train the smaller model, aiming to achieve better performance and accuracy. Specifically, knowledge from a large, high-performing network is transferred to a smaller network through transfer learning, enabling the smaller network to achieve comparable performance to the larger model, thus reducing computational costs.

[0009] Design a lightweight model architecture (compact model architecture): Construct a specially structured network layer and train it from scratch to obtain network performance suitable for deployment on resource-limited devices. There is no need to specially store pre-trained models or improve performance through fine-tuning, which reduces time costs and has the characteristics of small storage volume, low computational complexity and good network performance.

[0010] Among the five technical solutions mentioned above, deep neural network quantization is becoming increasingly important in reducing the energy and memory usage of deep neural network inference because low-precision operations can simultaneously reduce the memory usage of deep neural network inference, increase throughput, and reduce latency. In practical applications, a higher quantization bit width will produce lower quantization error, but the latency of deep neural network inference will be higher. To reduce quantization error and achieve a balance between efficiency and accuracy, automatically determining the optimal layer-by-layer precision allocation based on neural network search technology has shown good results.

[0011] Prior art has proposed a framework for searching mixed-precision networks, as described in Zhaowei Cai and Nuno Vasconcelos, Rethinking Differentiable Search for Mixed-Precision Neural Networks. This framework has the following features: it is based on a differentiable search algorithm; to avoid trivial choices of the highest bit width, a complexity budget loss is added to the total loss function to constrain the learning process; the network's learnable parameters and bit-width weight parameters are learned simultaneously; the weighted sum of the quantized input is applied as the new input, and the weighted sum of the quantized weights serves as the new kernel weight, allowing the convolution operator to be computed as usual without additional computational cost. This method adds a complexity constraint to the task loss and multiplies it by the Lagrange multiplier. However, the dimensionality of the weights is too large to accurately compute the Lagrange multiplier, necessitating expert experience, which consumes a significant amount of resources. The search results are heavily dependent on the complexity constraint, namely the Lagrange multiplier, and the final converged search results will not accurately meet the computational cost constraint.

[0012] WO2021CN077801W proposes a data processing method, comprising the following steps: based on the obtained structural information of the network model, describing the necessary level or non-essential level of each tag of the network model; determining the necessary level quantization bit width range and the non-essential level quantization bit width range respectively according to the hardware source information to be deployed; determining each optimal quantization bit width of the network model within the quantization bit width range; training the network model based on each optimal quantization bit width of the network model to obtain the optimal network model, and using the optimal network model for data processing. The bit width granularity adopted by this method is relatively large, and it has only two types of layers: a basic layer and a non-basic layer for determining the bit width distribution in the network. The allocation of quantization bit widths between layers is based on a full-precision model, and does not consider the influence of the distribution of quantized data. When grouping the basic layer and the non-basic layer, the threshold needs to be manually designed. Hardware source information that requires complex calculations is required to determine the bit width range of the basic layer and the non-basic layer. Summary of the Invention

[0013] The present invention provides a neural network training method, which can search for a high-precision neural network model that meets the computational cost constraints under limited search cost conditions.

[0014] According to one aspect of the present invention, a training method for a neural network model is provided, characterized in that the method includes: a pre-training step of pre-training the neural network model so that the neural network model includes at least one quantization unit, wherein each quantization unit contains multiple different quantization bit widths; a calculation step of calculating the sensitivity of the quantization unit, and updating the quantization bit width of each quantization unit and the quantization parameter based on the calculated sensitivity, thereby generating a mixed-precision neural network model, wherein the sensitivity represents the degree of influence of the quantization bit width of the quantization unit on the network output; and a retraining step of retraining the generated mixed-precision neural network model.

[0015] According to another aspect of the present invention, a method for applying a neural network model is provided, comprising: storing a neural network model trained based on the above-mentioned training method; receiving a data set corresponding to the task requirements that can be performed by the stored neural network model; operating the data set in each layer from top to bottom in the stored neural network model, and outputting the result.

[0016] According to another aspect of the present invention, there is provided an application device for a neural network model, comprising: a storage module configured to store a neural network model trained based on the above-mentioned training method; a receiving module configured to receive a data set corresponding to the task requirements that can be performed by the stored neural network model; and a processing module configured to operate the data set in each layer from top to bottom in the stored neural network model and output the result.

[0017] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing instructions is provided, which, when executed by a computer, causes the computer to perform a training method based on the above-mentioned neural network model.

[0018] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0020] Figure 1 A block diagram illustrating a hardware configuration according to an exemplary embodiment of the present invention is illustrated.

[0021] Figure 2 A flowchart illustrating a method for training a neural network model according to a first exemplary embodiment of the present invention is illustrated.

[0022] Figure 3 Illustrate the neural network model architecture.

[0023] Figure 4-5 A flowchart illustrating a method for training a neural network model according to a first exemplary embodiment of the present invention is illustrated.

[0024] Figure 6 A schematic diagram illustrating a training system according to a second exemplary embodiment of the present invention.

[0025] Figure 7 A schematic diagram illustrating a training device according to a third exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present invention will be described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of the embodiments are described in this specification. However, it should be understood that many implementation-specific settings must be made in the process of implementing the embodiments in order to achieve the developer's specific goals, such as meeting those restrictions related to equipment and services, and these restrictions may vary depending on the implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the content of this invention.

[0027] It should also be noted here that in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show processing steps and / or system structures that are closely related to at least the solution according to the present invention, while other details that are not very relevant to the present invention are omitted.

[0028] (Hardware Structure)

[0029] First, refer to Figure 1 Describes hardware configurations that can implement the techniques described below.

[0030] The hardware configuration 100 includes, for example, a central processing unit (CPU) 110, a random access memory (RAM) 120, a read-only memory (ROM) 130, a hard disk 140, an input device 150, an output device 160, a network interface 170, and a system bus 180. In one implementation, the hardware configuration 100 may be implemented by a computer, such as a tablet computer, a laptop computer, a desktop computer, or other suitable electronic devices.

[0031] In one implementation, the apparatus for training a neural network model according to the present invention is constructed by hardware or firmware and is used as a module or component of the hardware configuration 100. In another implementation, the method for training a neural network model according to the present invention is constructed by software stored in the ROM 130 or the hard disk 140 and executed by the CPU 110.

[0032] The CPU 110 is any suitable programmable control device (such as a processor) and can perform various functions to be described below by executing various application programs stored in the ROM 130 or the hard disk 140 (such as a memory). The RAM 120 is used to temporarily store programs or data loaded from the ROM 130 or the hard disk 140, and is also used as a space in which the CPU 110 executes various processes and other available functions. The hard disk 140 stores various information such as an operating system (OS), various applications, control programs, sample images, trained neural network models, predefined data (e.g., threshold values ​​(THs)), etc.

[0033] In one implementation, the input device 150 is used to allow a user to interact with the hardware configuration 100. In one example, the user can input a sample image and a label of the sample image (e.g., object region information, object category information, etc.) through the input device 150. In another example, the user can trigger a corresponding process of the present invention through the input device 150. Furthermore, the input device 150 can take various forms, such as buttons, a keyboard, or a touch screen.

[0034] In one implementation, the output device 160 is used to store the final trained neural network model in, for example, the hard disk 140 or to output the final generated neural network model to subsequent image processing such as object detection, object classification, image segmentation, etc.

[0035] The network interface 170 provides an interface for connecting the hardware configuration 100 to a network. For example, the hardware configuration 100 can communicate data with other electronic devices connected via the network via the network interface 170. Alternatively, a wireless interface can be provided for the hardware configuration 100 to enable wireless data communication. The system bus 180 can provide a data transmission path for mutually transmitting data between the CPU 110, RAM 120, ROM 130, hard disk 140, input device 150, output device 160, and network interface 170, etc. Although referred to as a bus, the system bus 180 is not limited to any specific data transmission technology.

[0036] The above hardware configuration 100 is merely illustrative and is in no way intended to limit the present invention, its application, or use. Figure 1 Only one hardware configuration is shown. However, multiple hardware configurations may be used as needed, and the multiple hardware configurations may be connected via a network. In this case, the multiple hardware configurations may be implemented, for example, by a computer (e.g., a cloud server) or by an embedded device such as a camera, a camcorder, a personal digital assistant (PDA), or other suitable electronic device.

[0037] Next, various aspects of the present invention will be described.

[0038] <First Exemplary Embodiment>

[0039] The following will refer to Figures 2 to 5 The training method of the neural network model according to the first exemplary embodiment of the present invention is described in detail as follows: The first embodiment shows the main workflow of searching for a neural network model with a smaller bit width according to the present invention.

[0040] See also Figure 2 ,The training method is described in detail as follows.

[0041] Step S2010: Construct a neural network model.

[0042] Specifically, in this step, a neural network model is created based on specific task objectives, such as image classification or instance segmentation. Existing neural network models can be used, or they can be obtained through general search methods such as DARTS. Based on this, corresponding quantization targets are constructed for all layers in the current neural network. For each quantization target, all possible bit-width paths are constructed, thereby constructing the desired neural network model. This process can be considered the initialization of the neural network model.

[0043] Step S2020: Use the training database to train the neural network model generated in step S2010.

[0044] The training of a neural network model is a cyclical and repetitive process. Each iteration includes three steps: forward calculation, backward calculation, and parameter update. Among them, the forward calculation is to input a batch of data to be trained into the network, calculate it from top to bottom and layer by layer in the network model, and obtain the network output result. The backward calculation is the process of calculating the loss function based on the true value of the batch of training data and the result of the network output, and passing the gradient of the loss function from the last layer of the network to the front. The parameter update is mainly based on the back-propagated gradient value and the corresponding optimization algorithm to calculate the updated value of the current parameter. This step trains the neural network model until the network converges or the exit condition is met.

[0045] Figure 3 A simple neural network model architecture is illustrated (the specific network architecture is not shown). After the data to be trained (feature map) x is input into the neural network model F, x is calculated layer by layer from top to bottom in the neural network model F, and finally the output from the neural network model F meets certain distribution requirements.

[0046] If the difference between the actual output of the neural network model and the expected output does not exceed a predetermined threshold, it indicates that the weights in the neural network model are optimal, the performance of the trained neural network model has reached the expected performance, and the training of the neural network model is complete. Conversely, if the difference between the actual output of the neural network model and the expected output exceeds a predetermined threshold, it is necessary to continue the backpropagation process. That is, based on the difference between the actual output and the expected output, the backpropagation process is performed layer by layer from bottom to top in the neural network model to update the weights in the model so that the performance of the network model after the weight update is closer to the expected performance.

[0047] The neural network model applicable to the present invention can be any known model, such as a convolutional neural network model, a recurrent neural network model, and a graph neural network model. The present invention does not limit the type of network model.

[0048] The computational accuracy of the neural network model applicable to the present disclosure can be of arbitrary accuracy, both high precision and low precision. The terms "high precision" and "low precision" refer to the relative high and low precision, and are not limited to specific numerical values. For example, high precision can be a 32-bit floating point type, and low precision can be a 1-bit fixed point type. Of course, other accuracies such as 16-bit, 8-bit, 4-bit, and 2-bit are also included in the computational accuracy range applicable to the scheme of the present disclosure. The term "computational accuracy" can refer to the accuracy of the weights in the neural network model, or it can refer to the accuracy of the input x to be trained, and the present disclosure does not limit this. The neural network model described in the present disclosure can be a binary neural network model (BNNs), and of course, it is not limited to neural network models of other computational accuracy.

[0049] Step S2030: Construct an initial quantized neural network model based on the neural network model trained in step S2020.

[0050] In this step, based on the neural network model trained in step S2020, the initial quantized neural network model adopts a similar network structure, and the path with the maximum bit width is selected for the quantization target of each layer. The network parameters of each layer are directly inherited from the neural network model trained in step S2020.

[0051] Step S2040: Determine the quantization target category for which the bit width is to be reduced.

[0052] In this step, the quantization targets of each layer are usually divided into filter parameters and feature maps. In this step, a random algorithm is used to select the quantization target for the subsequent bit width search, or a pre-set method can be used to select the quantization target for the subsequent bit width search.

[0053] Step S2050: Calculate the sensitivity of the low bit width of the quantization target of each layer of the current quantized neural network model to the impact on the network output.

[0054] In this step, given a set of training data, first, it is input into the current quantized neural network model for forward propagation, and the gradient of its initial last layer output feature map is recorded; then, for the selected quantization target of each layer, its current bit width is reduced to the adjacent smaller bit width, while keeping other layers unchanged, and then the training data is input into the quantized neural network model after the bit width is reduced for forward propagation, and the gradient of its corresponding new last layer output feature map is recorded; finally, the sensitivity of the quantization target of each layer is measured by the gradient change of the output feature map of the last layer of the network before and after the bit width change. The smaller the gradient change, the less sensitive the quantization target of the layer is to the network output. The sensitivity measurement method can use the sum of the absolute values ​​of the output gradient changes before and after, defined as follows:

[0055]

[0056] Where L is the objective function, f LastLayer It is the feature map output by the last layer of the network, L(layer i ) is the loss function when the bit width of the quantization target of layer i is not reduced, L(lower(layer i )) is the loss function corresponding to reducing the bit width of the quantization target of the i-th layer.

[0057] In addition, sensitivity metrics can also define other metrics based on gradient information, such as single network pruning (snip), gradient signal preservation (grasp), joint flow pruning (synflow), Fisher information, batch normalization scale factor, L2 norm, Jacobian determinant, etc. For example:

[0058] The calculation formula for single network pruning (snip) is:

[0059] The calculation formula for gradient signal preservation (grasp) is:

[0060] The calculation formula for joint flow pruning (synflow) is:

[0061] Where L is the objective function, θ is the learnable parameter in the operation, and H is the Hessen matrix.

[0062] Sensitivity can also be normalized by some indicators, including the number of floating-point operations, the number of multiplication and accumulation operations, the total memory consumption, the total amount of computation consumption, etc.

[0063] Step S2060: Reduce the bit width of the quantization target of the lowest sensitivity layer to generate a new quantized neural network model.

[0064] In this step, the sensitivity calculated for each layer is first sorted, and the layer with the lowest sensitivity is selected; then, the bit width of the quantization target of the lowest sensitivity layer is reduced to the adjacent smaller bit width, thereby obtaining a new quantized neural network model.

[0065] Step S2070: Determine whether the current quantized neural network model meets the task objectives.

[0066] In this step, the task goal can be the amount of computation and the amount of parameters of the quantized neural network, and specifically, the number of floating-point operations, the total amount of computation consumed, the total amount of memory consumed, the hardware constraints, the training cost, etc. of the current quantized neural network model. If the current quantized neural network model meets the task goal, the bit width search process ends, indicating that a qualified quantized neural network has been found, and the process proceeds to step S2090; otherwise, the process proceeds to step S2080 and repeats the search process. The filter parameter quantization bit width and the feature map quantization bit width can be obtained in this step, or they can be obtained by repeating steps S2080 to S2090 multiple times according to a random algorithm or a pre-set method.

[0067] Step S2080: Use the training database to adjust the current quantized neural network model.

[0068] In this step, part of the training set data is selected and the current quantized neural network is iteratively trained to achieve the purpose of fine-tuning its parameters.

[0069] Step S2090: Retrain the quantized neural network model using the training database.

[0070] In this step, when a quantized neural network model that meets the task objectives has been obtained, the searched neural network model is retrained according to the specific task requirements and all training set data until the network converges or the exit condition is met.

[0071] Through the solution of this exemplary embodiment, a highly accurate quantized neural network model that meets the computational overhead constraint can be searched.

[0072] According to this exemplary embodiment, a neural network is first constructed based on the scenario-specific design requirements and search space. The quantization units in each layer of this network include all possible quantization bit widths in the bit width search space, representing an integrated network structure of all candidate mixed-precision network structures. Based on the task objectives, training sets, and labeled data, each quantization unit in each layer selects one of the bit widths to quantize the input during forward propagation. During reverse gradient calculation, a gradient optimization algorithm is used to iteratively update all parameters of the network model until the network converges or an exit condition is met.

[0073] Then, the structure and weight parameters of the aforementioned network are inherited as the initial quantized neural network model. Based on the specified sensitivity measurement index, the sensitivity of the quantization unit of each layer in the current quantized neural network model to the final output of the network is calculated. The sensitivity calculated for each quantization unit is compared and sorted, and the quantization unit with the highest sensitivity is selected to adjust its bit width (increase to a larger bit width) to obtain a new quantized neural network model. All learnable parameters of the new quantized neural network model are fine-tuned based on the training set and labeled data of the task. The above search process is iterated until all neural network models that meet the task-related constraints and computational constraints are found.

[0074] Finally, the final searched quantized neural network model is retrained based on all the training sets and labeled data of the task.

[0075] According to this exemplary embodiment, a highly accurate quantized neural network model that satisfies computational overhead constraints can be searched for.

[0076] Tables 1 through 3 compare the technical performance of the method according to this exemplary embodiment with the prior art (EdMIPS) in classification tasks on the ImageNet training and evaluation sets, using the convolutional neural network architecture RestNet18. Table 1 compares the accuracy results of the method according to the prior art and the exemplary embodiment of this invention on a common dataset.

[0077] Table 1

[0078]

[0079]

[0080] Table 2 is a comparison of the feature map bit widths of the neural network models obtained according to the prior art and the method of the exemplary embodiment of the present invention.

[0081] Table 2

[0082] The feature map width of the searched model EdMIPS 2,2,2,2,2,2,4,2,2,3,3,3,2,2,3,3,3,2,2 The method of this embodiment 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2

[0083] Table 3 is a comparison of the filter bit widths of the neural network model obtained according to the prior art and the method of the exemplary embodiment of the present invention.

[0084] Table 3

[0085] Comparison of filter bit widths of the searched models EdMIPS 1,1,1,1,3,2,4,2,1,3,2,4,2,1,3,2,4,2,2 The method of this embodiment 4,1,1,1,3,2,4,2,1,3,2,4,2,1,3,2,4,2,2

[0086] Compared with the prior art, the method of the present invention has the following advantages:

[0087] (1) The method according to the exemplary embodiment of the present invention does not require allocating hyperparameters that affect the bit width allocation result.

[0088] (2) In the method according to the exemplary embodiment of the present invention, the adjustment of the quantization bit width is based on a well-converged quantized neural network model, which causes minimal interference to the model accuracy.

[0089] (3) The final quantization bit width allocation result of the method according to the exemplary embodiment of the present invention can meet the target of computational overhead.

[0090] Therefore, according to the mixed-precision neural network search method with hierarchical precision allocation of this exemplary embodiment, a high-precision neural network model that meets the computational cost constraints can be searched under limited search cost conditions.

[0091] Modification 1

[0092] The following will refer to Figure 4 The training method of the neural network model of this exemplary embodiment is described. This exemplary embodiment shows the main workflow of searching for a neural network model with a larger bit width. The specific description of the training method is as follows.

[0093] Step S4010: Similar to step S2010, a neural network model is constructed in this step.

[0094] Step S4020: Similar to step S2020, in this step the neural network model constructed in S4010 is trained using the training database.

[0095] Step S4030: constructing an initial quantized neural network model structure according to the neural network model.

[0096] In this step, based on the neural network model trained in step S4030, the initial quantized neural network model constructed adopts a similar network structure. The difference from the aforementioned exemplary embodiment is that the path with the smallest bit width is selected for the quantization target of each layer, and the network parameters of each layer are directly inherited from the neural network model.

[0097] Step S4040: Determine the quantization target category for which the bit width is to be increased.

[0098] The quantization targets of each layer are usually divided into filter parameters and feature maps. In this step, a cyclic alternation method is used to select the quantization targets for the subsequent bit width search.

[0099] Step S4050: Calculate the sensitivity of the high bit width of the quantization target of each layer of the current quantized neural network model to the network output.

[0100] In this step, given a set of training data, it is first fed into the current quantized neural network model for forward propagation, and the gradient of its initial final layer output feature map is recorded. Next, for each layer's selected quantization target, its current bit width is increased to the adjacent larger bit width, while keeping the other layers unchanged. The training data is then fed into the quantized neural network model with the increased bit width for forward propagation, and the gradient of its corresponding new final layer output feature map is recorded. Finally, the sensitivity of each layer's quantization target is measured by the gradient change of the network output before and after the bit width change. The smaller the gradient change, the less sensitive the quantization target of that layer is to the network output.

[0101] Step S4060: Increase the bit width of the quantization target of the highest sensitivity layer to generate a new quantized neural network model.

[0102] In this step, the sensitivity calculated for each layer is first sorted, and the layer with the highest sensitivity is selected; then, the bit width of the quantization target of the highest sensitivity layer is increased to the adjacent larger bit width, thereby obtaining a new quantized neural network model.

[0103] Step S4070: Determine whether the current quantized neural network model exceeds the upper limit of the model cost.

[0104] In this step, the model cost can be the amount of computation and the number of parameters of the quantized neural network. If the current quantized neural network exceeds the upper limit of the model cost, the bit width search process ends, indicating that the previous search has obtained a quantized neural network model that meets the conditions and has the highest possible performance, and the process proceeds to step S4090; otherwise, the process proceeds to step S4080 to continue the search process. The filter parameter quantization bit width and feature map quantization bit width can be obtained in this step, or they can be obtained by repeating steps S4040 to S4060 multiple times according to a random algorithm or a pre-set method.

[0105] Step S4080: Fine-tune the current quantized neural network model using the training database.

[0106] In this step, part of the training set data is selected and the current quantized neural network model is iteratively trained to achieve the purpose of fine-tuning its parameters.

[0107] Step S4090: Retrain the obtained quantized neural network model using the training database.

[0108] In this step, when a quantized neural network that meets the requirements has been searched, the searched neural network model is retrained according to the specific task requirements and all training set data until the network converges or the exit condition is met.

[0109] Through the solution of this exemplary embodiment, it is possible to gradually search for a network structure that is as small as possible and meets the accuracy requirements while maintaining the accuracy improvement as large as possible.

[0110] Modification 2

[0111] The following will refer to Figure 5 The training method of the neural network model of this exemplary embodiment is described. This embodiment shows the main workflow of the present disclosure for searching for a neural network model with a smaller bit width.

[0112] See also Figure 5 ,The training method is described in detail as follows.

[0113] Step S6010: Similar to step S2010, a neural network model is constructed.

[0114] Step S6020: Similar to step S2020, the neural network model generated in step S6010 is trained using the training database.

[0115] Step S6030: Similar to step S2030, an initial quantized neural network model is constructed based on the neural network model trained in step S6020.

[0116] Step S6040: Similar to step S2040, determine the quantization target category of the bit width to be reduced.

[0117] Step S6050: Similar to step S2050, calculate the sensitivity of the low bit width of the quantization target of each layer of the current quantized neural network model to the impact of the network output.

[0118] Step S6060: Derivation of the bit widths of all quantization targets in the neural network model under the current constraints. (Specify the current constraints and use an integer linear programming algorithm to derive the bit widths of all quantization targets in the neural network model.)

[0119] In this step, the current constraints and bit width search space are first calculated based on the search process. The search space of each layer of the neural network is a subset of the original search space with the currently searched bit width as the upper limit, that is, the candidate bit widths in the initial search space that are higher than the currently searched bit width are removed. Under the current constraints, with maximizing the performance of the neural network model (minimizing the network quantization sensitivity) as the optimization goal, the integer linear programming algorithm is used to derive the bit width of the quantization target for each layer. For the first search process, the current bit width search space is the original bit width search space itself.

[0120] Step S6070: Determine whether the current quantized neural network model meets the task objectives.

[0121] In this step, the task goal can be the amount of computation and the amount of parameters of the quantized neural network, and specifically, the number of floating-point operations, the total amount of computation consumed, the total amount of memory consumed, the hardware constraints, the training cost, etc. of the current quantized neural network model. If the current quantized neural network model meets the task goal, the bit width search process ends, indicating that a qualified quantized neural network has been found, and the process proceeds to step S6090; otherwise, the process proceeds to step S6080 and repeats the search process. The filter parameter quantization bit width and the feature map quantization bit width can be obtained in this step, or they can be obtained by repeating steps S6080 to S6090 multiple times according to a random algorithm or a pre-set method.

[0122] Step S6080: Use the training database to adjust the current quantized neural network model.

[0123] In this step, part of the training set data is selected and the current quantized neural network is iteratively trained to achieve the purpose of fine-tuning its parameters.

[0124] Step S6090: Retrain the quantized neural network model using the training database.

[0125] In this step, when a quantized neural network model that meets the task objectives has been obtained, the searched neural network model is retrained according to the specific task requirements and all training set data until the network converges or the exit condition is met.

[0126] Through the solution of this exemplary embodiment, a highly accurate quantized neural network model that meets the computational overhead constraint can be searched.

[0127] Tables 4 through 6 compare the technical performance of the method according to this exemplary embodiment with the prior art (LIMPQ) in classification tasks on the ImageNet training and evaluation sets, using the convolutional neural network architecture RestNet18. Table 4 compares the accuracy results of the method according to the prior art and the exemplary embodiment of this invention on a common dataset.

[0128] Table 4

[0129] Top-1 Top-5 Compression ratio LIMPQ 64.291 85.755 12.64 The method of this embodiment 64.532 85.890 12.67

[0130] Table 5 is a comparison of the feature map bit widths of the neural network models obtained according to the prior art and the exemplary embodiments of the present invention.

[0131] Table 5

[0132] The feature map width of the searched model LIMPQ 3,3,3,3,3,3,4,3,3,3,3,3,3,3,3,3,3,3,2 The method of this embodiment 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3

[0133] Table 6 is a comparison of the filter bit widths of the neural network model obtained according to the prior art and the method of the exemplary embodiment of the present invention.

[0134] Table 6

[0135] Comparison of filter bit widths of the searched models LIMPQ 3,3,3,3,3,3,5,3,3,3,3,4,3,3,2,2,4,2,2 The method of this embodiment 4,3,4,3,3,3,4,3,3,3,3,4,3,3,2,2,3,2,2

[0136] <Second Exemplary Embodiment>

[0137] Based on the aforementioned first exemplary embodiment, the second exemplary embodiment of the present invention describes a network model training system. This training system includes a terminal, a communication network, and a server. The terminal and server communicate via the communication network. The server uses a locally stored network model to online train the network model stored in the terminal, allowing the terminal to use the trained network model for real-time services. The following describes the various components of the training system of the second exemplary embodiment of the present invention.

[0138] The terminals in the training system can be embedded image acquisition devices such as security cameras, or they can be devices such as smartphones and PADs. Of course, the terminals can also be other terminals with stronger computing capabilities instead of terminals with weak computing capabilities such as embedded devices. The number of terminals in the training system can be determined according to actual needs. For example, if the training system is to train security cameras in a shopping mall, all security cameras in the shopping mall can be regarded as terminals. In this case, the number of terminals in the training system is fixed. For another example, if the training system is to train smartphones of users in the shopping mall, all smartphones connected to the shopping mall's wireless local area network can be regarded as terminals. In this case, the number of terminals in the training system is not fixed. In the second exemplary embodiment of the present invention, there is no limitation on the type and number of terminals in the training system, as long as the terminal can store and train the network model.

[0139] The servers in the training system can be high-performance servers with strong computing power, such as cloud servers. The number of servers in the training system can be determined according to the number of terminals it serves. For example, if the number of terminals to be trained in the training system is small or the geographical range of the terminal distribution is small, then the number of servers in the training system is small, such as only one server. If the number of terminals to be trained in the training system is large or the geographical range of the terminal distribution is large, then the number of servers in the training system is large, such as establishing a server cluster. In the second exemplary embodiment of the present invention, there is no limitation on the type and number of servers in the training system, as long as the server can store at least one network model and provide information for training the network model stored in the terminal.

[0140] The communication network in the second exemplary embodiment of the present invention is a wireless or wired network used to enable information transmission between the terminal and the server. Any network currently available for uplink / downlink transmission between a network server and a terminal can serve as the communication network in this embodiment. The second exemplary embodiment of the present invention does not limit the type of communication network or the communication method. Of course, the third exemplary embodiment of the present invention is also not limited to other communication methods. For example, a third-party storage area can be allocated for this training system. When a terminal and server want to transmit information to each other, the information to be transmitted is stored in the third-party storage area. The terminal and server periodically read the information in the third-party storage area to achieve information transmission between them.

[0141] The following combination Figure 6 , the online training process of the training system of the third exemplary embodiment of the present invention is described in detail. Figure 6 This example shows an example training system, assuming it includes a terminal and a server. The terminal can capture images in real time. Assuming the terminal stores a trainable network model capable of image processing, and the server stores the same network model, the training process of the training system is described below.

[0142] Step S201: The terminal initiates a training request to the server via the communication network.

[0143] The terminal initiates a training request to the server via the communication network, and the request includes information such as the terminal identification. The terminal identification is information that uniquely represents the identity of the terminal (for example, the terminal's ID or IP address, etc.).

[0144] This step S201 is described by taking one terminal initiating a training request as an example, but it is also possible for multiple terminals to initiate training requests in parallel. The processing process for multiple terminals is similar to that for one terminal and will not be repeated here.

[0145] Step S202: The server receives a training request.

[0146] exist Figure 6 The training system shown includes only one server, so the communication network can transmit the training request initiated by the terminal to the server. If the training system includes multiple servers, the training request can be transmitted to a relatively idle server based on the idle status of the server.

[0147] Step S203: The server responds to the received training request.

[0148] The server determines the terminal initiating the request based on the terminal identifier included in the received training request, and further determines the network model to be trained stored in the terminal. Alternatively, the server can determine the network model to be trained stored in the terminal initiating the request based on a comparison table of terminals and network models to be trained. Alternatively, the training request includes information about the network model to be trained, and the server can determine the network model to be trained based on this information. Determining the network model to be trained includes, but is not limited to, determining the network model's network architecture, hyperparameters, and other information that characterizes the network model.

[0149] After the server determines the network model to be trained, it can adopt the method of the first exemplary embodiment of the present invention to train the network model stored in the terminal that initiates the request using the same network model stored locally on the server. Specifically, the server updates the weights in the network model locally according to the method of steps S2010 to S2090 in the first exemplary embodiment, and transmits the updated weights to the terminal, so that the terminal synchronizes the network model to be trained stored in the terminal according to the received updated weights. Here, the network model in the server and the network model to be trained in the terminal can be the same network model, or the network model in the server can be more complex than the network model in the terminal, but the outputs of the two are close. The present disclosure does not limit the types of network models used for training in the server and the network models to be trained in the terminal, as long as the updated weights output from the server can synchronize the network model in the terminal, so that the output of the synchronized network model in the terminal is closer to the expected output.

[0150] exist Figure 6 In the training system shown, the terminal actively initiates the training request. Optionally, the second exemplary embodiment of the present invention is not limited to the server broadcasting the inquiry message and the terminal responding to the inquiry message to perform the above training process.

[0151] The training system described in the second exemplary embodiment of the present invention enables a server to perform online training on the network model within a terminal, improving training flexibility. This also significantly enhances the terminal's service processing capabilities and expands the terminal's service processing scenarios. The third exemplary embodiment above describes the training system using online training as an example, but the present invention is not limited to offline training processes and will not be further elaborated here.

[0152] <Third Exemplary Embodiment>

[0153] The third exemplary embodiment of the present invention describes a training device for a neural network model, which can execute the training method described in the first exemplary embodiment, and when the device is used in an online training system, it can be a device in the server described in the third exemplary embodiment. Figure 7 The software structure of the device is described in detail.

[0154] The training device in this third exemplary embodiment includes a pre-training unit 11, a quantization unit 12, and a retraining unit 13. The pre-training unit 11 is used to pre-train the neural network model, wherein the neural network model includes at least one quantization unit, wherein each quantization unit has multiple different quantization bit widths; the calculation unit 12 is used to calculate the sensitivity of the quantization unit, and adjust the optimal bit width of each quantization unit and the quantization parameter based on the calculated sensitivity, thereby generating a mixed precision neural network model; the retraining unit 13 is used to retrain the generated mixed precision neural network model

[0155] The training device of this embodiment also has a module for realizing the functions of the server in the training system, such as the function of identifying received data, the data encapsulation function, the network communication function, etc., which will not be described in detail here.

[0156] Other embodiments

[0157] Embodiments of the present invention may also be implemented by a computer of a system or device that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-described embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above-described embodiments, and by a method executed by a computer of the system or device, for example, by reading and executing computer-readable instructions from a storage medium to perform the functions of one or more of the above-described embodiments and / or controlling one or more circuits to perform the functions of one or more of the above-described embodiments. The computer may include one or more processors (e.g., central processing units (CPUs), microprocessing units (MPUs)), and may include independent computers or a network of independent processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer from, for example, a network or storage medium. The storage medium may include, for example, one or more of a hard disk, random access memory (RAM), read-only memory (ROM), storage of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray Disc (BD) (registered trademark)), a flash memory device, a memory card, and the like.

[0158] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0159] While the present invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all modifications, equivalent structures and functions.

Claims

1. A method for training a neural network model, the method comprising: A pre-training step of pre-training the neural network model so that the neural network model includes at least one quantization unit, wherein each quantization unit has a plurality of different quantization bit widths; a calculation step of calculating the sensitivity of the quantization unit, and updating the quantization bit width of each quantization unit and the quantization parameter based on the calculated sensitivity, thereby generating a mixed-precision neural network model, wherein the sensitivity represents the degree of influence of the quantization bit width of the quantization unit on the network output; Retraining step, retrain the generated mixed precision neural network model.

2. The method according to claim 1, wherein Each of the quantization units includes a filter parameter quantization unit and a feature map quantization unit.

3. The method according to claim 1, wherein the calculating step comprises: Measurement step, calculating the sensitivity of each quantization unit; An updating step is performed to reduce the bit width of a quantization unit with low sensitivity in the neural network model or to increase the bit width of a quantization unit with high sensitivity in the neural network model.

4. The method according to claim 3, wherein: When the neural network model updated in the updating step does not meet the predetermined conditions, the output of this step is input into the measurement step, and the measurement step and the updating step are repeated until the predetermined conditions are met.

5. The method according to claim 4, wherein the bit widths of the filter parameter quantization unit and the feature map quantization unit can be obtained in the first measurement step and the update step, or can be obtained according to a random algorithm or a pre-set method in multiple cycles of the measurement step and the update step.

6. The method according to claim 1, in, In the pre-training step, each quantization unit selects a bit width to quantize the input during forward propagation. The bit width selection method includes a random algorithm or a pre-set method.

7. The method according to claim 3, wherein: In the updating step, the quantization unit may have a maximum bit width or a minimum bit width.

8. The method according to claim 3, wherein the sensitivity measurement comprises: Single network pruning, gradient signal preservation, joint flow pruning, Fisher information, batch normalization scale factor, L2 norm, Jacobian determinant. The method according to claim 3 , wherein the sensitivities of the quantization units are ranked.

10. The method according to claim 3, wherein: In the updating step, the current bit width is reduced to an adjacent smaller bit width or the current bit width is increased to an adjacent larger bit width. 11 . The method according to claim 3 , wherein the low-sensitivity quantization unit or the high-sensitivity quantization unit can be obtained by a method including a sorting algorithm and an integer programming algorithm.

12. The method according to claim 11, wherein the constraints of the integer programming algorithm include global target constraints and current search phase constraints, wherein the current search phase constraints can be obtained based on the global target constraints and the current search times.

13. The method according to claim 4, wherein the predetermined conditions include the number of floating-point operations, total computational consumption, total memory consumption, hardware constraints, and training cost of the current quantized neural network model.

14. The method according to claim 3, wherein: The sensitivity can be normalized by predetermined indicators, including the number of floating-point operations, the number of multiplication and addition operations, the total memory consumption, the total computation consumption, etc.

15. A training device for a neural network model, characterized in that: The training device comprises: a pre-training unit for pre-training the neural network model, wherein the neural network model comprises at least one quantization unit, wherein each quantization unit comprises a plurality of different quantization bit widths; a calculation unit, configured to calculate the sensitivity of the quantization unit, and update the optimal bit width of each quantization unit and the quantization parameter based on the calculated sensitivity, thereby generating a mixed-precision neural network model, wherein the sensitivity represents the degree of influence of the quantization bit width of the quantization unit on the network output; The retraining unit is used to retrain the generated mixed precision neural network model.

16. A method for applying a neural network model, characterized in that: The application method comprises: Storing a neural network model trained based on the training method according to any one of claims 1 to 14; Receive a data set corresponding to the task requirements that can be performed by the stored neural network model; The data set is operated in each layer from top to bottom in the stored neural network model, and the results are output.

17. An application device of a neural network model, characterized in that: The application device comprises: A storage module configured to store a neural network model trained based on the training method according to any one of claims 1 to 14; A receiving module configured to receive a data set corresponding to a task requirement that can be performed by a stored neural network model; A processing module is configured to operate the data set in each layer from top to bottom in the stored neural network model and output the result.

18. A non-transitory computer-readable storage medium storing instructions, wherein when the instructions are executed by a computer, the computer is caused to perform the training method of the neural network model according to any one of claims 1 to 14.

19. A computer program product comprising a program, wherein the program is configured to cause a computer to execute the neural network model training method according to any one of claims 1 to 14.

Images