Model training methods, object processing methods and devices, electronic devices, media

Abstract

This disclosure relates to a model training method, object processing method and apparatus, electronic device, and storage medium, belonging to the field of computer technology. The model training method includes: acquiring a machine learning model, which is trained using floating-point precision; performing cross-layer weight regularization on the weights of the machine learning model to obtain regularized weights; determining quantization information corresponding to the inference engine, and determining multiple training parameters based on the quantization information; and performing quantization-aware training on the regularized weights and feature maps corresponding to the machine learning model based on the multiple training parameters to obtain a trained machine learning model. The technical solution of this disclosure introduces progressive quantization training during the quantization-aware training process, which can improve the accuracy of the quantized training model and obtain a more accurate quantized inference model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a model training method, a model training apparatus, an object processing method, an object processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] To reduce the amount of data used by neural networks, machine learning models can be trained using quantization.

[0003] In related technologies, machine learning models can be quantized offline to quantize the weights. However, this approach, which only considers weight quantization, has certain limitations. Furthermore, quantizing only the weights leads to significant losses during the quantization training process, resulting in lower model accuracy, lower computer performance, and higher computational resource consumption.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a model training method, object processing method and apparatus, electronic device, and storage medium, thereby overcoming, at least to some extent, the problem of low model accuracy caused by the limitations and defects of related technologies.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to one aspect of this disclosure, a model training method is provided, comprising: acquiring a machine learning model, the machine learning model being trained using floating-point precision; performing cross-layer weight regularization on the weights of the machine learning model to obtain regularized weights; determining quantization information corresponding to an inference engine, and determining multiple training parameters based on the quantization information; and performing quantization-aware training on the regularized weights and feature maps corresponding to the machine learning model based on the multiple training parameters to obtain a trained machine learning model.

[0008] According to one aspect of this disclosure, an object processing method is provided, comprising: obtaining an object to be processed; performing a convolution operation on the object to be processed using a trained machine learning model to obtain a prediction result corresponding to the object to be processed; wherein the trained machine learning model is trained according to any of the model training methods described above.

[0009] According to one aspect of this disclosure, a model training apparatus is provided, comprising: a model acquisition module for acquiring a machine learning model, the machine learning model being trained using floating-point precision; a weight processing module for performing cross-layer weight regularization processing on the weights of the machine learning model to obtain regularized weights; a quantization information determination module for determining quantization information corresponding to an inference engine and determining multiple training parameters based on the quantization information; and a quantization training module for performing quantization-aware training on the regularized weights and feature maps corresponding to the machine learning model based on the multiple training parameters to obtain a trained machine learning model.

[0010] According to one aspect of this disclosure, an object processing apparatus is provided, comprising: an object acquisition module for acquiring an object to be processed; and a prediction module for performing a convolution operation on the object to be processed using a trained machine learning model to obtain a prediction result corresponding to the object to be processed; wherein the machine learning model is trained according to any of the model training methods described above.

[0011] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the model training method or the object processing method described in any one of the preceding claims by executing the executable instructions.

[0012] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the model training method or the object processing method described in any one of the preceding claims.

[0013] The model training method, model training device, object processing method, object processing device, electronic device, and computer-readable storage medium provided in this disclosure, on the one hand, avoid the limitations of only performing weight quantization by performing quantization-aware training on the regularized weights and feature maps corresponding to the machine learning model separately. This allows for model training from two dimensions—weight quantization and feature map quantization—improving comprehensiveness and accuracy. On the other hand, quantizing the machine learning model from multiple dimensions such as weights and feature maps based on multiple training parameters enables progressive quantization training of the machine learning model. This introduces quantization loss into the quantization training process in stages, avoiding excessive loss caused by quantization during training and preventing its impact on the network convergence direction. This improves the accuracy and reliability of model quantization training, thus ensuring accurate quantization training. Furthermore, because progressive quantization training is possible, quantization-aware training of the model can be implemented in low-power devices or heterogeneous platforms, reducing the amount of data required by the computer during model training. This improves the computer's processing speed and computing performance while reducing power consumption and computational resources.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] Figure 1 A schematic diagram of a system architecture for which the model training method or object processing method of the present disclosure embodiments can be applied is shown.

[0017] Figure 2 The diagram illustrates a model training method according to an embodiment of the present disclosure.

[0018] Figure 3 The schematic diagram illustrates the principle of cross-layer weight regularization in the embodiments of this disclosure.

[0019] Figure 4 The schematic diagram illustrates the process of progressive quantization training in an embodiment of this disclosure.

[0020] Figure 5 The schematic diagram illustrates the overall process of quantitative training in an embodiment of this disclosure.

[0021] Figure 6 The schematic diagram illustrates the process of quantitative perception training in an embodiment of this disclosure.

[0022] Figure 7 The schematic diagram illustrates the process of performing pseudo-quantization operations on a convolutional neural network model in an embodiment of this disclosure.

[0023] Figure 8 The schematic diagram illustrates a flowchart of an object processing method according to an embodiment of the present disclosure.

[0024] Figure 9 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 10 A block diagram of an object processing apparatus in an embodiment of the present disclosure is shown schematically.

[0026] Figure 11 A block diagram of an electronic device according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0028] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0029] The relevant technologies mainly include the following solutions: offline quantization of network weights in logarithmic space, with compensatory quantization to reduce quantization loss. Alternatively, the model training device can perform at least one round of iterative training on the machine learning model, followed by testing in a simulated hardware deployment environment. This method is used for model testing but does not address the issue of large quantization training errors during the quantization training process. Another approach is to calculate the quantization parameters corresponding to the operators to be quantized in the machine learning model based on the binary approximation method; then quantize the operators based on these parameters. However, for more complex machine learning models, this approach suffers from high computational complexity and slow execution. Furthermore, it only provides offline quantization and does not consider feature map quantization, resulting in a small quantization range and large quantization errors.

[0030] To address the aforementioned technical issues, this disclosure provides a model training method that can be applied to various application scenarios such as information extraction, face recognition, and image classification.

[0031] Figure 1 A schematic diagram of a system architecture for a model training method and apparatus or an object processing method and apparatus that can be applied to embodiments of the present disclosure is shown.

[0032] like Figure 1 As shown, the system architecture 100 may include a client 101, a network 102, and a server 103. The client can be a mobile device, such as a smartphone, computer, tablet, or smart speaker. The network 102 serves as the medium for providing a communication link between the client 101 and the server 103. The network 102 may include various connection types, such as wired communication links, wireless communication links, etc. In this embodiment, the network 102 between the client 101 and the server 103 can be a wired communication link, for example, provided through a serial port cable, or a wireless communication link, provided through a wireless network. The server 103 can be a server with computing capabilities or a client, such as a portable computer, desktop computer, or smartphone, used to process data or models sent by the client.

[0033] This model training method can be applied to training machine learning models. (Reference) Figure 1As shown, client 101 can send a machine learning model to server 103 via network 102. Server 103 performs quantization training on the received machine learning model to obtain a trained machine learning model. Further, the client sends an object to be processed to the server. The server calls the trained machine learning model to perform inference on the object to be processed to obtain a prediction result corresponding to the object. The prediction result can then be sent back to client 101 for display and other processing operations. The server can be the same as the client, meaning both the client and server are smart devices, such as smartphones. The server can also be different from the client; no special limitation is made here.

[0034] It should be noted that the model training method and object processing method provided in this embodiment can be executed entirely by the server or the client. Accordingly, the model training device and the object processing device can be located in the server, or they can be executed entirely by the client.

[0035] Next, refer to Figure 2 The model training method in the embodiments of this disclosure will be described in detail.

[0036] In step S210, a machine learning model is obtained, which is trained using floating-point precision.

[0037] In this embodiment of the disclosure, the machine learning model can be any type of machine learning model, such as a convolutional neural network model. When the machine learning model is a convolutional neural network model, the model may include a multi-layer structure such as convolutional layers, pooling layers, fully connected layers, and activation layers. Specifically, the machine learning model can be a model trained using floating-point precision. Training using floating-point precision refers to a training process without adjusting the data type and without any constraints.

[0038] When training a model using floating-point precision, the model can be trained based on the training data and its corresponding labels. The model parameters can be adjusted until convergence, thus obtaining the machine learning model. In this embodiment, the machine learning model can be a pre-trained model or a real-time trained model; no special limitation is made here. The training data can be a large amount of image data. The labels of the training data can be the type of image data. The machine learning model can be applied to any scenario, such as face recognition, image classification, information extraction, etc.

[0039] In step S220, the weights of the machine learning model are subjected to cross-layer weight regularization to obtain the regularized weights.

[0040] In this embodiment, cross-layer weight regularization refers to regularizing the weights of different layers in a machine learning model to make the weight distribution of multiple different channels more concentrated, thereby facilitating better quantization-based perceptual training. Specifically, the cross-layer weight regularization process can be as follows: multiply the target channel of the weight of the current layer in the machine learning model by a first parameter, and multiply the target channel of the weight of the next layer connected to the current layer by a second parameter to perform cross-layer weight regularization processing on the weights; wherein the product of the first parameter and the second parameter is 1. When the machine learning model is a convolutional neural network model, the current layer can be any layer in the convolutional neural network model, such as a convolutional layer or a fully connected layer, etc. After performing a convolution operation on an RGB image, more channels can be generated depending on the number of filters. One convolution kernel yields one channel, so the number of output channels is equal to the number of convolution kernels. The target channel can be any one of the multiple channels corresponding to the convolution, for example, the i-th channel. Quantization requires scaling the parameter tensor. If the scaling factors for different channels differ significantly, it will lead to large quantization errors. Therefore, the scaling characteristics of activation functions can be used to adjust the scaling factors for different channels between layers. To optimize this, a first parameter can be determined, which can be the reciprocal of the scaling factor, for example, 1 / Si. Based on this, the target channel weights of the first layer can be multiplied by the first parameter. If the variance of the first layer weights in the output channel range is large, layers with large ranges can be multiplied by smaller coefficients, and layers with small ranges by larger coefficients, making the ranges of each channel as consistent as possible. This equals a given range, minimizing the overall tensor quantization error. To ensure internal numerical equivalence, the coefficients multiplied in the first layer must be divided back in the second layer. Specifically, if the i-th output channel of the current layer's weights is multiplied by a coefficient, the i-th input channel of the corresponding weight in the next layer must be divided by the same coefficient. Based on this, the target channel weights of the next layer connected to the current layer are multiplied by a second parameter. The second parameter can be a scaling factor, for example, Si. The layer connected to the current layer refers to the layer immediately following the current layer. For example, if the current layer is a convolutional layer, the next layer can be a fully connected layer. The target channel of the next layer can be the same as the target channel of the current layer; for example, both can be the i-th channel.

[0041] Figure 3 The diagram illustrates the principle of cross-layer weight regularization. (See reference...) Figure 3As shown, the i-th channel of the weight W(1) in the current layer is multiplied by 1 / Si and processed by the function f(.) to obtain the output y(1). Then, the i-th channel of the weight W(2) in the next layer is multiplied by Si and processed by the function f(.) to obtain the output y(2). These two operations can cancel each other out, so they do not affect the output of y(2). Therefore, after cross-layer weight regularization, the range of different channels can be adjusted without changing the final output distribution, thereby making the weight distribution of the convolutional neural network model more balanced and reducing quantization loss.

[0042] Continue to refer Figure 2 As shown, in step S230, the quantization information corresponding to the inference engine is determined, and multiple training parameters are determined based on the quantization information.

[0043] In this embodiment, the inference engine is used to solve the problem of running machine learning models for inference on the edge. For example, it can be used to optimize and transform machine learning models and perform inference calculations on data. Quantization information may include, but is not limited to, quantization granularity, data type, and data range. Quantization granularity can include layer-by-layer quantization and channel-by-channel quantization. Layer-by-layer quantization refers to specifying a quantizer (defined by scale and zeros) to quantize the tensor. Channel-by-channel quantization uses a different quantizer for each convolutional kernel. Generally, weight quantization can be channel-by-channel quantization, while activation output quantization, i.e., feature map quantization, can be layer-by-layer quantization.

[0044] Based on this, the quantization information, i.e., the quantization scheme, corresponding to the inference engine can be determined according to the quantization inference hardware platform to be deployed. The data type is Uint8 / int8, and the quantization granularity is per channel or per layer, i.e., Per channel / Per layer, with per channel quantization having a finer granularity. The specific quantization granularity is determined based on the types supported by the inference engine. For Uint8 Asymmetric (asymmetric unsigned 8-bit integer data), the quantized data range is 0–255; for Int8 Symmetric (symmetric signed 8-bit integer), the quantized data range is -127 to 127.

[0045] After determining the quantization information, multiple training parameters can be determined based on it. Furthermore, the quantization type can be determined based on the data type of the quantization information. Here, training parameters refer to the parameters used for quantization-aware training, which can be determined according to the data type in the quantization information and the corresponding pseudo-quantization nodes. Multiple training parameters can be used to progressively quantize the machine learning model. Training parameters represent the quantization level during quantization training; different training parameters correspond to different quantization levels.

[0046] Continue to refer Figure 2 As shown, in step S240, based on the multiple training parameters, the regularized weights and feature maps corresponding to the machine learning model are subjected to quantization-perception training to obtain a trained machine learning model.

[0047] In this embodiment of the disclosure, when training the machine learning model, the regularized weights can be quantized, and the feature maps can also be quantized. Regularized weights refer to weights with a concentrated distribution range. Feature maps refer to each two-dimensional image used to compose the data in the convolutional layer. Specifically, pseudo-quantization nodes can be inserted into the machine learning model, and pseudo-quantization operations can be performed on the weights and feature maps of the machine learning model based on the training parameters determined by the pseudo-quantization nodes, for quantization-aware training. The pseudo-quantization nodes are used to perform pseudo-quantization operations.

[0048] In this embodiment, pseudo-quantization nodes can be inserted into the weights and feature maps. Furthermore, these pseudo-quantization nodes can be of various types. These different types of pseudo-quantization nodes can represent different levels of pseudo-quantization operations, with the quantization level gradually decreasing. The types of pseudo-quantization nodes can be determined based on the data type in the quantization information of the inference engine. For example, they may include, but are not limited to, 16-bit quantization nodes, 12-bit quantization nodes, 10-bit quantization nodes, 8-bit quantization nodes, 4-bit quantization nodes, 1-bit quantization nodes, etc. For instance, when the data type corresponding to the inference engine is int8, int8 quantization can be performed. Int8 quantization refers to mapping the original 32-bit stored data to 8-bit storage. The range of int8 is [-128, 127], and the range of uint8 is [0, 255]. When performing int8 quantization, the various types of pseudo-quantization nodes can be 16-bit quantization nodes, 12-bit quantization nodes, 10-bit quantization nodes, and 8-bit quantization nodes.

[0049] The progressive shrinkage method refers to using multiple types of pseudo-quantization nodes sequentially according to priority. Priority order refers to the order of quantization levels from high to low. Therefore, for the multiple types of pseudo-quantization nodes corresponding to int8 quantization, the 16-bit pseudo-quantization node has the highest priority, and the 8-bit pseudo-quantization node has the lowest priority.

[0050] Figure 4 The diagram illustrates a flowchart of pseudo-quantization operations using a progressive shrinkage method. (See reference...) Figure 4 As shown, the main steps include:

[0051] In step S410, a pseudo-quantization node of the current type is inserted into the machine learning model, and pseudo-quantization operation is performed on the machine learning model according to the pseudo-quantization node of the current type to obtain the current machine learning model.

[0052] In step S420, a pseudo-quantization node of the next type is inserted into the current machine learning model, and quantization-aware training is performed on the current machine learning model based on the pseudo-quantization node of the next type to obtain the next machine learning model;

[0053] In step S430, the remaining pseudo-quantization nodes are inserted into the next machine learning model until all pseudo-quantization nodes of multiple types are inserted into the next machine learning model, so as to obtain the trained machine learning model.

[0054] In this embodiment, the pseudo-quantization node of the current type refers to the highest priority pseudo-quantization node, such as a 16-bit pseudo-quantization node. During pseudo-quantization operations on the machine learning model using the current type of pseudo-quantization node, 16-bit pseudo-quantization nodes can be inserted into the weights and feature maps. Pseudo-quantization operations consist of quantization and dequantization. Quantization refers to converting a floating-point number to a fixed-point number, and dequantization refers to converting the quantized fixed-point number back to a floating-point number. After inserting the 16-bit pseudo-quantization node, quantization operations can be performed on the weights and feature maps based on the 16-bit pseudo-quantization node to perform quantization-aware training on the machine learning model, obtaining the current machine learning model.

[0055] Furthermore, the next type of pseudo-quantized node can be a pseudo-quantized node at the next lower level adjacent to the current type of pseudo-quantized node; that is, a node with a lower priority than the current type of pseudo-quantized node, such as a 12-bit pseudo-quantized node. A 12-bit pseudo-quantized node can be inserted into the current machine learning model. After inserting the 12-bit pseudo-quantized node, quantization operations can be performed on the weights and feature maps based on the 12-bit pseudo-quantized node to perform quantization-aware training on the current machine learning model, resulting in the next machine learning model, i.e., the machine learning model trained with the 12-bit pseudo-quantized node.

[0056] The remaining pseudo-quantized nodes can be any remaining nodes from various types of pseudo-quantized nodes, and they can be inserted into the machine learning model sequentially, one type of node at a time. When the data type in the quantized information is int8 but imt8 quantization is performed, the remaining pseudo-quantized nodes can be any nodes with a priority lower than the next type of pseudo-quantized node, such as 10-bit pseudo-quantized nodes and 8-bit pseudo-quantized nodes. Specifically, all nodes included in the remaining pseudo-quantized nodes can be inserted into the next machine learning model. For example, first insert 10-bit pseudo-quantized nodes, then insert 8-bit pseudo-quantized nodes after training, and so on, until all remaining pseudo-quantized nodes are inserted into the machine learning model. Based on this, after inserting 10-bit pseudo-quantized nodes, quantization operations can be performed on the weights and feature maps based on the 10-bit pseudo-quantized nodes to perform quantization-aware training on the next machine learning model, obtaining an intermediate machine learning model. Then, the 8-bit pseudo-quantized nodes from the remaining pseudo-quantized nodes can be inserted into the intermediate machine learning model for further quantization-aware training. Furthermore, the above steps can be repeated to continue inserting remaining pseudo-quantization nodes (e.g., 4-bit pseudo-quantization nodes and 1-bit pseudo-quantization nodes) into the intermediate machine learning model. The model is then trained using quantization awareness based on the remaining pseudo-quantization nodes until all pseudo-quantization nodes of various types are inserted into the machine learning model, thus ending the entire loop process and obtaining a trained machine learning model.

[0057] Figure 5 The diagram illustrates the overall flowchart for training the model. (See reference) Figure 5 As shown, the main steps include:

[0058] In step S501, a machine learning model is trained using floating-point precision.

[0059] In step S502, it is determined whether the machine learning model has converged. If so, proceed to step S503.

[0060] In step S503, cross-layer weight regularization is performed on the weights, i.e., weight CLE.

[0061] In step S504, quantization information is determined.

[0062] In step S505, a 16-bit pseudo-quantization node is added.

[0063] In step S506, the model is trained using 16-bit pseudo-quantized nodes.

[0064] In step S507, it is determined whether the machine learning model has converged. If so, proceed to step S508.

[0065] In step S508, a 12-bit pseudo-quantization node is added.

[0066] In step S509, the model is trained using 12-bit pseudo-quantized nodes.

[0067] In step S510, it is determined whether the machine learning model has converged. If so, proceed to step S511.

[0068] In step S511, a 10-bit pseudo-quantization node is added.

[0069] In step S512, the model is trained using 10-bit pseudo-quantized nodes.

[0070] In step S513, it is determined whether the machine learning model has converged. If so, proceed to step S514.

[0071] In step S514, an 8-bit pseudo-quantization node is added.

[0072] In step S515, the model is trained using 8-bit pseudo-quantized nodes.

[0073] In step S516, the device is deployed to the terminal device.

[0074] The technical solution in this disclosure, for a given quantization scheme, implements pseudo-quantization operations through multiple types of pseudo-quantization nodes. This progressively shrinking training method, from 16-bit to 8-bit, quantizes the network, avoiding the limitations of directly transforming from 16-bit to 8-bit without multiple types of pseudo-quantization operations. By gradually quantizing the network through multiple types of pseudo-quantization operations, the quantization loss is introduced into the quantization training process in stages, preventing excessive loss from quantization during training and reducing quantization errors from direct conversion without affecting the network's convergence direction. This gradually approaches the globally optimal weights during quantization inference, thereby improving the inference accuracy of the quantization model during deployment and enhancing the model's stability and accuracy.

[0075] When using pseudo-quantized nodes to train machine learning models with quantization awareness, this can be achieved through quantization and dequantization. During the quantization process—the conversion from floating-point to fixed-point numbers—quantization can be performed based on quantization parameters. These parameters ensure that all variables within the floating-point range are mapped seamlessly to the value range of the bits to be quantized. Quantization parameters can include scale and zeropoint. Scale and zeropoint are calculated by linearly mapping the minimum and maximum values ​​of the floating-point number (min and max) to 0 to 255 during quantization. The scale is determined by the ratio of the difference between the minimum and maximum values ​​to the data range. The zeropoint is determined by the ratio of the minimum value to the scale.

[0076] After determining the scale Q and the zero point RQM, the weights can be quantized based on the scale and zero point. The quantization can be performed by converting the original floating-point number used to represent the weights into a fixed-point number. For example, first calculate the product of the original floating-point number and the scale, then round the product using a rounding function to obtain the rounded result as an intermediate value. Finally, calculate the difference between the intermediate value and the zero point to perform the quantization. Specifically:

[0077] Def Quant(Vx,Q,ROM):return round(Q*Vx)-RQM

[0078] Furthermore, a floating-point number can be restored based on the quantization parameters. For example, a floating-point number can be returned based on the ratio of the sum of the fixed-point number and the zero point to the scale, in order to perform dequantization. Specifically:

[0079] Def QuantRevert(VxQuant,Q,ROM):return(VxQuant+RQM) / Q

[0080] In this embodiment of the disclosure, the weights and feature maps can be subjected to quantization-based perceptual training respectively. Figure 6 The flowchart for quantization sensing is illustrated in the diagram. (See reference) Figure 6 As shown, the main steps include:

[0081] In step S610, pseudo-quantization is performed on the weights according to multiple types of pseudo-quantization nodes, and convolution operation is performed on the input parameters and the quantized weights to obtain the convolution result.

[0082] In step S620, bias calculation is performed on the convolution result, activation operation is performed on the bias result to obtain activation result, and pseudo-quantization operation is performed on the feature map corresponding to the activation result to perform quantization-aware training.

[0083] refer to Figure 7 As shown in Figure A, the input parameters and weights of the convolutional layer Conv are both in uint8 format. After convolution, a uint32 convolution result is obtained. Then, the bias is calculated, and finally the uint32 result is quantized to uint8 and output after passing through the ReLU6 activation layer. Figure 7 Figure B in the diagram shows the corresponding quantization-aware training scheme. To address the technical issue of significant loss in this scheme, refer to... Figure 7As shown in Figure B, before performing the convolution operation, the weights are pseudo-quantized to simulate the quantization loss of the weights. Then, a convolution operation is performed on the input parameters and the quantized weights to obtain the convolution result. A bias calculation is performed on the convolution result to obtain the bias result. An activation operation is then performed on the bias result to obtain the activation result. Finally, pseudo-quantization is performed on the feature map corresponding to the activation result for quantization-aware training. It should be noted that this can be done according to... Figure 7 As shown in Figure B, 16-bit quantization, 12-bit quantization, 10-bit quantization, and 8-bit quantization are implemented sequentially, with only the quantization model differing. After performing 8-bit quantization on the weights and output, a trained machine learning model can be obtained.

[0084] It should be noted that for feature maps, the quantization training method is the same as for weight quantization. Specifically, after determining the scale Q and zero-point RQM, the feature map can be quantized based on the scale and zero-point. It can be quantized into fixed-point numbers based on the quantization parameters and the original floating-point quantization used to represent the feature map. For example, first calculate the product of the original floating-point number and the scale, then round the product using a rounding function to obtain the rounded result as an intermediate value, and then calculate the difference between the intermediate value and the zero-point for quantization. Furthermore, it can be restored to floating-point numbers based on the quantization parameters, for example, by returning a floating-point number from the ratio of the sum of the fixed-point number and the zero-point to the scale for dequantization.

[0085] Based on this, the entire process can be described as follows: train a machine learning model using floating-point precision based on image data, perform cross-layer weight regularization on the weights of the machine learning model, determine the quantization information of the inference engine to be inferred and multiple training parameters, add pseudo-quantization nodes to the machine learning model based on multiple training parameters, and progressively quantize the weights and activation results of the machine learning model based on pseudo-quantization nodes of different levels to perform quantization-aware training.

[0086] In this embodiment, both weight quantization and feature map quantization are considered simultaneously, enabling model quantization training from multiple dimensions. This avoids the limitations of quantization based solely on weights, improving accuracy and reliability. Furthermore, by gradually quantizing weights and feature maps through different types of pseudo-quantization nodes, the quantization loss is introduced into the quantization training process in stages. This avoids excessive loss caused by quantization during training, preventing it from affecting the network's convergence direction, thereby improving the accuracy and reliability of model training.

[0087] In addition, progressive training methods can also be used in areas such as neural network pruning. This approach can also be combined with neural network pruning and distillation methods for comprehensive model compression and optimization.

[0088] This disclosure also provides an object processing method, referencing... Figure 8 As shown, the main steps include:

[0089] In step S810, the object to be processed is obtained;

[0090] In step S820, the object to be processed is subjected to a convolution operation by the trained machine learning model to obtain the prediction result corresponding to the object to be processed; wherein, the machine learning model is trained according to any of the model training methods described above.

[0091] In this embodiment, the object to be processed can be determined according to the application scenario, such as an image to be processed or any type of text, data, corpus, etc. This embodiment uses an image to be processed as an example. In practical applications, the object to be processed is simply input into a trained machine learning model for convolution to extract its feature data, and then the feature data is used for prediction processing to obtain the prediction result corresponding to the object to be processed. The prediction result can be determined according to the actual application scenario, such as a face recognition result, image category, etc. For example, inputting the image to be processed into a trained machine learning model yields the face recognition result corresponding to the image to be processed.

[0092] In this embodiment, after quantization-aware training of the machine learning model, there is a buffering process compared to the direct quantization process from floating-point to fixed-point numbers, resulting in higher accuracy. Furthermore, this technical solution can be used with any machine learning model. It enables the model to run on low-power devices or heterogeneous platforms, reducing data volume and improving processing speed.

[0093] This disclosure provides a model training apparatus, with reference to... Figure 9 As shown, the model training device 900 may include:

[0094] The model acquisition module 901 is used to acquire a machine learning model, which is trained using floating-point precision.

[0095] The weight processing module 902 is used to perform cross-layer weight regularization on the weights of the machine learning model to obtain the regularized weights.

[0096] The quantization information determination module 903 is used to determine the quantization information corresponding to the inference engine and determine multiple training parameters based on the quantization information;

[0097] The quantization training module 904 is used to perform quantization-aware training on the regularized weights and feature maps corresponding to the machine learning model based on the multiple training parameters, so as to obtain a trained machine learning model.

[0098] In one exemplary embodiment of this disclosure, the weight processing module includes: a regularization module, configured to multiply the target channel of the weight of the current layer in the machine learning model by a first parameter, and multiply the target channel of the weight of the next layer connected to the current layer by a second parameter, so as to perform cross-layer weight regularization processing on the weight; wherein the product of the first parameter and the second parameter is 1.

[0099] In one exemplary embodiment of this disclosure, the quantization information determination module includes: a type determination module, used to determine the data type of the inference engine and determine the data range to be quantized based on the data type.

[0100] In one exemplary embodiment of this disclosure, the quantization training module includes: a pseudo-quantization operation module, configured to insert multiple types of pseudo-quantization nodes into the machine learning model, and perform pseudo-quantization operations on the machine learning model using a progressive shrinkage method based on the training parameters determined by the multiple types of pseudo-quantization nodes, so as to perform quantization-aware training to obtain a trained machine learning model.

[0101] In one exemplary embodiment of this disclosure, the pseudo-quantization operation module includes: a first operation module, configured to insert pseudo-quantization nodes of the current type into the machine learning model, and perform pseudo-quantization operation on the machine learning model according to the pseudo-quantization nodes of the current type to obtain a current machine learning model; a second operation module, configured to insert pseudo-quantization nodes of the next type into the current machine learning model, and perform quantization-aware training on the current machine learning model according to the pseudo-quantization nodes of the next type to obtain a next machine learning model; and a loop operation module, configured to insert the remaining pseudo-quantization nodes among all pseudo-quantization nodes for the next machine learning model, until all pseudo-quantization nodes of multiple types of pseudo-quantization nodes are inserted for the next machine learning model, to obtain a trained machine learning model.

[0102] In one exemplary embodiment of this disclosure, the pseudo-quantization operation module includes: a weight quantization module, used to perform pseudo-quantization operation on weights according to multiple types of pseudo-quantization nodes, and to perform convolution operation on input parameters and quantized weights to obtain a convolution result; and a feature map quantization module, used to perform bias calculation on the convolution result, perform activation operation on the bias result to obtain an activation result, and perform pseudo-quantization operation on the feature map corresponding to the activation result to perform quantization-aware training.

[0103] In one exemplary embodiment of this disclosure, the first calculation module includes: a quantization module, configured to perform quantization calculation on the weights according to quantization parameters to obtain quantized values; and an inverse quantization module, configured to perform a rounding operation on the quantized values ​​to obtain intermediate values, and perform inverse quantization calculation on the intermediate values ​​according to the quantization parameters.

[0104] It should be noted that the specific details of each module in the above-mentioned model training device have been described in detail in the corresponding model training methods, so they will not be repeated here.

[0105] This disclosure also provides an object processing apparatus, with reference to... Figure 10 As shown, the object processing device 1000 may include:

[0106] Object acquisition module 1001 is used to acquire objects to be processed;

[0107] The prediction module 1002 is used to perform a convolution operation on the object to be processed using a trained machine learning model to obtain a prediction result corresponding to the object to be processed; wherein the machine learning model is trained according to any of the model training methods described above.

[0108] Figure 11 A schematic diagram is shown that is suitable for implementing exemplary embodiments of the present disclosure. The terminal of the present disclosure can be configured as follows: Figure 11 The electronic device shown is in the form of [the device]; however, it should be noted that... Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0109] The electronic device disclosed herein includes at least a processor and a memory, the memory being used to store one or more programs, which, when executed by the processor, enable the processor to implement the methods of exemplary embodiments of this disclosure.

[0110] Specifically, such as Figure 11As shown, the electronic device 1100 may include: a processor 1110, an internal memory 1121, an external memory interface 1122, a Universal Serial Bus (USB) interface 1130, a charging management module 1140, a power management module 1141, a battery 1142, an antenna 1, an antenna 2, a mobile communication module 1150, a wireless communication module 1160, an audio module 1170, a speaker 1171, a receiver 1172, a microphone 1173, a headphone jack 1174, a sensor module 1180, a display screen 1190, a camera module 1191, an indicator 1192, a motor 1193, buttons 1194, and a Subscriber Identification Module (SIM) card interface 1195, etc. The sensor module 1180 may include depth sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, and bone conduction sensors, etc.

[0111] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1100. In other embodiments of this application, the electronic device 1100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0112] Processor 1110 may include one or more processing units, such as an application processor, modem processor, graphics processor, image signal processor, controller, video codec, digital signal processor, baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. Additionally, processor 1110 may include memory for storing instructions and data. The image processing method in this exemplary embodiment can be executed by an application processor, graphics processor, or image signal processor; when the method involves neural network-related processing, it can be executed by the NPU.

[0113] Internal memory 1121 can be used to store computer executable program code, which includes instructions. Internal memory 1121 may include a program storage area and a data storage area. External memory interface 1122 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of electronic device 1100.

[0114] The communication function of mobile terminal 1100 can be implemented through a mobile communication module, antenna 1, a wireless communication module, antenna 2, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1100. The wireless communication module can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 200.

[0115] The display screen is used to implement display functions, such as displaying the user interface, images, and videos. The camera module is used to implement shooting functions, such as capturing images and videos. The audio module is used to implement audio functions, such as playing audio and capturing voice. The power module is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status.

[0116] This application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0117] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0118] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0119] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0120] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0121] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0122] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0123] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims. It should be understood that this disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that, include: Obtain a machine learning model, which is trained using floating-point precision; The weights of the machine learning model are subjected to cross-layer weight regularization to obtain the regularized weights. Determine the quantization information corresponding to the inference engine, and determine multiple training parameters based on the quantization information; Based on the multiple training parameters, quantization-aware training is performed on the regularized weights and feature maps corresponding to the machine learning model to obtain a trained machine learning model. This includes inserting multiple types of pseudo-quantization nodes into the machine learning model, and performing pseudo-quantization operations on the machine learning model using a progressive shrinkage method according to the training parameters determined by the multiple types of pseudo-quantization nodes to perform quantization-aware training and obtain a trained machine learning model. The step of inserting multiple types of pseudo-quantization nodes into the machine learning model and performing pseudo-quantization operations on the machine learning model using a progressive shrinkage method based on the training parameters determined by the multiple types of pseudo-quantization nodes includes: Insert pseudo-quantization nodes of the current type into the machine learning model, and perform pseudo-quantization operations on the machine learning model based on the pseudo-quantization nodes of the current type to obtain the current machine learning model; Insert the next type of pseudo-quantization node into the current machine learning model, and perform quantization-aware training on the current machine learning model based on the next type of pseudo-quantization node to obtain the next machine learning model; Insert the remaining pseudo-quantized nodes from all pseudo-quantized nodes into the next machine learning model, until all pseudo-quantized nodes of multiple types are inserted into the next machine learning model, to obtain the trained machine learning model.

2. The model training method according to claim 1, characterized in that, The cross-layer weight regularization processing of the weights of the machine learning model includes: The target channel of the weights of the current layer in the machine learning model is multiplied by a first parameter, and the target channel of the weights of the next layer connected to the current layer is multiplied by a second parameter to perform cross-layer weight regularization on the weights; wherein the product of the first parameter and the second parameter is 1.

3. The model training method according to claim 1, characterized in that, The quantization information corresponding to the determined inference engine includes: Determine the data type of the inference engine, and determine the range of data to be quantized based on the data type.

4. The model training method according to claim 1, characterized in that, The step of performing pseudo-quantization operations on the machine learning model using a progressive shrinking method based on the training parameters determined according to the multiple types of pseudo-quantization nodes, to obtain a trained machine learning model through quantization-aware training, includes: The weights are pseudo-quantized using multiple types of pseudo-quantization nodes, and then convolution is performed on the input parameters and the quantized weights to obtain the convolution result. The convolution result is biased, the biased result is activated to obtain the activation result, and the feature map corresponding to the activation result is pseudo-quantized to perform quantization-aware training.

5. The model training method according to claim 4, characterized in that, The step of performing pseudo-quantization operations on weights based on multiple types of pseudo-quantization nodes includes: The weights are quantized according to the quantization parameters to obtain quantized values. The quantized value is rounded to obtain an intermediate value, and then the intermediate value is dequantized according to the quantization parameters.

6. An object processing method, characterized in that, include: Get the object to be processed; The object to be processed is subjected to a convolution operation by a trained machine learning model to obtain the prediction result corresponding to the object to be processed; wherein the machine learning model is trained by the model training method according to any one of claims 1-5.

7. A model training device, characterized in that, include: The model acquisition module is used to acquire a machine learning model, which is trained using floating-point precision. The weight processing module is used to perform cross-layer weight regularization on the weights of the machine learning model to obtain the regularized weights. The quantization information determination module is used to determine the quantization information corresponding to the inference engine and to determine multiple training parameters based on the quantization information. The quantization training module is used to perform quantization-aware training on the regularized weights and feature maps corresponding to the machine learning model based on the multiple training parameters to obtain a trained machine learning model. The module includes: inserting multiple types of pseudo-quantization nodes into the machine learning model, and performing pseudo-quantization operations on the machine learning model using a progressive shrinkage method according to the training parameters determined by the multiple types of pseudo-quantization nodes to perform quantization-aware training and obtain a trained machine learning model. The step of inserting multiple types of pseudo-quantization nodes into the machine learning model and performing pseudo-quantization operations on the machine learning model using a progressive shrinkage method based on the training parameters determined by the multiple types of pseudo-quantization nodes includes: Insert pseudo-quantization nodes of the current type into the machine learning model, and perform pseudo-quantization operations on the machine learning model based on the pseudo-quantization nodes of the current type to obtain the current machine learning model; Insert the next type of pseudo-quantization node into the current machine learning model, and perform quantization-aware training on the current machine learning model based on the next type of pseudo-quantization node to obtain the next machine learning model; Insert the remaining pseudo-quantized nodes from all pseudo-quantized nodes into the next machine learning model, until all pseudo-quantized nodes of multiple types are inserted into the next machine learning model, to obtain the trained machine learning model.

8. An object processing apparatus, characterized in that, include: The object retrieval module is used to retrieve the object to be processed. The prediction module is used to perform a convolution operation on the object to be processed using a trained machine learning model to obtain the prediction result corresponding to the object to be processed; wherein the machine learning model is trained by the model training method according to any one of claims 1-5.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the model training method of any one of claims 1-5 or the object processing method of claim 6 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1-5 or the object processing method according to claim 6.

Images