A model compression method, device and readable storage medium

Abstract

Embodiments of the present application provide a model compression method, device and readable storage medium. The method comprises: obtaining a teacher network model and a student network model, the teacher network model and the student network model being the same structure of floating point models that have been trained; segmenting the teacher network model and the student network model in the same way; training each segment of the student network model in turn; fixing the network parameters of each segment that has been trained before a current segment of the student network model during training of the current segment, quantizing the current segment, and performing knowledge distillation on the current segment using the teacher network model and unlabeled training data; and obtaining a target network model when all segments of the student network model have been trained. Embodiments of the present application can protect the privacy and security of user data while ensuring the accuracy of the compressed model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a model compression method, apparatus, and readable storage medium. Background Technology

[0002] With the development of deep learning technology, deep neural network models are widely used in various application scenarios, such as image processing, speech recognition, reasoning / prediction, knowledge representation, and operation control.

[0003] To improve the performance of deep neural network models, the number of parameters and computational cost have increased dramatically, posing significant challenges to model training and deployment. Particularly in deployment, the sheer volume of parameters and computational demands makes it difficult to deploy deep neural network models on resource-constrained hardware devices such as mobile devices.

[0004] To reduce the hardware consumption of deep neural network models and enable their deployment on resource-constrained devices, model compression techniques are commonly used to improve operational efficiency and reduce computational and storage resource usage. Examples of model compression techniques include knowledge distillation and model quantization. During model compression, labeled training data is typically used to continuously train the model to be compressed, ensuring the accuracy of the compressed model. However, with increasing emphasis on user data privacy, labeled training data is becoming increasingly difficult to obtain. Summary of the Invention

[0005] This invention provides a model compression method, apparatus, and readable storage medium that can protect the privacy and security of user data while ensuring the accuracy of the compressed model.

[0006] In a first aspect, embodiments of the present invention disclose a model compression method, the method comprising:

[0007] Obtain a teacher network model and a student network model, wherein the teacher network model and the student network model are pre-trained floating-point models with the same structure;

[0008] The teacher network model and the student network model are segmented in the same way;

[0009] Each segment of the student network model is trained sequentially; when training the current segment of the student network model, the network parameters of each segment that has been trained before the current segment are fixed, the current segment is quantized, and knowledge distillation is performed on the current segment using the teacher network model and unlabeled training data;

[0010] The target network model is obtained when all segments of the student network model have been trained.

[0011] Secondly, embodiments of the present invention disclose a model compression apparatus, the apparatus comprising:

[0012] The model acquisition module is used to acquire the teacher network model and the student network model, wherein the teacher network model and the student network model are pre-trained floating-point models with the same structure.

[0013] The model segmentation module is used to segment the teacher network model and the student network model in the same way;

[0014] The compressed training module is used to train each segment of the student network model sequentially. When training the current segment of the student network model, the network parameters of the segments that have been trained before the current segment are fixed, the current segment is quantized, and knowledge distillation is performed on the current segment using the teacher network model and unlabeled training data. When all segments of the student network model have been trained, the target network model is obtained.

[0015] Thirdly, embodiments of the present invention disclose a machine-readable medium having instructions stored thereon that, when executed by one or more processors of a device, cause the device to perform the model compression method as described above.

[0016] The embodiments of the present invention have the following advantages:

[0017] The model compression method provided in this invention combines model quantization and knowledge distillation, and employs a segmented, progressive approach to quantize and distill the student network model segment by segment, ultimately obtaining a target network model. This target network model is obtained by segmenting the student network model under the guidance of the teacher network model, resulting in high accuracy and a lightweight design, allowing deployment on resource-constrained hardware devices (such as mobile devices). Furthermore, this invention utilizes unlabeled training data during the compression process of the student network model, independent of the original labeled training data, thus protecting the privacy and security of user data. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart illustrating the steps of an embodiment of the model compression method of the present invention;

[0020] Figure 2 This is a schematic diagram of training the (k-1)th segment of a student network model in an example of the present invention;

[0021] Figure 3 This is a structural block diagram of an embodiment of a model compression device according to the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0024] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a model compression method according to the present invention. The method may include the following steps:

[0025] Step 101: Obtain the teacher network model and the student network model, wherein the teacher network model and the student network model are pre-trained floating-point models with the same structure;

[0026] Step 102: Segment the teacher network model and the student network model in the same way;

[0027] Step 103: Train each segment of the student network model in sequence; when training the current segment of the student network model, fix the network parameters of each segment that has been trained before the current segment, quantize the current segment, and use the teacher network model and unlabeled training data to perform knowledge distillation on the current segment;

[0028] Step 104: When all segments of the student network model have been trained, the target network model is obtained.

[0029] The model compression method provided in this invention combines model quantization and knowledge distillation, and employs a segmented, progressive approach to quantize and distill the student network model segment by segment, ultimately obtaining a target network model. This target network model is obtained by compressing the student network model under the guidance of the teacher network model, exhibiting high accuracy and a lightweight design, allowing deployment on resource-constrained hardware devices (such as mobile devices). Furthermore, this invention utilizes unlabeled training data during the compression process of the student network model, independent of the original labeled training data, thus protecting the privacy and security of user data.

[0030] In this embodiment of the invention, the student network model is a model to be compressed, and the teacher network model is used to perform knowledge distillation on the student network model during the compression process, so as to reduce the accuracy loss of the student network model during the compression process.

[0031] It should be noted that the application scenarios of the teacher network model, the student network model, and the target network model are not limited in the embodiments of the present invention. The teacher network model, the student network model, and the target network model can be used to perform the same target task, which includes, but is not limited to, face recognition, image classification, object detection, semantic segmentation, speech recognition, machine translation, natural language processing, and recommendation systems.

[0032] The teacher network model and the student network model are deep neural network models with the same structure. Furthermore, the teacher network model and the student network model can be the same model.

[0033] This invention does not limit the model structure of the teacher network model and the student network model. For example, the model structure can include convolutional neural networks of various sizes, such as one or more of 3×3 convolutions, 5×5 convolutions, dilated convolutions, and grouped convolutions; the model structure can also include batch normalization layers; the activation function of the model structure can be any one of ReLU, Swish, and sigmoid; the model structure can also include residual structures, such as the residual structure of ResNet or the inverse residual structure of Mobilenent; the final output layer of the model structure can be a global pooling layer or a fully connected layer.

[0034] The teacher network model and student network model obtained in step 101 are pre-trained floating-point models with the same structure. A floating-point model refers to a model where the weights and activations of each layer are floating-point numbers. The teacher network model and student network model can be high-precision floating-point models trained using labeled training data. In specific implementations, for different application scenarios, training data required for the corresponding target task can be collected to obtain a training set. For example, for image classification applications, a large number of images can be collected and manually labeled to obtain a training set. The teacher network model and student network model trained using this training set can perform the target task of image classification. Similarly, for speech recognition applications, a large number of recordings can be collected and manually labeled to obtain a training set. The teacher network model and student network model trained using this training set can perform the target task of speech recognition. The collected training set is used to train the teacher network model until the model converges, resulting in a fully trained teacher network model.

[0035] In practical applications, the following scenario exists: Party B possesses a pre-trained floating-point model (denoted as Model A), which was trained by Party A using its private labeled training data. Party B needs to compress Model A so that it can be deployed on resource-constrained hardware devices (such as mobile devices). However, Party B does not have the private labeled training data used to train Model A. In this case, Party B can use Model A as the teacher network model and a copy of Model A (denoted as Model A') as the student network model. By executing steps 101 to 104 above, the student network model A' is compressed to obtain the target network model. During this model compression process, this embodiment of the invention utilizes a high-precision floating-point model (teacher network model) to guide the quantization process of the student network model in segments, without relying on the original labeled training data of Model A.

[0036] After obtaining the student network model to be compressed and the teacher network model used to guide the student network model, this embodiment of the invention segments the teacher network model and the student network model in the same way; each segment of the student network model is trained sequentially; when training the current segment of the student network model, the network parameters of the previously trained segments are fixed, the current segment is quantized, and knowledge distillation is performed on the current segment using the teacher network model and unlabeled training data; when all segments of the student network model have been trained, the target network model is obtained.

[0037] In an optional embodiment of the present invention, the method may further include: during the training of the current segment of the student network model, for input unlabeled training data, the current segment of the student network model and each segment trained before the current segment are inferred based on fixed-point parameters, and each segment after the current segment of the student network model is inferred based on floating-point parameters.

[0038] In this embodiment of the invention, training the current segment of the student network model can include the following two stages: Stage 1, quantizing the current segment; Stage 2, after quantizing the current segment, to reduce the accuracy loss caused by quantization, knowledge distillation is performed on the quantized current segment using the teacher network model and unlabeled training data. When training the current segment of the student network model, the network parameters of each segment trained before the current segment are fixed, only the current segment is quantized, and the subsequent segments retain floating-point parameters. Knowledge distillation is then performed on the current segment using the teacher network model and unlabeled training data.

[0039] For example, the first segment of the student network model is trained first; this first segment is the current segment. The first segment is quantized to obtain its quantized parameters, while the second to the last segment retain floating-point parameters. Then, the teacher network model and unlabeled training data are used to perform knowledge distillation on the first segment, optimizing its network parameters, including the quantized parameters. Each segment of the student network model can undergo multiple iterations of training. When the iteration stopping condition is met, the training of that segment is considered complete, and the next segment can be trained.

[0040] Next, the second segment of the student network model is trained; this second segment is the current segment. The network parameters of the first segment are fixed, and only the second segment is quantized. The third to the last segment retain floating-point parameters. Knowledge distillation is performed on the second segment using the teacher network model and unlabeled training data to optimize its network parameters, including the quantization parameters. This process is repeated until the last segment of the student network model is trained, resulting in the target network model. This target network model can also perform the target task.

[0041] Quantization refers to the process of approximating floating-point activations or weights (usually represented as 32-bit floating-point numbers) to low-bit integers (such as 16-bit or 8-bit), thereby completing calculations in low-bit representation. Generally, quantization can compress model parameters, thereby reducing model storage overhead; and by reducing memory access and effectively utilizing low-bit computation instructions, inference speed can be improved. This embodiment of the invention performs progressive segmented quantization on the student network model, and after quantizing the current segment, uses the teacher network model and unlabeled training data to perform knowledge distillation on the current segment to minimize the accuracy loss caused by quantization. The final target network model's weights and activations are represented as low-bit integers.

[0042] It should be noted that the embodiments of the present invention do not impose restrictions on the iteration stopping conditions for each segment of training the student network model. For example, the iteration stopping conditions may include: the total loss of the model is less than a preset threshold, or the number of iterations reaches a preset number.

[0043] In machine learning models, the difference between the predicted value and the true value for each sample is called the loss. The smaller the difference, the smaller the loss, and the better the learning model. The loss function is a non-negative real-valued function used to calculate the loss. This invention employs a piecewise, progressive approach, quantizing the student network model in segments and using the teacher network model and unlabeled training data to perform knowledge distillation on the currently quantized segment. For the current segment of the student network model, the output of the teacher network model when given the same training data is used as the label (true value) of that training data. The student network model's output (predicted value) continuously fits the teacher network model's output, converging to a level similar to or the same as the teacher network model. This improves the learning ability of the student network model without using labeled training data.

[0044] During the segment-by-segment training of the student network model, the network parameters of the teacher network model remain fixed, and only the network parameters of the current segment in the student network model are optimized. The teacher network model participates in the inference process of the training data during knowledge distillation, providing knowledge transfer without updating gradients. By performing segment-by-segment feature distillation on the student network model, the output distribution of each quantized segment is constrained by the high-precision teacher network model in both spatial and feature dimensions. This constrains the feature representation capability of the final target network model, thus significantly improving the final quantization accuracy.

[0045] Before training the student network model in segments, this embodiment of the invention requires segmenting both the teacher network model and the student network model in the same way. This embodiment of the invention does not limit the segmentation method.

[0046] In an optional embodiment of the present invention, segmenting the teacher network model and the student network model in the same manner may include: segmenting the teacher network model and the student network model with the same granularity; the granularity is determined based on the number of network layers, or the granularity is determined based on the network structure.

[0047] This invention does not limit the segmentation method for the teacher network model and the student network model. Since both models have the same structure, the same segmentation granularity can be used. The segmentation granularity can be determined based on the number of network layers; for example, segmentation can be based on the number of layers in a custom convolutional network. Alternatively, the segmentation granularity can be determined based on the network structure; for example, segmentation can be based on the network structure (e.g., blocks). A block can describe a single layer, a component composed of multiple layers, or the entire model itself. For example, similar structures (blocks) can be grouped into the same segment. Furthermore, segmentation can also be based on network stages; a downsampling operation in a deep neural network model is called a stage. Typically, a deep neural network model can consist of multiple modules with the same network structure, such as residual connections in ResNet, which are called blocks. Several blocks can form a stage.

[0048] For example, the teacher network model and the student network model can be segmented according to a preset number of network layers. For instance, a single convolutional layer can be used as the segmentation granularity; then, when training the student network model in segments, the student network model is trained layer by layer, quantizing each layer and performing knowledge distillation on that layer. Alternatively, the teacher network model and the student network model can be segmented according to a preset network structure. For instance, a single block can be used as the segmentation granularity; then, when training the student network model in segments, the student network model is trained block by block, quantizing each block and performing knowledge distillation on that block.

[0049] The above segmentation method is only an example. In specific implementations, a preset number of network layers (which can be greater than or equal to 1) can be used as the granularity of segmentation, or a preset number of blocks (which can be greater than or equal to 1) can be used as the granularity of segmentation, and so on.

[0050] In an optional embodiment of the present invention, quantizing the current segment and performing knowledge distillation on the current segment using the teacher network model and unlabeled training data may include:

[0051] Step S11: Insert pseudo-quantized nodes into the current segment of the student network model and initialize the quantization parameters of each pseudo-quantized node in the current segment.

[0052] Step S12: Obtain the training set, which includes unlabeled training data;

[0053] Step S13: Input the same unlabeled training data into the teacher network model and the student network model with inserted pseudo-quantized nodes, respectively, and optimize the network parameters of the current segment of the student network model according to the total model loss; the total model loss is determined according to the first loss and the second loss; the first loss is determined according to the output of the current segment of the student network model and the output of the corresponding segment in the teacher network model; the second loss is determined according to the output of the network output layer of the student network model and the output of the network output layer of the teacher network model.

[0054] For example, suppose the teacher network model and the student network model are each divided into k segments in the same way. First, the first segment of the student network model is trained. Specifically, pseudo-quantized nodes are inserted into the first segment of the student network model, and the quantization parameters of each pseudo-quantized node in the first segment are initialized. The second to kth segments of the student network model still retain floating-point parameters.

[0055] A training set is obtained, which includes unlabeled training data. This embodiment of the invention does not limit the type of unlabeled training data included in the training set; corresponding unlabeled training data can be collected according to different application scenarios. For example, in a pet recognition application scenario, a large number of images containing pets or animals can be collected, and the number and variety of images collected should be as large as possible. Similarly, in a speech recognition application scenario, a large number of audio recordings containing human speech can be collected, and the number and variety of audio recordings collected should be as large as possible.

[0056] The embodiments of this invention use unlabeled training data, eliminating the need to collect labeled privacy data, thus reducing the difficulty of collecting training data. Furthermore, because unlabeled training data is abundant and diverse, possessing strong randomness and diversity, complex data augmentation operations are unnecessary for the training data input to the model, reducing data manipulation costs.

[0057] After obtaining a training set containing unlabeled training data, the same unlabeled training data is input into both the teacher network model and the student network model with inserted pseudo-quantized nodes. For example, during the training of the first segment of the student network model, a specific unlabeled training data is extracted from the training set and input into both the teacher network model and the student network model with inserted pseudo-quantized nodes in the first segment. The network parameters of the first segment of the student network model are optimized based on the total model loss. The first loss is determined based on the output of the first segment of the student network model and the output of the first segment of the teacher network model; the second loss is determined based on the output of the network output layer of the student network model and the output of the network output layer of the teacher network model. When calculating the second loss, the student network model performs inference on the input training data, and the network output layer outputs a prediction result. This prediction result is obtained by progressively inferring from the quantized first segment and the floating-point second to k-th segments. Since the second to kth segments still maintain floating-point parameters, the accuracy of inference can be guaranteed. Only the output of the first segment may suffer from precision loss. Therefore, using the output of the first segment of the teacher network model and the output of the network output layer of the teacher network model to guide the parameter optimization of the first segment of the student network model can improve the accuracy of the first segment of the student network model.

[0058] It should be noted that the embodiments of the present invention do not limit the method of calculating the total model loss based on the first loss and the second loss. For example, the first loss and the second loss can be added together or weighted together to obtain the total model loss.

[0059] After the first segment of the student network model is trained, the network parameters of the first segment are fixed. Pseudo-quantized nodes are inserted into the second segment, and the quantization parameters of each pseudo-quantized node in the second segment are initialized. Knowledge distillation is then performed on the second segment using the same method. At this point, although the first segment is a quantized network, it has already undergone knowledge distillation from the high-precision teacher network model, achieving the same or similar accuracy as the first segment of the teacher network model. The third to kth segments retain floating-point parameters, with only the output of the second segment potentially experiencing accuracy loss. Therefore, using the output of the second segment of the teacher network model and the output layer of the teacher network model to guide parameter optimization of the second segment of the student network model can improve its accuracy. This process continues until the last segment of the student network model completes quantization and knowledge distillation. At this point, a compressed, low-bit student network model (i.e., the target network model) is obtained, which can be used to perform the target task and obtain accurate output results.

[0060] Reference Figure 2 This illustrates a schematic diagram of training the (k-1)th segment of a student network model in an example of an embodiment of the present invention. Figure 2As shown, the top represents segment k of the student network model, and the bottom represents segment k of the teacher network model. Segments 1 to k-2 of the student network model have been trained. Currently, segment k-1 needs to be trained, and it is the current segment. The network parameters of segments 1 to k-2 of the student network model are fixed. Pseudo-quantized nodes are inserted into segment k-1, and the quantization parameters of each pseudo-quantized node in segment k-2 are initialized using a calibration set. The same unlabeled training data is input into both the student network model and the teacher network model. Segments 1 to k-1 of the student network model are inferred based on fixed-point parameters, while segment k is inferred based on floating-point parameters. Each segment of the teacher network model is inferred based on floating-point parameters. A first loss is calculated based on the output of the current segment (segment k-1) of the student network model and the corresponding segment (segment k-1) of the teacher network model. A second loss is calculated based on the output of the network output layer (segment k) of both the student network model and the teacher network model. The total loss of the model is calculated based on the first loss and the second loss, and the network parameters of the current segment (segment k-1) of the student network model are optimized in reverse using the total loss of the model.

[0061] It is understandable that when training reaches the last segment of the student network model (such as the network output layer), the first loss is the same as the second loss. At this point, it is only necessary to calculate the loss between the output of the student network model's network output layer and the output of the teacher network model's network output layer.

[0062] In an optional embodiment of the present invention, initializing the quantization parameters of each pseudo-quantization node in the current segment may include:

[0063] Step S21: Obtain a calibration set, which is a subset of the training set;

[0064] Step S22: Input the calibration data in the calibration set into the student network model that has been inserted with pseudo-quantization nodes in sequence, and obtain the quantization information of each pseudo-quantization node in the current segment; the quantization information includes the maximum and minimum floating-point values ​​flowing through each pseudo-quantization node, as well as the preset maximum and minimum fixed-point values ​​corresponding to each pseudo-quantization node.

[0065] Step S22: Initialize the quantization parameters of each pseudo-quantization node in the current segment based on the quantization information of each pseudo-quantization node in the current segment.

[0066] The pseudo-quantization node is used for quantization and dequantization of data. Inserted into the student network model, the pseudo-quantization node first quantizes the high-precision input data to map it to low-precision data, and then dequantizes the low-precision data to obtain the high-precision output data. The pseudo-quantization node keeps both input and output data as floating-point numbers, but the difference is that the input data is a continuously variable floating-point number, while the output data is a discretized floating-point number. A continuously variable floating-point number refers to any floating-point number within a preset continuous range. For example, if the preset continuous range is 0 to 1, then the continuously variable floating-point number can include any decimal between 0 and 1. A discretized floating-point number refers to any floating-point number within a preset discrete range. For example, if the preset discrete range includes the following three floating-point numbers: 0.33, 0.66, and 0.99, then the discretized floating-point number can include any of these three floating-point numbers.

[0067] In this embodiment of the invention, pseudo-quantized nodes are inserted into the current segment of the student network model. Furthermore, pseudo-quantized nodes can be inserted at target nodes within the current segment of the student network model. The target node refers to a network layer node in the current segment of the student network model that supports quantization operations. For example, the target node includes, but is not limited to, nodes for weights, activations, and the input and output of operators.

[0068] In practical implementation, deep learning algorithms are composed of computational units, also known as operators. In deep neural network models, operators correspond to the computational logic within network layers. For example, a convolutional layer implementing convolution is an operator; a pooling layer implementing pooling is an operator; an activation layer implementing activation is an operator; a fully connected layer implementing fully connected operations is an operator; and so on. For an operator requiring quantization, this embodiment of the invention can insert a pseudo-quantization node before (input node) and after (output node) the operator.

[0069] In this embodiment of the invention, pseudo-quantization nodes are inserted at the target node in the current segment of the student network model. The calibration data from the calibration set are sequentially input into the student network model with the pseudo-quantization nodes inserted. The quantization information of each pseudo-quantization node in the current segment is statistically analyzed. Based on the quantization information of each pseudo-quantization node in the current segment, the quantization parameters of each pseudo-quantization node in the current segment are initialized.

[0070] The calibration set can be a subset of the training set. For example, a portion of the data can be extracted from the training set as the calibration set. The quantization information includes the maximum and minimum floating-point values ​​flowing through each pseudo-quantization node, as well as the preset maximum and minimum fixed-point values ​​corresponding to each pseudo-quantization node. In this embodiment of the invention, the inserted pseudo-quantization nodes can be used to statistically analyze the maximum and minimum floating-point values ​​flowing through the node during training, and can also be used to simulate the accuracy loss caused by quantization. Based on this accuracy loss, the model parameters are adjusted in reverse to continuously reduce the accuracy loss caused by quantization, thereby improving the accuracy of the quantized student network model. It should be noted that this embodiment of the invention does not limit the position and number of inserted pseudo-quantization nodes.

[0071] Based on the quantization information of each pseudo-quantization node in the current segment obtained through statistics, the quantization parameters of each pseudo-quantization node in the current segment can be initialized. The quantization parameters include a scaling factor s (scale) and a zero point z (zero_point). The goal of quantization is to calculate the scaling factor s and the zero point z based on the numerical range of the floating-point input and the quantized output. Here, floating-point input refers to floating-point type input data. Quantized output refers to the pseudo-quantization node using the quantization parameters to manipulate the floating-point input, mapping the continuous numerical floating-point input to discrete numerical values, and then mapping it to the preset output range to obtain the quantized output. The value of zero_point may be 0 (corresponding to symmetric quantization) or not 0 (corresponding to asymmetric quantization).

[0072] It should be noted that the specific method used for quantization is not limited in the embodiments of the present invention.

[0073] Preferably, embodiments of the present invention can employ a quantization-aware training method. Specifically, pseudo-quantized nodes are inserted into the student network model, and the quantization parameters corresponding to each pseudo-quantized node are statistically analyzed. Based on the quantization parameters, quantization-aware training is performed on the student network model with the inserted pseudo-quantized nodes to obtain a quantized student network model. After each segment of the student network model is sequentially quantized, the space occupied by the parameters can be reduced, thereby reducing the hardware memory occupied during runtime.

[0074] In an optional embodiment of the present invention, the first loss is the KL divergence loss calculated based on the output of the last layer of the current segment in the student network model and the output of the last layer of the corresponding segment in the teacher network model, and the second loss is the KL divergence loss calculated based on the Logits output of the student network model and the Logits output of the teacher network model.

[0075] In deep neural network models, Logits output is the output of the last layer of the network. For example, in a typical classification problem, such as image classification, after inputting an image, it undergoes various non-linear transformations within the deep neural network. Before the final Softmax layer, the network outputs numerical values ​​z indicating whether the image belongs to each category. i The value z of a certain category i The larger the value, the greater the likelihood that the model considers the input image to belong to that category. These summaries, which aggregate various information within the network, result in a total score z for each category. i This refers to Logits, where i represents the i-th category and z... i This represents the probability of belonging to the i-th class. Since Logits are not probability values, the Softmax function is typically used to transform the Logits values, and the resulting probability value is used as the final classification result. Softmax normalizes the probability of the Logits values ​​across classes, ensuring that the values ​​for each class classification follow a probability distribution.

[0076] For example, when training the first segment of the student network model, pseudo-quantized nodes are inserted into the first segment, and the quantization parameters of each pseudo-quantized node in the first segment are initialized. The second to kth segments retain floating-point parameters. Unlabeled training data is taken from the training set and input into both the teacher network model and the student network model with inserted pseudo-quantized nodes in the first segment for inference. The feature maps of the last layer output of the first segment of the student network model and the last layer output of the first segment of the teacher network model are extracted, and the KL divergence loss (first loss) of both is calculated. The first segment of the student network model uses fixed-point parameters for inference, while the second to the last segment still use floating-point parameters. The teacher network model is a pre-trained high-precision floating-point model. During the knowledge distillation process of the student network model segment by segment, the parameters of the teacher network model remain fixed, and each segment of the teacher network model uses floating-point parameters for inference. When training the first segment of the student network model, the Logits output of the student network model and the Logits output of the teacher network model are extracted, and the KL divergence loss (second loss) of both is calculated.

[0077] The total model loss is determined based on the first loss and the second loss. The network parameters of the first segment of the student network model are then optimized using this total model loss. The optimized network parameters include the quantization parameters of each pseudo-quantized node in the first segment. After the first segment of the student network model is trained, each subsequent segment is trained sequentially using the same method.

[0078] It should be noted that the embodiments of the present invention do not limit the loss functions used to calculate the first loss and the second loss. For example, any loss function such as cross-entropy loss, KL divergence loss, L2 loss, MGD loss, and FGD loss can be used.

[0079] In an optional embodiment of the present invention, the method may further include: removing pseudo-quantized nodes in the target network model and using the target network model to perform a target task.

[0080] After training the target network model, pseudo-quantized nodes can be removed, and the target network model can be deployed on the target device to perform the target task. This target network model can be deployed on resource-constrained hardware devices and can quickly infer correct prediction results with low power consumption.

[0081] The target devices include, but are not limited to: smart home terminals (including air conditioners, refrigerators, rice cookers, water heaters, etc.), smart business terminals (including video phones, smart conference desktop terminals, etc.), wearable devices (including smartwatches, smart glasses, etc.), smart financial terminals, as well as smartphones, tablets, personal digital assistants (PDAs), in-vehicle devices, computers, etc.

[0082] In summary, the model compression method provided by this invention combines model quantization and knowledge distillation, and employs a segmented, progressive approach to quantize and distill the student network model segment by segment, ultimately obtaining a target network model. This target network model is obtained by segmenting the student network model under the guidance of the teacher network model, exhibiting high accuracy and a lightweight design, allowing deployment on resource-constrained hardware devices (such as mobile devices). Furthermore, this invention utilizes unlabeled training data during the compression process of the student network model, independent of the original labeled training data, thus protecting the privacy and security of user data.

[0083] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0084] Reference Figure 3 The diagram illustrates a structural block diagram of an embodiment of a model compression device according to the present invention. The device may include:

[0085] The model acquisition module 301 is used to acquire the teacher network model and the student network model, wherein the teacher network model and the student network model are trained floating-point models with the same structure.

[0086] The model segmentation module 302 is used to segment the teacher network model and the student network model in the same way;

[0087] The compressed training module 303 is used to train each segment of the student network model sequentially; when training the current segment of the student network model, the network parameters of each segment that has been trained before the current segment are fixed, the current segment is quantized, and knowledge distillation is performed on the current segment using the teacher network model and unlabeled training data; when all segments of the student network model have been trained, the target network model is obtained.

[0088] Optionally, during the training of the current segment of the student network model, the compressed training module performs inference based on fixed-point parameters for the current segment of the student network model and the segments trained before the current segment, and performs inference based on floating-point parameters for the input unlabeled training data.

[0089] Optionally, the compressed training module includes:

[0090] The quantization submodule is used to insert pseudo-quantized nodes in the current segment of the student network model and initialize the quantization parameters of each pseudo-quantized node in the current segment.

[0091] The training set acquisition submodule is used to acquire the training set, which includes unlabeled training data;

[0092] The knowledge distillation submodule is used to input the same unlabeled training data into the teacher network model and the student network model with pseudo-quantized nodes inserted, respectively, and optimize the network parameters of the current segment of the student network model according to the total model loss; the total model loss is determined according to a first loss and a second loss; the first loss is determined according to the output of the current segment of the student network model and the output of the corresponding segment in the teacher network model; the second loss is determined according to the output of the network output layer of the student network model and the output of the network output layer of the teacher network model.

[0093] Optionally, the quantization submodule includes:

[0094] A calibration set acquisition unit is used to acquire a calibration set, which is a subset of the training set;

[0095] The quantization unit is used to sequentially input the calibration data from the calibration set into the student network model that has been inserted with pseudo-quantization nodes, and to obtain the quantization information of each pseudo-quantization node in the current segment; the quantization information includes the maximum and minimum floating-point values ​​flowing through each pseudo-quantization node, as well as the preset maximum and minimum fixed-point values ​​corresponding to each pseudo-quantization node; and to initialize the quantization parameters of each pseudo-quantization node in the current segment according to the quantization information of each pseudo-quantization node in the current segment.

[0096] Optionally, the first loss is the KL divergence loss calculated based on the output of the last layer of the current segment in the student network model and the output of the last layer of the corresponding segment in the teacher network model, and the second loss is the KL divergence loss calculated based on the Logits output of the student network model and the Logits output of the teacher network model.

[0097] Optionally, the model segmentation module is specifically used to segment the teacher network model and the student network model according to the same segmentation granularity; the segmentation granularity is determined based on the number of network layers, or the segmentation granularity is determined based on the network structure.

[0098] Optionally, the device further includes:

[0099] The model application module is used to remove pseudo-quantized nodes in the target network model and use the target network model to perform the target task.

[0100] The model compression device provided in this invention combines model quantization and knowledge distillation, and employs a segmented, progressive approach to quantize and distill the student network model segment by segment, ultimately obtaining a target network model. This target network model is obtained by segmenting the student network model under the guidance of the teacher network model, exhibiting high accuracy and a lightweight design, and can be deployed on resource-constrained hardware devices (such as mobile devices). Furthermore, this invention utilizes unlabeled training data during the compression process of the student network model, independent of the original labeled training data, thus protecting the privacy and security of user data.

[0101] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0103] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0104] This invention also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by a processor of a device (server or terminal), the device is able to execute the foregoing text. Figure 1 The description of the model compression method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For any technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0105] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0106] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0108] The above provides a detailed description of the model compression method, apparatus, and machine-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A model compression method, characterized in that, The method includes: Obtain a teacher network model and a student network model, wherein the teacher network model and the student network model are pre-trained floating-point models with the same structure; the teacher network model and the student network model are obtained based on a training set, wherein the training set includes image data or speech data; The teacher network model and the student network model are segmented in the same way, including: segmenting the teacher network model and the student network model with the same granularity; the granularity is determined based on the number of network layers, or the granularity is determined based on the network structure. Each segment of the student network model is trained sequentially. When training the current segment of the student network model, the network parameters of the segments that have been trained before the current segment are fixed. Pseudo-quantized nodes are inserted into the current segment of the student network model, and the quantization parameters of each pseudo-quantized node in the current segment are initialized to quantize the current segment. Knowledge distillation of the current segment is performed using the teacher network model and unlabeled training data. The target network model is obtained when all segments of the student network model have been trained. The initialization of the quantization parameters for each pseudo-quantization node in the current segment includes: Obtain a calibration set, which is a subset of the training set; The calibration data in the calibration set is sequentially input into the student network model that has been inserted with pseudo-quantization nodes, and the quantization information of each pseudo-quantization node in the current segment is obtained; the quantization information includes the maximum and minimum floating-point values ​​flowing through each pseudo-quantization node, as well as the preset maximum and minimum fixed-point values ​​corresponding to each pseudo-quantization node. The quantization parameters of each pseudo-quantization node in the current segment are initialized based on the quantization information of each pseudo-quantization node in the current segment.

2. The method according to claim 1, characterized in that, The method further includes: During the training of the current segment of the student network model, for the input unlabeled training data, the current segment of the student network model and the segments trained before the current segment are inferred based on fixed-point parameters, while the segments after the current segment of the student network model are inferred based on floating-point parameters.

3. The method according to claim 1, characterized in that, The step of quantizing the current segment and performing knowledge distillation on the current segment using the teacher network model and unlabeled training data includes: Obtain a training set, which includes unlabeled training data; The same unlabeled training data is input into the teacher network model and the student network model with pseudo-quantized nodes inserted, respectively. The network parameters of the current segment of the student network model are optimized according to the total model loss. The total model loss is determined according to a first loss and a second loss. The first loss is determined according to the output of the current segment of the student network model and the output of the corresponding segment in the teacher network model. The second loss is determined according to the output of the network output layer of the student network model and the output of the network output layer of the teacher network model.

4. The method according to claim 3, characterized in that, The first loss is the KL divergence loss calculated based on the output of the last layer of the current segment in the student network model and the output of the last layer of the corresponding segment in the teacher network model. The second loss is the KL divergence loss calculated based on the Logits output of the student network model and the Logits output of the teacher network model.

5. The method according to claim 1, characterized in that, The method further includes: Remove pseudo-quantized nodes from the target network model and use the target network model to perform the target task.

6. A model compression device, characterized in that, The device includes: The model acquisition module is used to acquire a teacher network model and a student network model, wherein the teacher network model and the student network model are pre-trained floating-point models with the same structure; the teacher network model and the student network model are obtained based on a training set, wherein the training set includes image data or speech data; The model segmentation module is used to segment the teacher network model and the student network model in the same way, including: segmenting the teacher network model and the student network model with the same partitioning granularity; the partitioning granularity is determined based on the number of network layers, or the partitioning granularity is determined based on the network structure; The compressed training module is used to train each segment of the student network model sequentially. When training the current segment of the student network model, the network parameters of the previously trained segments are fixed, pseudo-quantization nodes are inserted into the current segment of the student network model, and the quantization parameters of each pseudo-quantization node in the current segment are initialized to quantize the current segment. Knowledge distillation is performed on the current segment using the teacher network model and unlabeled training data. When all segments of the student network model have been trained, the target network model is obtained. The compressed training module includes an initialization module for obtaining a calibration set, which is a subset of the training set; sequentially inputting the calibration data from the calibration set into the student network model with pseudo-quantized nodes inserted, and obtaining the quantization information of each pseudo-quantized node in the current segment; the quantization information includes the maximum and minimum floating-point values ​​flowing through each pseudo-quantized node, as well as the preset maximum and minimum fixed-point values ​​corresponding to each pseudo-quantized node; and initializing the quantization parameters of each pseudo-quantized node in the current segment according to the quantization information of each pseudo-quantized node in the current segment.

7. The apparatus according to claim 6, characterized in that, The compressed training module includes: The training set acquisition submodule is used to acquire the training set, which includes unlabeled training data; The knowledge distillation submodule is used to input the same unlabeled training data into the teacher network model and the student network model with pseudo-quantized nodes inserted, respectively, and optimize the network parameters of the current segment of the student network model according to the total model loss; the total model loss is determined according to a first loss and a second loss; the first loss is determined according to the output of the current segment of the student network model and the output of the corresponding segment in the teacher network model; the second loss is determined according to the output of the network output layer of the student network model and the output of the network output layer of the teacher network model.

8. A machine-readable storage medium having instructions stored thereon that, when executed by one or more processors of a device, cause the device to perform the model compression method as described in any one of claims 1 to 5.

Images