Model compression method and device, electronic equipment and storage medium

Abstract

The invention provides a model compression method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, model compression, pruning, knowledge distillation and the like. According to the specific implementation scheme, in the process of performing knowledge transfer to a student network based on a teacher network, at least one to-be-pruned channel is screened out from the student network based on pruning loss and distillation loss; and compressing the student network based on the at least one channel to be pruned. According to the method and the device, automatic pruning can be carried out according to the pruning technology while knowledge distillation is realized, and the requirements of the student network on the reasoning speed and precision are effectively ensured, so that the occupation of storage resources and the consumption of computing resources by the student network are reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, model compression, pruning and knowledge distillation. Background Art

[0002] Deep neural networks, relying on large model parameters and excellent reasoning capabilities, have achieved certain success in industry and academia. However, due to limited resources, many devices cannot install large neural network models.

[0003] Therefore, model compression technology is needed. Model compression is a key technology in the field of deep learning. It aims to reduce the size and computational complexity of the model to reduce the model's consumption of storage resources (such as memory) and computing resources, and ultimately enable the model to be deployed and run on resource-constrained devices. Summary of the invention

[0004] The present disclosure provides a model compression method, device, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, a model compression method is provided, comprising:

[0006] In the process of transferring knowledge from the teacher network to the student network, at least one channel to be pruned is selected from the student network based on the pruning loss and the distillation loss;

[0007] Compress the student network based on at least one channel to be pruned.

[0008] According to another aspect of the present disclosure, there is provided a model compression device, comprising:

[0009] A screening module, used for screening at least one channel to be pruned from the student network based on pruning loss and distillation loss in the process of transferring knowledge from the teacher network to the student network;

[0010] A compression module is used to compress the student network based on at least one channel to be pruned.

[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a flow chart of a model compression method provided according to an embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of a process for determining a channel to be pruned according to an embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of a process for determining pruning hyperparameters and distillation hyperparameters according to an embodiment of the present disclosure;

[0022] Figure 4 is a flowchart of self-pruning knowledge distillation training provided according to an embodiment of the present disclosure;

[0023] Figure 5 is a structural schematic diagram of a model compression device provided according to an embodiment of the present disclosure;

[0024] Figure 6 It is a block diagram of an electronic device used to implement the model compression method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The breakthrough progress of deep learning technology has greatly improved the application of artificial intelligence technology in the industrial world. For example, computer vision technology can be used to assist in intelligent driving.

[0028] However, many devices currently have limited resources, and some larger neural network models cannot be installed on these devices. Therefore, a model compression method is provided in the embodiments of the present disclosure to reduce the scale and computational complexity of the model without reducing the accuracy of the model as much as possible, so as to facilitate the deployment and operation of the model on a device with limited resources.

[0029] It should be understood that model compression can reduce the model's requirements for storage resources, such as reducing the use of storage resources and computing resources, thereby increasing the processing speed of the device according to the model on the corresponding task and improving the computing efficiency of the device on which the model is installed.

[0030] Model compression technologies mainly include quantization, pruning, knowledge distillation, and neural network architecture search. Among them, pruning and knowledge distillation are currently the most efficient model compression methods.

[0031] Based on this, the model compression method provided in the embodiment of the present disclosure, and the model obtained based on this method can be applicable to a variety of scenarios, such as computer vision task scenarios and language processing scenarios, and the embodiment of the present disclosure does not limit this.

[0032] like Figure 1 FIG. 1 is a flow chart of a model compression method provided by an embodiment of the present disclosure, including the following contents:

[0033] S101, in the process of transferring knowledge from the teacher network to the student network, at least one channel to be pruned is selected from the student network based on pruning loss and distillation loss.

[0034] In the field of deep learning, the teacher network usually refers to a well-trained, high-performance complex model that can provide accurate predictions and rich feature representations. The student network is a simpler model that aims to learn and imitate the behavior of the teacher network in order to reduce the consumption of computing resources while maintaining similar performance. Knowledge transfer refers to the process of transferring the "knowledge" of the teacher network to the student network, so that the student network can reach or approach the performance of the teacher network.

[0035] The Teacher-Student Training Paradigm is a knowledge distillation technique in which the student network can learn the knowledge of the teacher network and thus improve its own performance.

[0036] Among them, pruning is a deep learning model optimization technology that reduces the complexity of the model by removing redundant connections or neurons in the neural network. Doing so can not only reduce the size and amount of computation of the model, but also improve the running speed of the model to a certain extent while maintaining the model performance. There are two main types of pruning, including unstructured pruning and structured pruning. Because unstructured pruning requires dedicated hardware or acceleration libraries, the commonly used pruning method is structured pruning.

[0037] S102: Compress the student network based on at least one channel to be pruned.

[0038] It can be understood that the channels to be pruned are removed from the student network to reduce the model size and accelerate reasoning.

[0039] If pruning and knowledge distillation are used to compress the model, the model compression effect will be unsatisfactory due to the misalignment of the experience of pruning and knowledge distillation. For example, the important channels considered by the pruning task are inconsistent with the knowledge distillation task, and the channels considered important by the knowledge distillation task may be easily pruned during pruning. This will result in the loss being unable to be compensated in the knowledge distillation process and the subsequent model compression process, resulting in impaired model accuracy.

[0040] Therefore, in the disclosed embodiment, the pruning loss and the distillation loss jointly decide the channels to be pruned. In this way, the experience of pruning and knowledge distillation can be aligned, and the channels that both consider important can be retained. It is possible to automatically prune according to the pruning technology while knowledge distillation is in progress, effectively ensuring the requirements of the student network in terms of reasoning speed and accuracy, thereby reducing the student network's occupancy of storage resources and consumption of computing resources. For example, the model size is reduced, so that the student network can save storage resources relative to the teacher network when installed on the device. In the reasoning stage when the model performs the corresponding task, pruning can reduce the memory consumption of the intermediate data generated in the reasoning process, so that the student network model can complete the task as accurately as the teacher network on a device with limited resources, and can also improve the reasoning speed.

[0041] In the disclosed embodiment, in order to easily and quickly determine the channel to be pruned, Figure 2 As shown, selecting at least one channel to be pruned from the student network based on the pruning loss and the distillation loss can be implemented as follows:

[0042] S201, determining scaling factors of multiple channels in the student network based on pruning loss and distillation loss.

[0043] S202: Screen out at least one channel to be pruned based on the scaling factors of the multiple channels.

[0044] The expression of pruning loss can be expressed as shown in formula (1):

[0045] L J =∑|γ| (1)

[0046] In expression (1), the pruning loss L J L1 regularization is used to express it. Therefore, during the pruning process, the scaling factor γ of multiple channels in the student network can be optimized so that the scaling factor gradually approaches 0, thereby screening out unimportant channels as channels to be pruned.

[0047] For example, in the disclosed embodiment, channels with scaling factors less than a preset threshold can be screened to obtain at least one channel to be pruned. Thus, the channels to be pruned that need to be pruned can be directly and easily screened out by the size of the scaling factor, which can improve the efficiency of pruning and the efficiency of model compression.

[0048] In the disclosed embodiment, the scaling factor of the channel can be jointly optimized by pruning loss and distillation loss, and knowledge distillation can be used jointly in the pruning process to screen out channels that both parties consider important for retention, and screen out unimportant channels for model compression.

[0049] In implementation, the scaling factors of multiple channels in the student network are determined based on the pruning loss and the distillation loss, which can be implemented as:

[0050] Step A1, determine the expression of the total loss based on the pruning loss and the distillation loss.

[0051] During implementation, the expression of the total loss may be determined based on the relationship that the total loss is positively correlated with the pruning loss, and the total loss is positively correlated with the distillation loss. For example, the total loss may be determined to be the sum of the pruning loss and the distillation loss.

[0052] In order to further better balance the sparsity and accuracy of the student network, in the disclosed embodiment, the pruning loss and the distillation loss may be weightedly summed to obtain an expression for the total loss.

[0053] For example, the weight of the pruning loss is λ P , the corresponding distillation loss weight is (1-λ P ). P It can be used to adjust the sparsity of the student network, and also affects the accuracy of the student network. P The larger the value, the sparser the student model is. You can increase λ while maintaining the accuracy of the learning network. P The value of λ can be determined based on experience or experiments. P Get the value.

[0054] Therefore, through the weighted summation method, a weight parameter is introduced, which can balance the sparsity of the student network, thereby effectively compressing the scale of the model and reducing the occupation of storage resources and computing resources. It can also adjust the accuracy of the student network, thereby finding a reasonable balance between reasoning speed and accuracy.

[0055] As described above, the pruning loss is defined by the scaling factors of multiple channels, so the pruning loss itself includes the scaling factor parameter. In order to obtain a suitable scaling factor to achieve the purpose of reasonable pruning, in step A2, based on the expression of the total loss, the total loss is minimized to obtain the scaling factors of multiple channels in the pruning loss.

[0056] In the disclosed embodiment, the relationship between the total loss, pruning loss and distillation loss is established. As a result, the scaling factor of the pruning loss becomes a parameter that can be continuously optimized and solved based on the total loss, thereby achieving the purpose of joint training based on pruning loss and distillation loss. Therefore, solving the optimized scaling factor can help screen out channels that are considered important by both pruning and distillation, and can effectively ensure the requirements of the student network in terms of reasoning speed and accuracy.

[0057] In some embodiments, the distillation loss may be the loss between the student network and the teacher network.

[0058] For example, the loss may include at least one of the following losses:

[0059] 1) Response-based knowledge loss, which is, for example, the loss between the prediction results of the student network and the prediction results of the teacher network.

[0060] 2) Feature-based knowledge loss, which is, for example, the loss between the feature distribution of the middle layer of the student network and the feature distribution of the middle layer of the teacher network. These middle layers can be hidden neural network layers that are important to the task, and can be determined according to the specific task during implementation.

[0061] 3) Relationship-based knowledge loss, which can include the aforementioned response-based knowledge loss and at least feature-based loss. In addition, the relationship-based knowledge loss can also include the loss between the input layers of the student network and the teacher network.

[0062] In some embodiments, the distillation loss may also include a student network loss and a teacher network loss. As the name implies, the student network loss refers to the loss generated by the student network during the reasoning process, and the teacher network loss is the loss generated by the teacher network during the reasoning process. The specific expression of the student network loss and the specific expression of the teacher network loss are not limited in the disclosed embodiments. For example, the student network loss may be at least one of the three losses listed above, or it may be the loss between the predicted result and the true value of the student network. The teacher network loss may be the loss between the predicted result and the true value of the teacher network. During implementation, the teacher network loss is used for the student model to learn the pattern of the teacher network to achieve knowledge distillation.

[0063] In implementation, the distillation loss is positively correlated with the student network loss and positively correlated with the teacher network loss. According to the relationship, the specific expressions of the distillation loss, the student network loss and the teacher network loss can be set, for example, the sum of the two is determined as the distillation loss.

[0064] In some embodiments, not only can the weights of pruning loss and distillation loss be used to balance the inference speed and accuracy of the model, but the teacher network loss and the student network loss can also be weighted and summed to obtain the distillation loss. On this basis, through the weights of the student network loss and the teacher network loss, the distillation loss can focus on the student network or the teacher network, so as to balance the learning intensity of the teacher network, thereby balancing the inference speed and model accuracy.

[0065] For example, the distillation loss can be expressed as expression (2):

[0066] L z=(1-λ D )L S +λ D L T (2)

[0067] In expression (2), L z is the distillation loss, L S For student network loss, L T is the teacher network loss, λ D is the teacher network weight, (1―λ D ) is the student network weight.

[0068] Among them, λ D The larger the value, the more trust the teacher network has, and the model accuracy of the student network is closer to that of the teacher network. Therefore, by properly adjusting λ D To maintain the accuracy of the student network. During implementation, a reasonable λ can be determined based on experience. D .

[0069] Based on the above, the total loss can be expressed as (3):

[0070] L final =(1-λ P )((1―λ D )L S +λ D L T )+λ P ∑|γ| (3)

[0071] In expression (3), L final Represents the total loss. The meanings of other parameters have been explained in the previous article and will not be repeated here.

[0072] Based on expression (3), we can see that by minimizing L final γ can be optimized to obtain a reasonable scaling factor to screen out channels to be pruned.

[0073] In the disclosed embodiment, in order to better implement automatic pruning to balance the inference speed and model accuracy, the expression of the total loss may also include adjustable pruning hyperparameters and distillation hyperparameters, where:

[0074] The pruning hyperparameter is used to adjust the sparsity of the student network;

[0075] The distillation hyperparameters are used to tune the accuracy of the student network.

[0076] During the process of knowledge transfer, the model compression efficiency can be improved by optimizing these two hyperparameters in the total loss to balance the inference speed and model accuracy of the student network.

[0077] Exemplarily, the pruning hyperparameter is the weight of the pruning loss; the distillation hyperparameter is the weight of the teacher network loss in the distillation loss. P Can be used as pruning hyperparameters, λ D It can be used as a distillation hyperparameter. Therefore, the weights of different losses are adjustable and continuously optimized during the knowledge transfer process, so that the total loss can better measure the channels that are considered important by both pruning and knowledge distillation, thereby improving the model compression efficiency.

[0078] In some embodiments, in order to improve the optimization efficiency of hyperparameters, these hyperparameters can be iteratively optimized in an automated manner. In one possible implementation, these two hyperparameters can be respectively [10 -1 ,10 -6 ] interval to find the optimal value. In order to optimize these two hyperparameters as much as possible, the number of searches at any time can be n times, where n is a positive integer greater than 1, for example, n is 100 times. Thus, the hyperparameters that can make the model performance of the student network best can be found in these n random search results. Among them, the model performance includes inference speed and accuracy. A comprehensive evaluation can be made from these two aspects, and the specific evaluation method is not limited in the embodiments of this disclosure.

[0079] In another possible implementation, Figure 3 As shown, determining the pruning hyperparameters and distillation hyperparameters can also be implemented as follows:

[0080] S301, based on the prior estimation, sample m parameter pairs within the target value range, where m is a positive integer greater than 1, and each parameter pair includes a pruning hyperparameter sample point and a distillation hyperparameter sample point.

[0081] Among them, the pruning hyperparameters and distillation hyperparameters can be estimated a priori as a uniform distribution in the current value range [0,1].

[0082] During implementation, the pruning hyperparameters and the distillation hyperparameters can be uniformly sampled in the interval [0, 1] to obtain m parameter pairs.

[0083] S302, pruning the student network based on the m parameter pairs to obtain multiple candidate models.

[0084] That is, each parameter pair is used as a hyperparameter to determine the expression of the total loss based on the above method, and optimize the scaling factor, thereby achieving compression of the student model. After compression, each parameter pair obtains the corresponding compressed student model, which is the candidate model corresponding to the parameter pair.

[0085] S303, determining the model performances of the candidate models corresponding to the m parameter pairs respectively.

[0086] Among them, model performance can be measured based on model inference speed and model accuracy.

[0087] S304: Perform kernel density estimation based on the model performance of each candidate model to obtain the actual distribution probability of the parameter pair.

[0088] Among them, Kernel Density Estimation (KDE) is a non-parametric statistical method used to estimate the unknown probability density function. It does not rely on prior knowledge of data distribution, but directly studies the data distribution characteristics from the data sample itself. KDE estimates the probability density function of the entire data by placing a kernel function around each data point and superimposing it. Commonly used kernel functions include Gaussian kernel, Epanechnikov kernel and uniform kernel, among which Gaussian kernel is more popular due to its mathematical properties and smoothness.

[0089] Taking the Gaussian kernel as an example, the present disclosure embodiment involves kernel density estimation in three-dimensional space. The parameters of the three dimensions are pruning hyperparameters λ P , distillation hyperparameter λ D , and model performance Z.

[0090] The assumed three-dimensional parameter pair data set is: {(x1,y1,z1),(x2,y2,z2),…,(x m ,y m ,z m )}where (x i ,y i ,z i ) is the i-th parameter pair in the dataset. i Represents the pruning hyperparameter sample point λ in the i-th parameter pair P ;y i Represents the distillation hyperparameter sample point λ in the i-th parameter pair D ; z i Represents the model performance Z corresponding to the i-th parameter pair.

[0091] For any point (x c ,y c ,z c ), if you want to estimate its probability density, you need to calculate the probability density of each data point (x i ,y i ,z i ) applies a three-dimensional Gaussian kernel and computes all point pairs (x c ,y c ,z c ). The formula for kernel density estimation based on three-dimensional Gaussian kernel is shown in expression (4):

[0092]

[0093] In expression (4), h is the bandwidth, which determines the width of the two-dimensional Gaussian kernel function; K() represents the three-dimensional Gaussian kernel function; and n represents (the total number of parameter pairs - 1), that is, (m-1).

[0094] In implementation, in the parameter pair data set, each parameter pair can be used as a point (x c ,y c ,z c ), thus obtaining the kernel density probability of each parameter pair, which can also be called kernel contribution. Note that the kernel density probability here is normalized.

[0095] Finally, a smooth curve can be obtained by data fitting, namely λ P , D , and a curve of the model performance Z with respect to the actual distribution probability (Z). Thus, in S305, the parameter pair with the highest probability value is selected from the actual distribution probability to obtain the pruning hyperparameter and the distillation hyperparameter.

[0096] In the disclosed embodiment, high-quality hyperparameters can be automatically screened out through kernel density estimation, thereby optimizing the hyperparameters in the total loss and improving the effect of model compression.

[0097] It is understandable that model compression is not a one-shot process. Figure 1 The method shown can obtain a student network with good quality. Therefore, in the disclosed embodiment, after compressing the student network based on the channel to be pruned, it is necessary to fine-tune the compressed student network; if it is determined based on the fine-tuning result that the student network does not meet the convergence condition, return to the step of selecting at least one channel to be pruned from the student network based on the pruning loss and the distillation loss until the student network meets the convergence condition.

[0098] In the disclosed embodiment, through iterative optimization, model compression can be gradually completed, the sparsity and model accuracy of the model can be balanced, and the efficiency of model compression can be improved.

[0099] In summary, the overall model compression process can be as follows Figure 4 As shown:

[0100] After training to obtain a teacher network capable of knowledge transfer, the teacher network is trained normally to obtain the teacher network loss. At the same time, the student network is trained normally to obtain the student network loss. Based on these two losses, the joint training of pruning and sorting is implemented, that is, the distillation loss is determined based on the teacher network loss and the student network loss, and the expression of the total loss shown in expression (3) is determined based on the distillation loss and the pruning loss. In this way, the scaling factor of the channel in the pruning loss is optimized to screen out the channels to be pruned. Then the student network is pruned based on the channels to be pruned. After pruning, the student network is fine-tuned. This completes a round of iterations. In a round of iterations, as described above, it is necessary to first optimize the required hyperparameters, and then prune based on the total loss. After a round of iterations, it can be determined whether the student network meets the convergence conditions. If the convergence conditions are not met, the next round of iterations can be continued until the model compression is completed.

[0101] Take YOLOv5 as an example. First, use the pre-trained model provided by the official website to train the dataset for 100 epochs to obtain a baseline model. During the training process, the optimizer used is adamW, and the initial learning rate is set to 10 -4 , the temperature T is set to 5, and the pruning hyperparameter is 10 -2 , the distillation hyperparameter is 0.3, and the baseline model is trained with sparse and distillation for 100 epochs, the pruning threshold is set to 0.3 for pruning, and then fine-tuned for 20 epochs, iterating 5 times. The comparison between the baseline model and the compressed model is shown in Table 1:

[0102] Table 1

[0103] Model YOLOv5-t (baseline) YOLOv5-s mAP 37.2 37.3 Storage capacity (MB) 15.8 9.3 Parameters (MB) 7.2 4.8 Speed ​​(ms) 10.4 8.6

[0104] Based on the comparison results in Table 1, it can be seen that when the average accuracy of the models is consistent / similar (i.e. <1%), the storage capacity of the compressed model is reduced by 41%, the number of model parameters is reduced by 33%, and the inference speed is increased by 17%. Therefore, the embodiment of the present disclosure provides an efficient model compression solution.

[0105] In the field of autonomous driving, the method of the disclosed embodiment can be used to compress the model so that the compressed student network can perform reasoning on complex models on a low-cost hardware platform. For example, it can be applied to target detection (such as detecting obstacles such as pedestrians / vehicles based on the visual sensors of autonomous driving vehicles) and model compression in the field of target segmentation. Thus, the capabilities of the large model can be used to obtain a smaller student model through the model compression technology provided by the disclosed embodiment. The student model takes up less memory and consumes less memory during the reasoning process. Therefore, the student model can be applied to resource-constrained autonomous driving vehicles, so that the autonomous driving vehicle can use the compressed model for target detection in real time, such as identifying obstacles in real time, so as to improve the accuracy of autonomous driving decisions and improve driving safety.

[0106] It should be understood that the field of autonomous driving is only an example of the model compression method provided by the embodiment of the present disclosure. The model compression method provided by the embodiment of the present disclosure can be applied to model compression of computer vision, model compression of natural language processing, and model compression of multimodal information processing.

[0107] Taking computer vision as an example, the student network is a computer vision model, which is used to perform at least one of target detection, target segmentation and target classification tasks on the input image. That is, through the model compression method provided by the embodiment of the present disclosure, a large-scale computer vision model can be compressed into a student network with a smaller scale, lower computational complexity but almost no change in accuracy. The student network can be applied to a terminal device with limited resources to facilitate the acquisition of images captured by the terminal device, and the student network is used to process the image to complete image analysis. For example, tasks such as target detection, target segmentation, and target classification in an image can be completed.

[0108] Taking natural language processing as an example, the student network is a language processing model, which is used to perform reasoning analysis on the input voice and / or text information to complete at least one of the tasks of human-computer dialogue, semantic understanding and text map tasks. Taking a large language model as an example, the model compression method provided by the embodiment of the present disclosure can compress the large language model into a student network with a smaller scale, lower computational complexity but almost no change in accuracy. The student network can be applied to resource-constrained terminal devices to facilitate the acquisition of voice collected by the terminal device or text information provided by the terminal device, and the student network is used to process the voice or text information to complete language analysis. For example, a more humanized and accurate dialogue can be conducted with the target object, reading and comprehension of text information can be performed, and text map tasks can be performed, etc.

[0109] Taking multimodal information processing as an example, the compressed student network is a multimodal model, which is used to analyze the input multimodal information to complete at least one of the following tasks: resource recommendation and providing a basis for autonomous driving decision-making. Taking a large multimodal model as an example, the model compression method provided by the embodiment of the present disclosure can compress the large multimodal model into a student network with a smaller scale, lower computational complexity but almost no change in accuracy. The student network can be applied to resource-constrained terminal devices to facilitate the acquisition of multimodal information, such as audio, video, text, etc. The student network is used to process the multimodal information to complete the reasoning of the corresponding task. For example, big data mining can be performed based on multimodal information, relevant resources can be searched for information recommendation using multimodal information, and the surrounding environment can be understood using multimodal information to improve the accuracy of driving decisions in the field of autonomous driving.

[0110] Based on the same technical concept, the embodiment of the present disclosure also provides a model training device 500, such as Figure 5 As shown, including:

[0111] A screening module 501 is used to screen out at least one channel to be pruned from the student network based on pruning loss and distillation loss during the process of transferring knowledge from the teacher network to the student network;

[0112] The compression module 502 is used to compress the student network based on at least one channel to be pruned.

[0113] In some embodiments, the screening module comprises:

[0114] determining a subunit for determining scaling factors of multiple channels in a student network based on a pruning loss and a distillation loss;

[0115] The screening subunit is used to screen out at least one channel to be pruned based on the scaling factors of the multiple channels.

[0116] In some embodiments, the screening subunit is specifically used for:

[0117] Channels whose scaling factors are less than a preset threshold are screened to obtain at least one channel to be pruned.

[0118] In some embodiments, the determining subunit is specifically used to:

[0119] Determine the expression for the total loss based on pruning loss and distillation loss;

[0120] Based on the expression of the total loss, the total loss is minimized to obtain the scaling factors of multiple channels in the pruning loss.

[0121] In some embodiments, the determining subunit is specifically used for:

[0122] The pruning loss and the distillation loss are weightedly summed to obtain an expression for the total loss.

[0123] In some embodiments, the determining subunit is specifically used to:

[0124] The teacher network loss and the student network loss are weighted summed to obtain the distillation loss.

[0125] In some embodiments, the expression of the total loss includes learnable pruning hyperparameters and distillation hyperparameters, where:

[0126] The pruning hyperparameter is used to adjust the sparsity of the student network;

[0127] The distillation hyperparameters are used to adjust the accuracy of the student network.

[0128] In some embodiments, the pruning hyperparameter is a weight of the pruning loss;

[0129] The distillation hyperparameter is the weight of the teacher network loss in the distillation loss.

[0130] In some embodiments, the determining subunit is further used to:

[0131] Based on the prior estimation, m parameter pairs are sampled within the target value range, where m is a positive integer greater than 1, and each parameter pair includes a pruning hyperparameter sample point and a distillation hyperparameter sample point;

[0132] Pruning the student network based on the m parameter pairs to obtain multiple candidate models;

[0133] Determine the model performance of the candidate models corresponding to the m parameter pairs respectively;

[0134] Based on the model performance of each candidate model, kernel density estimation is performed to obtain the actual distribution probability of parameter pairs;

[0135] A parameter pair with the highest probability value is selected from the actual distribution probability to obtain the pruning hyperparameter and the distillation hyperparameter.

[0136] In some embodiments, it also includes:

[0137] A fine-tuning module, used for fine-tuning the compressed student network;

[0138] A convergence module is used to trigger the screening module to return to the step of screening at least one channel to be pruned from the student network based on the pruning loss and the distillation loss when it is determined based on the fine-tuning result that the student network does not meet the convergence condition, until the student network meets the convergence condition.

[0139] In some embodiments, the student network is a computer vision model, which is used to perform at least one of target detection, target segmentation and target classification tasks on an input image.

[0140] In some embodiments, the student network is a language processing model, which is used to perform reasoning analysis on input voice and / or text information to complete at least one of human-computer dialogue, semantic understanding and text-based diagram tasks.

[0141] In some embodiments, the student network is a multimodal model, and the multimodal model is used to analyze the input multimodal information to complete at least one task of recommending resources and providing a basis for autonomous driving decisions.

[0142] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0143] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0144] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0145] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0146] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0147] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0148] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the model compression method. For example, in some embodiments, the model compression method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the model compression method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the model compression method in any other appropriate manner (e.g., by means of firmware).

[0149] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0151] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] In addition, an embodiment of the present disclosure also provides a vehicle, comprising the aforementioned electronic device.

[0154] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0155] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0157] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model compression method, comprising: In the process of transferring knowledge from the teacher network to the student network, selecting at least one channel to be pruned from the student network based on the pruning loss and the distillation loss; Based on the at least one channel to be pruned, the student network is compressed.

2. The method according to claim 1, wherein: The step of selecting at least one channel to be pruned from the student network based on the pruning loss and the distillation loss includes: determining scaling factors for a plurality of channels in the student network based on the pruning loss and the distillation loss; The at least one channel to be pruned is screened out based on the scaling factors of the multiple channels.

3. The method according to claim 2, wherein: The step of screening out the at least one channel to be pruned based on the scaling factors of the multiple channels includes: Channels whose scaling factors are less than a preset threshold are screened to obtain the at least one channel to be pruned.

4. The method according to claim 2, wherein: The step of determining scaling factors of multiple channels in the student network based on the pruning loss and the distillation loss includes: Determine an expression for a total loss based on the pruning loss and the distillation loss; Based on an expression of the total loss, the total loss is minimized to obtain scaling factors of the multiple channels in the pruning loss.

5. The method according to claim 4, wherein: The expression for determining the total loss based on the pruning loss and the distillation loss includes: The pruning loss and the distillation loss are weightedly summed to obtain an expression for the total loss.

6. The method according to claim 4, wherein: Determining the distillation loss comprises: The teacher network loss and the student network loss are weighted summed to obtain the distillation loss.

7. The method according to claim 4, wherein the expression of the total loss includes learnable pruning hyperparameters and distillation hyperparameters, wherein: The pruning hyperparameter is used to adjust the sparsity of the student network; The distillation hyperparameters are used to adjust the accuracy of the student network.

8. The method according to claim 7, wherein: The pruning hyperparameter is the weight of the pruning loss; The distillation hyperparameter is the weight of the teacher network loss in the distillation loss.

9. The method according to claim 7, wherein: Determining the pruning hyperparameter and the distillation hyperparameter includes: Based on the prior estimation, m parameter pairs are sampled within the target value range, where m is a positive integer greater than 1, and each parameter pair includes a pruning hyperparameter sample point and a distillation hyperparameter sample point; Pruning the student network based on the m parameter pairs to obtain multiple candidate models; Determine the model performance of the candidate models corresponding to the m parameter pairs respectively; Based on the model performance of each candidate model, kernel density estimation is performed to obtain the actual distribution probability of parameter pairs; A parameter pair with the highest probability value is selected from the actual distribution probability to obtain the pruning hyperparameter and the distillation hyperparameter.

10. The method according to any one of claims 1 to 9, further comprising: Fine-tuning the compressed student network; If it is determined based on the fine-tuning result that the student network does not meet the convergence condition, return to the step of selecting at least one channel to be pruned from the student network based on the pruning loss and the distillation loss until the student network meets the convergence condition.

11. The method according to claim 1, wherein: The student network is a computer vision model, which is used to perform at least one of target detection, target segmentation and target classification tasks on an input image.

12. The method according to claim 1, wherein: The student network is a language processing model, which is used to perform reasoning analysis on input voice and / or text information to complete at least one of human-computer dialogue, semantic understanding and text-based image tasks.

13. The method according to claim 1, wherein: The student network is a multimodal model, which is used to analyze input multimodal information to complete at least one of the tasks of resource recommendation and providing a basis for autonomous driving decision-making.

14. A model compression device, comprising: A screening module, used for screening at least one channel to be pruned from the student network based on pruning loss and distillation loss during the process of transferring knowledge from the teacher network to the student network; A compression module is used to compress the student network based on the at least one channel to be pruned.

15. The device according to claim 14, wherein: The screening module comprises: A determination subunit, configured to determine scaling factors of a plurality of channels in the student network based on the pruning loss and the distillation loss; The screening subunit is configured to screen out the at least one to-be-pruned channel based on the scaling factors of the multiple channels.

16. The device according to claim 15, wherein: The screening subunit is specifically used for: Channels whose scaling factors are less than a preset threshold are screened to obtain the at least one channel to be pruned.

17. The device according to claim 15, wherein: The determining subunit is specifically used for: Determine an expression for a total loss based on the pruning loss and the distillation loss; Based on an expression of the total loss, the total loss is minimized to obtain scaling factors of the multiple channels in the pruning loss.

18. The device according to claim 17, wherein: The determining subunit is specifically used for: The pruning loss and the distillation loss are weightedly summed to obtain an expression for the total loss.

19. The device according to claim 17, wherein: The determining subunit is specifically used for: The teacher network loss and the student network loss are weighted summed to obtain the distillation loss.

20. The apparatus according to claim 17, wherein the expression of the total loss includes learnable pruning hyperparameters and distillation hyperparameters, wherein: The pruning hyperparameter is used to adjust the sparsity of the student network; The distillation hyperparameters are used to adjust the accuracy of the student network.

21. The device according to claim 20, wherein: The pruning hyperparameter is the weight of the pruning loss; The distillation hyperparameter is the weight of the teacher network loss in the distillation loss.

22. The device according to claim 20, wherein: The determining subunit is further used for: Based on the prior estimation, m parameter pairs are sampled within the target value range, where m is a positive integer greater than 1, and each parameter pair includes a pruning hyperparameter sample point and a distillation hyperparameter sample point; Pruning the student network based on the m parameter pairs to obtain multiple candidate models; Determine the model performance of the candidate models corresponding to the m parameter pairs respectively; Based on the model performance of each candidate model, kernel density estimation is performed to obtain the actual distribution probability of parameter pairs; A parameter pair with the highest probability value is selected from the actual distribution probability to obtain the pruning hyperparameter and the distillation hyperparameter.

23. The apparatus according to any one of claims 14 to 22, further comprising: A fine-tuning module, used for fine-tuning the compressed student network; A convergence module is used to trigger the screening module to return to the step of screening at least one channel to be pruned from the student network based on the pruning loss and the distillation loss when it is determined based on the fine-tuning result that the student network does not meet the convergence condition, until the student network meets the convergence condition.

24. The device according to claim 14, wherein: The student network is a computer vision model, which is used to perform at least one of target detection, target segmentation and target classification tasks on an input image.

25. The device according to claim 14, wherein: The student network is a language processing model, which is used to perform reasoning analysis on input voice and / or text information to complete at least one of human-computer dialogue, semantic understanding and text-based image tasks.

26. The device according to claim 14, wherein: The student network is a multimodal model, which is used to analyze input multimodal information to complete at least one of the tasks of resource recommendation and providing a basis for autonomous driving decision-making.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images