A model compression deployment method and device, a server and a storage medium

Abstract

The application discloses a model compression deployment method and device, a server and a storage medium, relates to the technical field of model processing, and can solve the problem of how to determine the optimal compression optimization method according to the deployment requirements of a user. The specific scheme comprises the following steps: obtaining a to-be-compressed model and compression requirement information of the to-be-compressed model, calling a preset first estimation function, estimating model indexes of the to-be-compressed model, and obtaining first indexes; the first estimation function is used to predict the model indexes of the to-be-compressed model after knowledge distillation processing, and the model indexes comprise model size, parameter quantity, calculation quantity and memory quantity; determining a first processing speed of a model corresponding to the first indexes on a to-be-deployed device according to the first indexes, a preset penalty factor and a preset value; determining a compression strategy of the to-be-compressed model according to the first processing speed and a processing speed of the deployment device, and compressing the to-be-compressed model by using the determined compression strategy to obtain a target compressed model of the to-be-compressed model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model processing, in particular to a model compression deployment method and device, a server and a storage medium. BACKGROUND

[0002] Deep learning has achieved many important results in the fields of computer vision, natural language processing, speech recognition, etc. However, in practical applications, due to the large parameter quantity and calculation quantity of deep learning models, the models consume a large amount of computing resources and storage space when deployed.

[0003] The main methods for compressing models at present are pruning, distillation, quantization, etc. The compressed model can be further deployed according to the type of the device by selecting a corresponding deployment tool. Whether it is pruning, distillation, quantization or computational graph optimization, there are many methods. For the same method, different compression parameters will affect the effect of the compressed model. For example, a higher compression rate often makes the compressed model have a smaller model size and parameter quantity, and is more likely to be deployed on a resource-constrained intelligent terminal, but also brings a certain degree of precision loss. For the deployment requirements given by the user and the original model, how to select the optimal compression optimization method and compression ratio to obtain a model that meets the deployment requirements under the condition of as little precision loss as possible is an important problem to be solved. SUMMARY

[0004] The present application provides a model compression deployment method, device, server and storage medium, which can solve the problem of how to determine the optimal compression optimization method according to the deployment requirements of the user.

[0005] To achieve the above purpose, the present application adopts the following technical scheme:

[0006] In an embodiment of the present application, a model compression deployment method is provided, which comprises:

[0007] Obtaining a to-be-compressed model and compression requirement information of the to-be-compressed model, the compression requirement information comprising: input parameters of the to-be-compressed model, a to-be-deployed device of the to-be-compressed model, and a processing speed of the to-be-deployed device;

[0008] Calling a preset first estimation function to estimate the model indicators of the to-be-compressed model to obtain a first indicator; the first estimation function is used to predict the model indicators of the to-be-compressed model after knowledge distillation processing, and the model indicators comprise model size, parameter quantity, calculation quantity and memory quantity;

[0009] According to the first indicator, a preset penalty factor and a preset value, determining a first processing speed of the model corresponding to the first indicator on the to-be-deployed device;

[0010] A compression strategy of the to-be-compressed model is determined according to the first processing speed and a processing speed of the deployment device, and the to-be-compressed model is compressed by using the determined compression strategy to obtain a target compressed model of the to-be-compressed model.

[0011] In one embodiment, after determining the first processing speed of the model corresponding to the first index on the to-be-deployed device, the method further comprises:

[0012] If the first processing speed is greater than the processing speed of the deployment device, the to-be-compressed model is subjected to a knowledge distillation compression process to obtain a target compressed model of the to-be-compressed model.

[0013] If the first processing speed is less than the processing speed of the deployment device, the model index of the to-be-compressed model is estimated by sequentially calling the first estimation function and a preset second estimation function to obtain a second index, and the second estimation function is used to predict the model index of the to-be-compressed model after pruning processing.

[0014] A second processing speed of the model corresponding to the second index on the to-be-deployed device is determined according to the second index, a penalty factor, and a preset value.

[0015] If the second processing speed is greater than the processing speed of the deployment device, the to-be-compressed model is subjected to knowledge distillation and model pruning processing in sequence to obtain a target compressed model of the to-be-compressed model.

[0016] In one embodiment, after determining the second processing speed of the model corresponding to the second index on the to-be-deployed device, the method further comprises:

[0017] If the second processing speed is less than the processing speed of the deployment device, the pruning ratio of the second estimation function is increased by a preset step to obtain a second updated estimation function.

[0018] The model index of the to-be-compressed model is estimated by sequentially calling the first estimation function and the second updated estimation function to obtain a third index.

[0019] A third processing speed of a third model corresponding to the third index on the to-be-deployed device is determined according to the third index, a penalty factor, and a preset value.

[0020] The above operations are repeated until the third processing speed is greater than the processing speed of the deployment device, and the to-be-compressed model is subjected to knowledge distillation and model pruning processing according to the updated pruning ratio to obtain a target compressed model of the to-be-compressed model.

[0021] In one embodiment, before calling the preset first estimation function, the method further comprises:

[0022] The device information of the to-be-deployed device is obtained from a preset database according to the to-be-deployed device, and the device information includes device computing power, storage capacity, and device bandwidth.

[0023] obtain initial model indicators of the to-be-compressed model from a database;

[0024] determine an estimated calculation time of the to-be-compressed model on the to-be-deployed device according to the device information and the initial model indicators.

[0025] In an embodiment, after determining the estimated calculation time of the to-be-compressed model on the to-be-deployed device, the method further comprises:

[0026] determine an estimated processing speed of the to-be-compressed model on the to-be-deployed device according to the estimated calculation time, the penalty factor and the first preset value;

[0027] if the estimated processing speed is greater than the processing speed of the to-be-deployed device, determine not to perform compression processing on the to-be-compressed model.

[0028] In an embodiment, before obtaining the to-be-compressed model and the compression requirement information of the to-be-compressed model, the method further comprises:

[0029] generate a first estimation function according to the initial model indicators of the to-be-compressed model;

[0030] generate a second estimation function according to the initial model indicators and the initial pruning ratio and a preset step length.

[0031] In an embodiment, before obtaining the to-be-compressed model and the compression requirement information of the to-be-compressed model, the method further comprises:

[0032] obtain device information of a plurality of to-be-deployed devices and initial model indicators of a plurality of to-be-compressed models;

[0033] store the device information and the initial model indicators in the database in correspondence.

[0034] In a second aspect, an embodiment of the present application provides a model compression deployment apparatus, which comprises:

[0035] an obtaining module, configured to obtain a to-be-compressed model and compression requirement information of the to-be-compressed model, the compression requirement information comprising input parameters of the to-be-compressed model, a to-be-deployed device of the to-be-compressed model and a processing speed of the to-be-deployed device;

[0036] a processing module, configured to call a preset first estimation function to estimate model indicators of the to-be-compressed model to obtain first indicators; the first estimation function is used to predict model indicators of the to-be-compressed model after knowledge distillation processing, and the model indicators comprise model size, parameter quantity, calculation quantity and storage quantity;

[0037] a first determining module, configured to determine a first processing speed of a model corresponding to the first indicators on the to-be-deployed device according to the first indicators, a preset penalty factor and a preset value.

[0038] The second determining module is configured to determine a compression strategy of the to-be-compressed model according to the first processing speed and the processing speed of the deployment device, and compress the to-be-compressed model by using the determined compression strategy to obtain a target compressed model of the to-be-compressed model.

[0039] In a third aspect, a server is provided, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the model compression and deployment method in the first aspect is implemented.

[0040] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the model compression and deployment method in the first aspect is implemented.

[0041] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0042] The model compression and deployment method provided by the embodiments of the present application comprises the following steps: obtaining a to-be-compressed model and compression requirement information of the to-be-compressed model, the compression requirement information comprising input parameters of the to-be-compressed model, a to-be-deployed device of the to-be-compressed model, and a processing speed of the to-be-deployed device; calling a preset first estimation function to estimate a model index of the to-be-compressed model to obtain a first index; the first estimation function is used to predict the model index of the to-be-compressed model after knowledge distillation processing, and the model index comprises a model size, a parameter quantity, a calculation quantity, and an imitation storage quantity; determining a first processing speed of the model corresponding to the first index on the to-be-deployed device according to the first index, a preset penalty factor, and a preset value; determining a compression strategy of the to-be-compressed model according to the first processing speed and the processing speed of the deployment device, and compressing the to-be-compressed model by using the determined compression strategy to obtain a target compressed model of the to-be-compressed model. The model compression and deployment method provided by the embodiments of the present application estimates the performance of the model compressed by different strategies according to the compression requirement and the pre-designed estimation function, selects an appropriate compression method for model compression, and thus obtains a compressed model suitable for the to-be-deployed device. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of a model compression and deployment method provided by the embodiments of the present application is shown in the figure;

[0044] Figure 2 A structural diagram of a model compression and deployment device provided by the embodiments of the present application is shown in the figure;

[0045] Figure 3 An internal structure diagram of a server provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0047] Hereinafter, the terms “first” and “second” are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first” and “second” can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of “a plurality of” is two or more.

[0048] In addition, the use of “based on” or “according to” means openness and inclusiveness, because the process, step, calculation or other action “based on” or “according to” one or more conditions or values can be based on additional conditions or values in practice.

[0049] Deep learning has achieved many important results in the fields of computer vision, natural language processing, speech recognition, etc. However, in practical applications, due to the large number of parameters and large amount of calculation of deep learning models, the models consume a large amount of computing resources and storage space when deployed.

[0050] A one-key model optimization and deployment system is designed, which can compress and optimize operators for any network model structure while ensuring the reliability of model accuracy, greatly reducing the computational complexity of the model, reducing resource consumption, and speeding up the inference response time.

[0051] The main methods for model compression at present are pruning, distillation, quantization, etc. The compressed model can be further selected according to the type of the device. Whether it is pruning, distillation, quantization or computation graph optimization, there are many methods. For the same method, the value of the parameter will also affect the effect of the compressed model, for example, a higher compression rate will make the compressed model have a smaller model size and parameter quantity, and it is more likely to be deployed on a resource-limited intelligent terminal, but it will also bring a certain degree of accuracy loss.

[0052] For the deployment requirements and original model given by the user, how to choose the optimal compression optimization method and compression ratio, under the condition of as little accuracy influence as possible, to obtain a model that meets the deployment requirements, is an important problem to be solved.

[0053] At the same time, there are different reasoning tools for underlying optimization of hardware device types. It is necessary to design a one-key model optimization deployment system to automatically generate a compression deployment strategy, to establish a knowledge base of the maximum model size and computational complexity allowed by different hardware environments, to design a predictor to estimate the performance of the model compressed by different strategies, to select appropriate compression methods, compression parameters and reasoning tools, and to obtain deployable models suitable for intelligent terminals.

[0054] The model compression deployment method provided by the embodiments of the present application estimates the performance of the model compressed by different strategies according to the compression requirements and the pre-designed estimation function, selects appropriate compression methods for model compression, and thus obtains a compressed model suitable for the device to be deployed.

[0055] As shown in Figure 1 The model compression deployment method provided by the embodiments of the present application comprises:

[0056] Step 101, obtaining a model to be compressed and compression requirement information of the model to be compressed.

[0057] The compression requirement information includes input parameters of the model to be compressed, a device to be deployed for the model to be compressed, and a processing speed of the device to be deployed.

[0058] Optionally, the model to be compressed and the compression requirement information of the model to be compressed sent by a client or a terminal device can be received. The device to be deployed for the model to be compressed can be the client or the terminal device, and the embodiments of the present application do not make specific limitations thereon.

[0059] It should be noted that after step 101 is performed, the device information of the device to be deployed is obtained from a pre-set database according to the device to be deployed, the device information including device computing power, storage capacity and device bandwidth; the initial model index of the model to be compressed is obtained from the database; and the estimated computing time of the model to be compressed on the device to be deployed is determined according to the device information and the initial model index. The estimated processing speed of the model to be compressed on the device to be deployed is determined according to the estimated computing time, a penalty factor and a first pre-set value; and if the estimated processing speed is greater than the processing speed of the device to be deployed, it is determined that the model to be compressed is not compressed.

[0060] Wherein, the predicted computing time t of the to-be-compressed model on the to-be-deployed device can be predicted according to the device computing power and the device bandwidth in the device information and according to the model computing amount and the model computing amount and model memory access amount in the initial model index, and by a preset RoofLine model; at this time, the time t is obtained from the theoretical peak value, and the actual running time is much larger than t due to reasons of hardware design or implementation. Therefore, t is punished, and t*n (punishment factor) is taken as the actual running time. If 1000 / (t*n) < m, the first estimation function is called; otherwise, the compression strategy is returned as no compression. Wherein, m refers to the processing speed of the to-be-deployed device, which can be m frames per second.

[0061] It can be understood that if the preset processing speed of the to-be-compressed model on the to-be-deployed device determined before the to-be-compressed model is uncompressed is greater than the processing speed of the to-be-deployed device, it means that the to-be-compressed model can meet the deployment requirements of the to-be-deployed device, and there is no need to perform compression processing again.

[0062] In actual execution process, a knowledge base recording memory, computing power and bandwidth of different to-be-deployed devices needs to be established in advance, and the field information included in the knowledge base is shown in Table 1. At the same time, the index of the model is defined, and the index of a plurality of models is obtained in advance and stored in a preset database. The index of the model includes: model size (size), parameter quantity (para), computing amount (flops), and memory access amount (space).

[0063] By obtaining the to-be-compressed model M and the model input parameters such as image resolution, the description index of the to-be-compressed model is obtained by calling the functions provided by the ptflops library and the torchstat library, and is respectively denoted as M.si, M.para, M.flops and M.space according to the pt file of the network model ze.

[0064] Table 1 shows the table structure of the maximum model size and computing complexity allowed by the hardware environment of different to-be-deployed devices

[0065] Field name Field meaning Device Device name Memory Memory size FLOPS Floating-point operations per second (FLOPS) Space Bytes of memory accessed during computation

[0066] In addition, before step 101 is executed, a first estimation function needs to be generated in advance according to the initial model index of the to-be-compressed model; and a second estimation function is generated according to the initial model index and the initial pruning ratio and the preset step length. And obtain the device information of a plurality of to-be-deployed devices and the initial model index of a plurality of to-be-compressed models; the device information and the initial model index are stored in the database.

[0067] Wherein, the determination process of the first estimation function and the second estimation function can be:

[0068] When the compression method is knowledge distillation, the model M is the teacher model, the model S is the student model, and the first estimation function (size_kd, para_kd, flops_kd, space_kd) = Predict_kd (M, S) is designed to estimate the model size, parameter quantity, calculation quantity and memory access quantity indicators after distillation.

[0069] If the compression method is pruning, the second estimation function (size_pruning, para_pruning, flops_pruning, space_pruning) = Predict_pruning (M, a) is designed to output the estimation of the model size, parameter quantity, calculation quantity and memory access quantity of the model M after pruning under the pruning ratio a.

[0070] For example, under the pruning ratio a, the process of estimating the parameter quantity para_pruning of the model M after pruning can be:

[0071] Traverse the network structure to determine whether each layer is a layer that needs to be pruned. If so, add the parameter quantity of the layer to obtain the total parameter quantity prune_params of the pruned layer in the network.

[0072] Model the relationship between the total parameter quantity prune_params of the layer to be pruned and the input channel number and the output channel number, i.e. prune_params = f (cin, cout). For structured pruning, the pruning ratio is the ratio of the input channels and the output channels to be pruned. At this time, the remaining parameters are: f ((1-a) *cin, (1-a) *cout), so the estimation value of the network parameter quantity after structured pruning under the pruning ratio a can be estimated:

[0073] para_pruning = (M.params-prune_params) + f ((1-a) *cin, (1-a) *cout)

[0074] Wherein, the first part is the parameter quantity that cannot be pruned in the model, and the second part is the remaining parameter quantity that can be pruned when the pruning ratio is a.

[0075] Under the pruning ratio a, the model size size_pruning of the model M after pruning is estimated.

[0076] Under the pruning ratio a, the process of estimating the model calculation quantity flops_pruning of the model M after pruning can be:

[0077] Traverse the network structure, judge whether each layer is a layer that needs to be pruned, if so, accumulate the calculation amount of the layer, obtain the total calculation amount of the pruned layer in the network, denoted as flops'.

[0078] Model the relationship between the calculation amount flops' of the layer to be pruned and the input channel number cin and the output channel number cout, i.e. flops' = g(cin, cout). For structured pruning, the pruning ratio is the ratio of the input channels and output channels pruned, at this time the remaining calculation amount is: g((1- a) * cin, (1- a) * cout), so the estimated value of the calculation amount of the pruned network under the condition of pruning ratio a can be estimated:

[0079] flops_pruning = g((1- a) * cin, (1- a) * cout)

[0080] Under the condition of pruning ratio a, the process of estimating the model memory amount space_pruning of the model M after pruning can be:

[0081] Traverse the network structure, judge whether each layer is a layer that needs to be pruned, if so, accumulate the memory amount of the layer, obtain the total memory amount of the pruned layer in the network, denoted as space'.

[0082] Model the relationship between the memory amount space' of the layer to be pruned and the input channel number cin and the output channel number cout, i.e. space' = k(cin, cout). For structured pruning, the pruning ratio is the ratio of the input channels and output channels pruned, at this time the remaining memory amount is: k((1- a) * cin, (1- a) * cout), so the estimated value of the memory amount of the pruned network under the condition of pruning ratio a can be estimated:

[0083] space_pruning = k((1- a) * cin, (1- a) * cout)

[0084] If the compression method is quantization, design an estimation function (size_quan, para_quan, flops_quan, space_quan) = Predict_quan(M) to output the estimated values of the model size, parameter amount, calculation amount and memory amount of the model M after quantization.

[0085] Step 102, call a preset first estimation function to estimate the model indicators of the to-be-compressed model, and obtain first indicators.

[0086] The first estimation function is used to predict the model indicators of the to-be-compressed model after knowledge distillation processing, and the model indicators include model size, parameter amount, calculation amount and memory amount.

[0087] Step 103, determining a first processing speed of the model corresponding to the first index on the device to be deployed according to the first index, a preset penalty factor and a preset value;

[0088] Step 104, determining a compression strategy of the model to be compressed according to the first processing speed and the processing speed of the deployment device, and compressing the model to be compressed by using the determined compression strategy to obtain a target compressed model of the model to be compressed.

[0089] Optionally, the above process can be: if the first processing speed is greater than the processing speed of the deployment device, performing knowledge distillation compression processing on the model to be compressed to obtain a target compressed model of the model to be compressed;

[0090] If the first processing speed is less than the processing speed of the deployment device, the model index of the model to be compressed is estimated by sequentially calling the first estimation function and a preset second estimation function, to obtain a second index, the second estimation function is used to predict the model index of the model to be compressed after pruning processing; according to the second index, the penalty factor and the preset value, a second processing speed of the model corresponding to the second index on the device to be deployed is determined; if the second processing speed is greater than the processing speed of the deployment device, the model to be compressed is sequentially subjected to knowledge distillation and model pruning processing to obtain a target compressed model of the model to be compressed.

[0091] In addition, after determining the second processing speed of the model corresponding to the second index on the device to be deployed, if the second processing speed is less than the processing speed of the deployment device, the pruning ratio of the second estimation function is increased by a preset step to obtain a second updated estimation function; the model index of the model to be compressed is estimated by sequentially calling the first estimation function and the second updated estimation function to obtain a third index; according to the third index, the penalty factor and the preset value, a third processing speed of a third model corresponding to the third index on the device to be deployed is determined; the above operation is repeated until the third processing speed is greater than the processing speed of the deployment device, then the model to be compressed is subjected to knowledge distillation and model pruning processing according to the updated pruning ratio to obtain a target compressed model of the model to be compressed.

[0092] In actual execution process, step 1, given compression requirement information can be: a model to be compressed M, model input parameters such as image resolution, a device to be deployed E, processing speed requirement (m frames per second) of the device, strategy generation. Since quantization is often performed in deployment tools, quantization is not considered in the recommended strategy.

[0093] Step 2, querying the device computing power, storage capacity and device bandwidth of the device to be deployed in the knowledge base, denoted as F, O and S.

[0094] Get the model size M.size, parameter amount M.para, computation amount index M.flops and memory access amount M.Space of the model to be compressed. According to F, S, M.flops and M.space, the predicted computation time t (unit: ms) of the model is obtained from the RoofLine model.

[0095] At this time, the time t is obtained from the theoretical peak value, and the actual running time is much larger than t due to the reasons of hardware design or implementation. Therefore, t is punished, and t*n is taken as the actual running time. If 1000 / (t*n) < m, step 3 is called; otherwise, the compression strategy is returned as no compression.

[0096] Step 3, call Predict_kd(Teacher, Student) to estimate the indicators of the compressed model. The teacher model Teacher in the Predict_kd() function is the user's model to be compressed M, and the estimation results are recorded as M_comp.size, M_comp.para, M_comp.flops and M_comp.space.

[0097] According to the computing power F, bandwidth S and obtained model computation amount M_comp.flops and model memory access amount M_comp.space of the device, the processing time of the compressed model on the device is t (unit: ms). If 1000 / (t*n) > m, the compression strategy is output as {distillation}; otherwise, step 4 is executed.

[0098] Step 4, call Predict_kd(Teacher, Student) and Predict_pruning(M, a) functions in turn to estimate the indicators of the finally compressed model according to the compression strategy of {distillation, pruning}. The teacher model Teacher in the Predict_kd() function is the user's model to be compressed M, and the input model in the Predict_pruning() is the model after knowledge distillation. The pruning ratio is initialized as a = 0.1, and the step size step = 0.1. The final prediction results are recorded as M_comp.size, M_comp.para, M_comp.flops and M_comp.space.

[0099] From the calculation formula of M_comp.flops and M_comp.space, the time after pruning with pruning ratio a is a function of the execution time of the original model t new = q(a, t).

[0100] The time t in step 3 is brought in to obtain the processing time of the device after pruning t new (unit: ms). If 1000 / (tnew If n > m, the output compression strategy is {distillation, pruning (pruning ratio a)}. Otherwise, update the pruning ratio a = a + step, and re-execute step 4.

[0101] Step 5, according to different deployment device types, provide appropriate deployment tools, and send the deployment tools and the target compression model to the to-be-deployed device, so that the deployment model obtains more optimal inference performance for specific devices: for Nvidia series devices, use tensorRT for inference acceleration; for Android end deployment, use TVM for inference acceleration; and for other devices, use ONNX deployment.

[0102] The model compression deployment method provided in the embodiments of the present application estimates the performance of the model compressed by different strategies according to the compression requirements and the pre-designed estimation function, selects an appropriate compression method for model compression, and thus obtains a compression model suitable for a to-be-deployed device.

[0103] As shown in Figure 2 The embodiments of the present application provide a model compression deployment device, which comprises:

[0104] The acquisition module 11 is configured to acquire a to-be-compressed model and compression requirement information of the to-be-compressed model, wherein the compression requirement information comprises input parameters of the to-be-compressed model, a to-be-deployed device of the to-be-compressed model, and a processing speed of the to-be-deployed device.

[0105] The processing module 12 is configured to call a preset first estimation function to estimate a model index of the to-be-compressed model, to obtain a first index; the first estimation function is used to predict the model index of the to-be-compressed model after knowledge distillation processing, and the model index comprises a model size, a parameter quantity, a calculation quantity, and an in-memory quantity.

[0106] The first determination module 13 is configured to determine a first processing speed of the model corresponding to the first index on the to-be-deployed device according to the first index, a preset penalty factor, and a preset value.

[0107] The second determination module 14 is configured to determine a compression strategy of the to-be-compressed model according to the first processing speed and the processing speed of the deployment device, and compress the to-be-compressed model by using the determined compression strategy, to obtain a target compression model of the to-be-compressed model.

[0108] In one embodiment, the second determination module 14 is specifically configured to:

[0109] If the first processing speed is greater than the processing speed of the deployment device, perform knowledge distillation compression processing on the to-be-compressed model, to obtain a target compression model of the to-be-compressed model.

[0110] If the first processing speed is less than the processing speed of the deployment device, the first estimation function and a preset second estimation function are sequentially called to estimate the model index of the to-be-compressed model, and a second index is obtained, the second estimation function being used to predict the model index of the to-be-compressed model after pruning processing.

[0111] According to the second index, a penalty factor and a preset value, a second processing speed of a model corresponding to the second index on the to-be-deployed device is determined.

[0112] If the second processing speed is greater than the processing speed of the deployment device, knowledge distillation and model pruning processing are sequentially performed on the to-be-compressed model, and a target compressed model of the to-be-compressed model is obtained.

[0113] In an embodiment, the processing module 12 is further configured to:

[0114] If the second processing speed is less than the processing speed of the deployment device, the pruning ratio of the second estimation function is increased by a preset step size, and a second updated estimation function is obtained.

[0115] The first estimation function and the second updated estimation function are sequentially called to estimate the model index of the to-be-compressed model, and a third index is obtained.

[0116] According to the third index, a penalty factor and a preset value, a third processing speed of a third model corresponding to the third index on the to-be-deployed device is determined.

[0117] The above operations are repeated until the third processing speed is greater than the processing speed of the deployment device, and then knowledge distillation and model pruning processing according to the updated pruning ratio are performed on the to-be-compressed model, and a target compressed model of the to-be-compressed model is obtained.

[0118] In an embodiment, the obtaining module 11 is further configured to:

[0119] According to the to-be-deployed device, device information of the to-be-deployed device is obtained from a preset database, the device information including device computing power, storage capacity and device bandwidth.

[0120] An initial model index of the to-be-compressed model is obtained from the database.

[0121] According to the device information and the initial model index, a predicted computation time of the to-be-compressed model on the to-be-deployed device is determined.

[0122] In an embodiment, the first determining module 13 is further configured to:

[0123] According to the predicted computation time, a penalty factor and a first preset value, a predicted processing speed of the to-be-compressed model on the to-be-deployed device is determined.

[0124] If the predicted processing speed is greater than the processing speed of the to-be-deployed device, it is determined that the to-be-compressed model is not compressed.

[0125] In one embodiment, the first processing module 13 is further configured to:

[0126] According to the initial model indicators of the to-be-compressed models, a first estimation function is generated;

[0127] According to the initial model indicators and the initial pruning ratio and the preset step length, a second estimation function is generated.

[0128] In one embodiment, the obtaining module 11 is further configured to:

[0129] Obtain device information of a plurality of to-be-deployed devices and initial model indicators of a plurality of to-be-compressed models;

[0130] The device information and the initial model indicators are stored in the database in correspondence.

[0131] The malicious request identification device provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described in detail here. For specific limitations of the malicious request identification device, refer to the limitations of the malicious request identification method described above, which will not be described here.

[0132] The execution subject of the model compression and deployment method provided in the embodiment can be a server or a server cluster, which is not limited in the embodiment.

[0133] Figure 3 A schematic diagram of the internal structure of a server provided in the embodiment is shown in FIG. 1. Figure 3 As shown in the figure, the server includes a processor and a memory connected by a system bus. The processor is used to provide computing and control capabilities. The memory can include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement the steps of the model compression and deployment method provided in the above embodiments. The internal memory provides a cache running environment for the operating system and the computer program in the non-volatile storage medium.

[0134] Those skilled in the art can understand that Figure 3 The internal structure diagram of the server is shown in FIG. 1, which is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific server can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0135] In another embodiment of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the model compression deployment method according to the embodiments of the present application.

[0136] In another embodiment of the present application, a computer program product is provided, and the computer program product comprises computer instructions. When the computer instructions are executed on a server, the server performs each step of the model compression deployment method in the method flow shown in the method embodiments.

[0137] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, in whole or in part. When implemented by software, the implementation can be in the form of a computer program product, in whole or in part. The computer program product comprises one or more computer instructions. When the computer instructions are loaded and executed on a computer, the computer performs the flow or function according to the embodiments of the present application, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium, or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or data storage device such as one or more servers, data centers, etc. integrated with one or more media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0138] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0139] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A model compression deployment method, characterized in that, The method includes: Obtain the model to be compressed and its compression requirements, the compression requirements including: the input parameters of the model to be compressed, the device to which the model to be deployed, and the processing speed of the device to be deployed; A preset first estimation function is invoked to estimate the model metrics of the model to be compressed, thereby obtaining the first metric. The first estimation function is used to predict the model metrics of the model to be compressed after knowledge distillation. The model metrics include model size, number of parameters, computational cost, and memory access cost. Based on the first indicator, the preset penalty factor, and the preset value, determine the first processing speed of the model corresponding to the first indicator on the device to be deployed. Based on the first processing speed and the processing speed of the deployment device, a compression strategy for the model to be compressed is determined, and the determined compression strategy is used to compress the model to be compressed to obtain the target compressed model of the model to be compressed. The step of determining the compression strategy for the model to be compressed based on the first processing speed and the processing speed of the deployment device includes: If the first processing speed is greater than the processing speed of the deployment device, then the model to be compressed is subjected to knowledge distillation compression processing to obtain the target compressed model of the model to be compressed. If the first processing speed is less than the processing speed of the deployment device, the first estimation function and the preset second estimation function are called in sequence to estimate the model index of the model to be compressed, and the second index is obtained. The second estimation function is used to predict the model index of the model to be compressed after pruning. Based on the second indicator, the penalty factor, and the preset value, determine the second processing speed of the model corresponding to the second indicator on the device to be deployed; If the second processing speed is greater than the processing speed of the deployment device, then the model to be compressed is subjected to knowledge distillation and model pruning in sequence to obtain the target compressed model of the model to be compressed.

2. The method according to claim 1, characterized in that, After determining the second processing speed of the model corresponding to the second indicator on the device to be deployed, the method further includes: If the second processing speed is less than the processing speed of the deployment device, the pruning ratio of the second estimation function is increased by a preset step size to obtain the second updated estimation function; The first estimation function and the second update estimation function are called sequentially to estimate the model index of the model to be compressed, and a third index is obtained; Based on the third indicator, the penalty factor, and the preset value, determine the third processing speed of the third model corresponding to the third indicator on the device to be deployed. Repeat the above operations until the third processing speed is greater than the processing speed of the deployment device. Then, perform knowledge distillation on the model to be compressed and prune the model according to the updated pruning ratio to obtain the target compressed model of the model to be compressed.

3. The method according to claim 1, characterized in that, Before invoking the preset first estimation function, the method further includes: The device information of the device to be deployed is obtained from a preset database according to the device to be deployed. The device information includes: device computing power, storage capacity and device bandwidth. Obtain the initial model metrics of the model to be compressed from the database; Based on the device information and the initial model metrics, the estimated computation time of the model to be compressed on the device to be deployed is determined.

4. The method according to claim 3, characterized in that, After determining the estimated computation time of the model to be compressed on the device to be deployed, the method further includes: Based on the estimated computation time, the penalty factor, and the first preset value, the estimated processing speed of the model to be compressed on the device to be deployed is determined; If the expected processing speed is greater than the processing speed of the device to be deployed, then it is determined that the model to be compressed will not be compressed.

5. The method according to claim 3, characterized in that, Before obtaining the model to be compressed and its compression requirements, the method further includes: The first estimation function is generated based on the initial model indices of the model to be compressed; A second estimation function is generated based on the initial model indices, the initial pruning ratio, and the preset step size.

6. The method according to claim 3, characterized in that, Before obtaining the model to be compressed and its compression requirements, the method further includes: Obtain device information for multiple devices to be deployed and initial model metrics for multiple models to be compressed; The device information and the initial model indicators are stored in the database.

7. A model compression deployment device, characterized in that, The device includes: The acquisition module is used to acquire the model to be compressed and the compression requirement information of the model to be compressed. The compression requirement information includes: the input parameters of the model to be compressed, the device to be deployed for the model to be compressed, and the processing speed of the device to be deployed. The processing module is used to call a preset first estimation function to estimate the model index of the model to be compressed and obtain the first index; the first estimation function is used to predict the model index of the model to be compressed after knowledge distillation, and the model index includes model size, number of parameters, computational cost and memory access cost; The first determining module is used to determine the first processing speed of the model corresponding to the first indicator on the device to be deployed based on the first indicator, the preset penalty factor and the preset value. The second determining module is used to determine the compression strategy of the model to be compressed based on the first processing speed and the processing speed of the deployment device, and to compress the model to be compressed using the determined compression strategy to obtain the target compressed model of the model to be compressed. The second determining module is specifically used for: If the first processing speed is greater than the processing speed of the deployment device, then the model to be compressed is subjected to knowledge distillation compression processing to obtain the target compressed model of the model to be compressed. If the first processing speed is less than the processing speed of the deployment device, the first estimation function and the preset second estimation function are called in sequence to estimate the model index of the model to be compressed, and the second index is obtained. The second estimation function is used to predict the model index of the model to be compressed after pruning. Based on the second indicator, the penalty factor, and the preset value, determine the second processing speed of the model corresponding to the second indicator on the device to be deployed; If the second processing speed is greater than the processing speed of the deployment device, then the model to be compressed is subjected to knowledge distillation and model pruning in sequence to obtain the target compressed model of the model to be compressed.

8. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the model compression deployment method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the model compression deployment method according to any one of claims 1-6.

Images