Model compression method, apparatus and device, and storage medium

Abstract

Provided in the present application are a model compression method, apparatus and device, and a storage medium. The method comprises: acquiring a target model, and splitting the target model to obtain a plurality of target sub-models; for each target sub-model, generating a plurality of corresponding compression parameter groups on the basis of a plurality of preset compression algorithms and the structure of the target sub-model; using each preset compression algorithm and the plurality of corresponding compression parameter groups to perform compression processing on each target sub-model, so as to obtain a plurality of compressed sub-models corresponding to each target sub-model; for each target sub-model, sending the plurality of corresponding compressed sub-models and sub-model input data to a model test device, such that the model test device performs an inference performance test on each compressed sub-model, so as to obtain a sub-model performance test result; receiving each sub-model performance test result sent by the model test device; and on the basis of each sub-model performance test result, screening out target compressed sub-models, and on the basis of the target compressed sub-models, generating a target compressed model.

Description

Model compression method, device, equipment and storage medium Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a model compression method, apparatus, device and storage medium. Background Art

[0002] Deep learning models, especially neural network models, are characterized by high computational complexity, large number of parameters, large memory usage, and high model accuracy. When deploying deep learning models in application scenarios that require real-time inference output, it's important to focus on the model's inference performance.

[0003] At present, in order to improve the reasoning performance of the model, accelerated computing equipment is used to process the model reasoning process, and model compression algorithms are used to compress the model, and then the compressed model is used for reasoning to improve the reasoning performance.

[0004] However, the model inference process varies significantly between different accelerated computing devices. Model compression algorithms are used to improve model inference performance by reducing the amount of computation and parameters required. However, these factors alone are not the sole determinants of model inference performance, and therefore cannot effectively improve model inference performance.

[0005] Summary of the Invention

[0006] This application provides a model compression method to solve the technical problem that makes it impossible to effectively improve the reasoning performance of the model.

[0007] In a first aspect, the present application provides a model compression method, which is applied to a model compression device, and includes:

[0008] Obtaining a target model to be compressed, and splitting the target model to obtain multiple target sub-models to be compressed;

[0009] For each target sub-model, generating corresponding multiple compression parameter groups according to multiple preset compression algorithms and the target sub-model structure;

[0010] Using each preset compression algorithm and corresponding multiple compression parameter groups to compress each of the target sub-models respectively, so as to obtain multiple compressed sub-models corresponding to each target sub-model;

[0011] For each target sub-model, sending the corresponding multiple compressed sub-models and sub-model input data to a model testing device, so that the model testing device uses the sub-model input data to perform an inference performance test on each of the compressed sub-models to obtain a sub-model performance test result;

[0012] receiving performance test results of each sub-model sent by the model testing device;

[0013] A target compression sub-model is screened out according to the performance test results of each sub-model, and a target compression model is generated based on the target compression sub-model.

[0014] In a second aspect, the present application provides a model compression device, which is located in a model compression device and includes:

[0015] A sub-model splitting module is used to obtain a target model to be compressed and split the target model to obtain multiple target sub-models to be compressed;

[0016] A compression parameter generation module, configured to generate corresponding compression parameter groups for each target sub-model according to a plurality of preset compression algorithms and the target sub-model structure;

[0017] A sub-model compression module is used to compress each of the target sub-models using each preset compression algorithm and corresponding multiple compression parameter groups to obtain multiple compressed sub-models corresponding to each target sub-model;

[0018] a sending module, configured to send, for each target sub-model, the corresponding plurality of compressed sub-models and sub-model input data to a model testing device, so that the model testing device performs an inference performance test on each of the compressed sub-models using the sub-model input data to obtain a sub-model performance test result;

[0019] A receiving module, configured to receive the performance test results of each sub-model sent by the model testing device;

[0020] The model generation module is used to screen out the target compression sub-model according to the performance test results of each sub-model, and generate a target compression model based on the target compression sub-model.

[0021] In a third aspect, the present application provides a model compression device, comprising: a processor, and a memory and a transceiver communicatively connected to the processor;

[0022] The memory stores computer-executable instructions; the transceiver is used to transmit and receive data;

[0023] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0025] The model compression method, apparatus, device and storage medium provided in the present application obtain a target model to be compressed and split the target model to obtain multiple target sub-models to be compressed; for each target sub-model, generate corresponding multiple compression parameter groups according to multiple preset compression algorithms and the target sub-model structure; use each preset compression algorithm and the corresponding multiple compression parameter groups to compress each target sub-model respectively to obtain multiple compressed sub-models corresponding to each target sub-model; for each target sub-model, send the corresponding multiple compressed sub-models and sub-model input data to a model testing device, so that the model testing device uses the sub-model input data to perform inference performance testing on each compressed sub-model to obtain a sub-model performance test result; receive the performance test results of each sub-model sent by the model testing device; filter out the target compression sub-model according to the performance test results of each sub-model, and generate a target compression model based on the target compression sub-model. After the target model is split, each target sub-model is compressed based on each preset compression algorithm and the generated multiple compression parameter groups, and performance testing is performed online in the model testing device based on each compressed sub-model. Therefore, based on the performance test results, the target compression sub-model with the best test performance during the inference process performed by the model testing device can be selected. Then, the target compression model generated based on the target compression sub-model also has the best performance test results. Then, the target compression model with the best performance test results is used to perform the inference process on the same computing device as the model testing device, which can also obtain the best inference performance and effectively improve the inference performance of the model. In addition, it is possible to truly screen out the compression model that has the greatest improvement in the model inference performance, and effectively avoid the problem of large differences in the inference process of different computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] FIG1 is a network architecture diagram of a model compression method provided by one embodiment of the present application;

[0028] FIG2 is a schematic diagram of a flow chart of a model compression method provided in an embodiment of the present application;

[0029] FIG3 is a flow chart of a model compression method provided in another embodiment of the present application;

[0030] FIG4 is a schematic diagram showing the principle of a model compression method provided by another embodiment of the present application;

[0031] FIG5 is a schematic structural diagram of a model compression device provided in one embodiment of the present application;

[0032] FIG6 is a schematic structural diagram of a model compression device provided in an embodiment of the present application.

[0033] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0035] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail.

[0036] Currently, there are two main approaches to improving model inference performance. The first is to leverage the high performance and low power consumption of accelerated computing devices such as GPUs, NPUs, TPUs, and DSPs to process model inference. However, each accelerated computing device has design differences, and the performance of different accelerated computing devices performing the same deep learning model inference process can vary significantly. Furthermore, due to differences in the network structure of deep learning models, the performance of the same accelerated computing device performing the inference process of different deep learning models can also vary significantly.

[0037] The second method is to compress the model using model compression algorithms such as model pruning, quantization, and knowledge distillation, and then use the compressed model for inference to improve inference performance. However, using a model compression algorithm to compress the model improves model inference performance by reducing the amount of computation and parameters in the model. For general computing devices such as CPUs, reducing the amount of computation and parameters in the model can effectively improve model inference performance. However, for accelerated computing devices, the amount of computation and parameters in the model are not the only factors that determine the model inference performance. Therefore, when the accelerated computing device performs the model inference process, the inference performance is not effectively improved. Many compression algorithms may even reduce the performance of the model inference process.

[0038] Therefore, when facing technical problems in the existing technology, in order to screen out compression models that can truly effectively improve the model reasoning performance and avoid the problem of large differences in the reasoning process of different computing devices, after obtaining the target model, the target model can be actually compressed based on a variety of preset compression algorithms. Moreover, after the compression process, the reasoning process of each compressed target model can be executed on a model test device that is the same computing device as the accelerated computing device, or a model test device with an accelerated computing device, to test the performance of the reasoning process, obtain various performance test results, and select a compression algorithm that effectively improves the reasoning performance based on the performance test results to compress the target model.

[0039] Moreover, since the network structure of the target model is large, in order to efficiently compress the target model and effectively improve the reasoning performance, the target model can be split into multiple target sub-models. For each target sub-model, a compression algorithm that effectively improves the reasoning performance is screened out to compress the target sub-model to obtain a target compressed sub-model, and then a target compressed model can be generated based on the target compressed sub-model.

[0040] FIG1 is a network architecture diagram of a model compression method provided by an embodiment of the present application. As shown in FIG1 , the network architecture corresponding to the model compression method provided by this embodiment may include a model compression device 11 and a model testing device 12. The model compression device 11 is in communication with the model testing device 12. Optionally, as shown in FIG1 , the model compression device 11 and the model testing device 12 may communicate using the Remote Procedure Call protocol (RPC). The model compression device 11 includes a model compression program, which is used to execute the method in the model compression link of the present application. The model testing device includes a model testing program, which is used to execute the method in the model testing link of the present application. As shown in FIG1 , the model compression device 11 may include a compression algorithm library, a transceiver, a sub-model splitting module, a compression parameter generation module, a sub-model compression module, and a model generation module. The transceiver includes a sending module and a receiving module. The model testing device may include: a transceiver, a parsing module, a compilation module, an execution module, and a statistics module.

[0041] Specifically, a compression algorithm library stores multiple preset compression algorithms. The sub-model splitting module splits the target model to obtain multiple target sub-models to be compressed. The compression parameter generation module retrieves multiple preset compression algorithms from the compression algorithm library and, for each target sub-model, generates multiple corresponding compression parameter sets based on the preset compression algorithms and the target sub-model structure. The sub-model compression module compresses each target sub-model using each preset compression algorithm and the corresponding multiple compression parameter sets to obtain multiple compressed sub-models corresponding to each target sub-model. The sending module sends the corresponding multiple compressed sub-models and sub-model input data for each target sub-model to the model testing device. The model testing device 12 uses the parsing module to parse the multiple compressed sub-models, and the compilation module compiles the multiple parsed compressed sub-models. The execution module uses the sub-model input data to execute each compressed sub-model, causing each compressed sub-model to undergo an inference process. The statistics module collects statistics on the performance test results after inference. The statistical sub-model performance test results are then sent to the model compression device via a transceiver. The model compression device uses a model generation module to screen out a target compression sub-model according to the performance test results of each sub-model, and generates a target compression model based on the target compression sub-model.

[0042] FIG2 is a flow chart of a model compression method provided in an embodiment of the present application. As shown in FIG2 , the execution subject of this embodiment is a model compression device, which is located in a model compression device. The model compression method provided in this embodiment includes the following steps:

[0043] Step 201: obtain a target model to be compressed, and split the target model to obtain multiple target sub-models to be compressed.

[0044] The target model is the model to be compressed. The target model can be a deep learning model, such as a neural network model. The specific type of neural network model is not limited, and can include a deep neural network model, a convolutional neural network model, a recurrent neural network model, a long short-term memory network, and the like.

[0045] In this embodiment, after obtaining the target model, the network structure of the target model can be determined, and the target model can be split based on the functional modules (blocks) in the network structure of the target model, with each split part being a target sub-model. Alternatively, the target model can be split in other ways, which are not limited in this embodiment.

[0046] The target sub-model may include at least one operator.

[0047] Step 202 : For each target sub-model, generate corresponding multiple compression parameter groups according to multiple preset compression algorithms and target sub-model structures.

[0048] Among them, the preset compression algorithm is a compression algorithm pre-stored in the compression algorithm library, such as: model pruning algorithm, quantization algorithm, knowledge distillation algorithm, etc.

[0049] For each compression algorithm, there is a corresponding compression parameter type. For example, for model pruning algorithms, it can include weight pruning algorithms, neuron pruning algorithms, convolution pruning algorithms, and layer pruning algorithms. The compression parameters corresponding to various pruning algorithms are pruned connection weights, pruned redundant neurons, pruned unimportant convolution kernels, and pruned unimportant network layers. For quantization algorithms, the corresponding compression parameter is to reduce the number of bits representing model parameters. Different compression algorithms have different corresponding compression parameter values, and the structure of the target sub-model is different, and the corresponding compression effects are also different.

[0050] Therefore, in this embodiment, for each target sub-model, in order to achieve all possible forms of compression, according to each preset compression algorithm and the structure of the target sub-model, at least one operator contained in the structure of the target sub-model can be traversed and valued within the compression parameter value range according to the compression parameter type of each preset compression algorithm to generate multiple compression parameter groups.

[0051] For example, if the preset compression algorithm is a quantization algorithm and the original number of bits used to represent the model parameters is 32 bits, then the compression parameter corresponding to the quantization algorithm has a value range of [16 bits, 8 bits, 4 bits] for reducing the number of bits used to represent the model parameters. The target sub-model structure includes two operators, each with one model parameter. Six compression parameter groups are formed, namely: [16 bits, 8 bits], [16 bits, 4 bits], [8 bits, 16 bits], [8 bits, 4 bits], [4 bits, 8 bits], and [4 bits, 16 bits].

[0052] In step 203 , each target sub-model is compressed using each preset compression algorithm and corresponding multiple compression parameter groups to obtain multiple compressed sub-models corresponding to each target sub-model.

[0053] In this embodiment, there are multiple preset compression algorithms for each target sub-model, and there are multiple compression parameter groups for each preset compression algorithm. Therefore, in order to achieve all possible forms of compression, three traversals are required to achieve various compressions for each target sub-model.

[0054] Specifically, we first traverse multiple compression parameter groups for a preset compression algorithm within a target sub-model. Each compression parameter group within each preset compression algorithm compresses the target sub-model once, obtaining a number of compressed sub-models that matches the number of compression parameter groups. We then traverse the preset compression algorithms for the target sub-model again. Finally, we traverse all target sub-models again. Assuming that the number of compression parameter groups corresponding to each preset compression algorithm is N, the number of preset compression algorithms is M, and the number of target sub-models is L, then the number of target sub-models obtained is M*N*L.

[0055] In step 204 , for each target sub-model, the corresponding multiple compressed sub-models and sub-model input data are sent to the model testing device, so that the model testing device uses the sub-model input data to perform inference performance testing on each compressed sub-model to obtain the sub-model performance test results.

[0056] The sub-model input data is the data input into the compressed sub-model during the sub-model inference process. The sub-model input data can be determined based on the function of each sub-model after splitting the target model.

[0057] In this embodiment, in order to improve the testing efficiency of the model testing device, after a target sub-model is compressed using a preset compression model and a compression parameter group to obtain a corresponding compressed sub-model, the compressed sub-model and the sub-model input data can be sent to the model testing device. Alternatively, after a target sub-model is compressed using a preset compression and all compression parameter groups to obtain multiple corresponding compressed sub-models, the multiple corresponding compressed sub-models and the model input data are collectively sent to the model testing device. Alternatively, after a target sub-model is compressed using all preset compressions and all compression parameter groups to obtain multiple corresponding compressed sub-models, the multiple corresponding compressed sub-models and the model input data are collectively sent to the model testing device. This is not limited in this embodiment.

[0058] After receiving the compressed sub-model and sub-model input data, the model testing device parses and compiles the compressed sub-model. After compilation is complete, the inference performance test of the compressed sub-model is performed. Specifically, the sub-model input data is input into the compressed sub-model, the inference process of the compressed sub-model is executed, and the performance after the inference execution is monitored to obtain the sub-model performance test results.

[0059] The sub-model performance test result is the performance test result when inferring the compressed sub-model. It can be expressed as the execution time of each operator included in the compressed sub-model, or as other performance parameters, which are not limited in this embodiment.

[0060] Step 205: Receive the performance test results of each sub-model sent by the model testing device.

[0061] In this embodiment, the model testing device may immediately send the sub-model performance test results corresponding to each compressed sub-model to the model compression device after the test of that compressed sub-model is completed. Alternatively, after multiple compressed sub-model tests are completed, the sub-model performance test results of the compressed sub-models may be collectively sent to the model compression device, although this embodiment is not limited thereto.

[0062] Step 206 , selecting a target compression sub-model according to the performance test results of each sub-model, and generating a target compression model based on the target compression sub-model.

[0063] The target compression sub-model is the selected compression sub-model, and the target compression model is the compression model corresponding to the final target model.

[0064] In this embodiment, the execution time of each operator in the sub-model performance test results can be summed to obtain the sub-model execution time. For each target sub-model, multiple sub-model performance test results are sorted based on the sub-model execution time, and the sub-model performance test result with the shortest sub-model execution time is selected as the optimal sub-model performance test result. The compressed sub-model with the optimal sub-model performance test result is then determined as the target compressed sub-model.

[0065] Then, multiple target compression sub-models are spliced ​​together according to the original network structure of the target model to form a target compression sub-model.

[0066] The model compression method provided in this embodiment obtains a target model to be compressed and splits the target model to obtain multiple target sub-models to be compressed; for each target sub-model, generates corresponding multiple compression parameter groups according to multiple preset compression algorithms and target sub-model structures; uses each preset compression algorithm and the corresponding multiple compression parameter groups to compress each target sub-model respectively to obtain multiple compressed sub-models corresponding to each target sub-model; for each target sub-model, sends the corresponding multiple compressed sub-models and sub-model input data to a model testing device, so that the model testing device uses the sub-model input data to perform inference performance testing on each compressed sub-model to obtain sub-model performance test results; receives the performance test results of each sub-model sent by the model testing device; filters out the target compression sub-model according to the performance test results of each sub-model, and generates a target compression model based on the target compression sub-model. After the target model is split, each target sub-model is compressed based on each preset compression algorithm and the generated multiple compression parameter groups, and performance testing is performed online in the model testing device based on each compressed sub-model. Therefore, based on the performance test results, the target compression sub-model with the best test performance during the inference process performed by the model testing device can be selected. Then, the target compression model generated based on the target compression sub-model also has the best performance test results. Then, the target compression model with the best performance test results is used to perform the inference process on the same computing device as the model testing device, which can also obtain the best inference performance and effectively improve the inference performance of the model. In addition, it is possible to truly screen out the compression model that has the greatest improvement in the model inference performance, and effectively avoid the problem of large differences in the inference process of different computing devices.

[0067] As an optional implementation, based on the above embodiment, for each target sub-model, corresponding multiple compression parameter groups are generated according to multiple preset compression algorithms and target sub-model structures, including the following technical solutions:

[0068] For each target sub-model, the compression parameter type and value range corresponding to each preset compression algorithm are determined, and at least one target operator in the target sub-model structure is determined. For each preset compression algorithm corresponding to each target sub-model, the compression parameter values ​​of each target operator are traversed according to the corresponding compression parameter type and value range to generate multiple compression parameter groups, each of which includes the compression parameter values ​​of each target operator.

[0069] In this embodiment, the compression algorithm library stores multiple preset compression algorithms, and for each preset compression algorithm, the corresponding compression parameter type and value range are also associated and stored. For example, a mapping relationship can be established between each preset compression algorithm identifier, compression parameter type identifier, and corresponding value range, and this mapping relationship can be stored in the compression algorithm library in the form of a data table. Then, when determining the compression parameter type and value range corresponding to each preset compression algorithm for each target sub-model, the compression parameter type and value range corresponding to each preset compression algorithm can be searched based on this data table.

[0070] The compression parameter type of each preset compression algorithm may be one or more, and the value range may be multiple discrete values.

[0071] Furthermore, in this embodiment, since the target sub-model is composed of at least one target operator, compressing the target sub-model is equivalent to compressing each target operator. Therefore, the number of target operators in each target sub-model structure can be determined.

[0072] After determining the compression parameter type and value range corresponding to the preset compression algorithm, as well as the target operators in the target sub-model, it is necessary to determine each possible compression parameter group for each target sub-model and each preset compression algorithm. Specifically, when determining each possible compression parameter group, the compression parameter values ​​of each target operator are traversed according to the corresponding compression parameter type and value range. Each possible value of the model parameters included in all traversed operators forms a compression parameter group. The compression parameter values ​​included in this compression parameter group are the sequential values ​​of the model parameters included in all operators in the target sub-model.

[0073] The model compression method provided in this embodiment generates corresponding multiple compression parameter groups for each target sub-model according to multiple preset compression algorithms and target sub-model structures. For each target sub-model, the compression parameter type and value range corresponding to each preset compression algorithm are determined, and at least one target operator in the target sub-model structure is determined. For each preset compression algorithm corresponding to each target sub-model, the compression parameter values ​​of each target operator are traversed according to the corresponding compression parameter type and value range to generate multiple compression parameter groups, each of which includes the compression parameter values ​​of each target operator. Since the multiple compression parameter groups corresponding to the target sub-model structure are determined by traversing the compression parameter values ​​of each target operator according to the compression parameter type and value range corresponding to the preset compression algorithm, all possible compression parameter groups can be obtained, and then all possible compression parameter groups and preset compression algorithms can be used to achieve various possible forms of compression for the target sub-model, so that all-round compression coverage is achieved when compressing the target sub-model.

[0074] As an optional implementation, based on any of the above embodiments, the target compression sub-model is selected according to the performance test results of each sub-model, including the following solutions:

[0075] The optimal sub-model performance test result is determined from the performance test results of each sub-model. In response to the compression sub-model corresponding to the optimal sub-model performance test result meeting the compression accuracy requirement, the compression sub-model corresponding to the optimal sub-model performance test result is determined as the target compression sub-model.

[0076] In this embodiment, the performance test parameter values ​​of the sub-model performance test results that can represent the performance test results can be sorted, and the optimal performance test parameter value can be selected based on the performance test parameter values. The sub-model performance test result with the optimal performance test parameter value is determined as the optimal performance test result.

[0077] Exemplarily, the performance test parameter characterizing the performance test result is the sub-model execution time, and the sub-model performance test result with the shortest sub-model execution time can be determined as the optimal performance test result.

[0078] After selecting the optimal performance test result, determine whether the compression sub-model corresponding to the optimal sub-model performance test result meets the compression accuracy requirements. If it is determined that the compression accuracy requirements are met, the compression sub-model corresponding to the optimal sub-model performance test result is determined as the target compression sub-model. If it is determined that the compression accuracy requirements are not met, determine whether the sub-optimal performance test result meets the compression accuracy requirements. If it still does not meet the requirements, continue to determine whether the next sub-optimal performance test result meets the compression accuracy requirements until it is determined that a certain sub-optimal performance test result meets the compression accuracy requirements. Then, determine the compression sub-model corresponding to the sub-optimal performance test result that meets the compression accuracy requirements as the target compression sub-model.

[0079] Among them, when judging whether the compression sub-model corresponding to the optimal sub-model performance test result meets the compression accuracy requirement, for example, it is judged whether the similarity between the output data of the compression sub-model corresponding to the optimal sub-model performance test result and the output data of the target sub-model corresponding to the compression sub-model is greater than a preset similarity threshold. If so, it is determined that the compression accuracy requirement is met; if not, it is determined that the compression accuracy requirement is not met.

[0080] In this embodiment, after each target compression sub-model is determined, the network structure of the target model before splitting can be obtained, and each target compression sub-model is spliced ​​according to the network structure before splitting to obtain the target compression model.

[0081] The model compression method provided in this embodiment determines the optimal sub-model performance test result from the performance test results of each sub-model when screening out the target compression sub-model based on the performance test results of each sub-model; in response to the compression sub-model corresponding to the optimal sub-model performance test result meeting the compression accuracy requirement, the compression sub-model corresponding to the optimal sub-model performance test result is determined as the target compression sub-model. Since, when determining the target compression sub-model, not only the sub-model performance test result is required to be the optimal sub-model performance test result, but also the compression sub-model corresponding to the optimal sub-model performance test result is required to meet the compression accuracy requirement, the determined target compression sub-model can effectively improve the inference performance and ensure the accuracy of the inference result.

[0082] As an optional implementation, based on any of the above embodiments, after compressing each target sub-model using each preset compression algorithm and corresponding multiple compression parameter groups to obtain multiple compressed sub-models corresponding to each target sub-model, the following solution is also included:

[0083] A preset performance prediction model is used to perform performance prediction on the multiple compression sub-models corresponding to each target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models. For each target sub-model, multiple candidate compression sub-models are screened from the multiple compression sub-models based on the corresponding multiple sub-model performance prediction results.

[0084] The preset performance prediction model may be a deep learning model. The preset performance test model is a model that has been trained in the performance prediction of a compressed sub-model of other models.

[0085] In this embodiment, the characteristic data of each compression sub-model is input into a preset performance prediction model, and the preset performance prediction model predicts the performance of the compression sub-model based on the characteristic data of the compression sub-model, determines and outputs the sub-model performance prediction result.

[0086] The characteristic data of the compression sub-model may include: the type of operators included in the compression sub-model, the type and shape of operator input data, attribute parameters of the operator, etc.

[0087] In this embodiment, after obtaining the sub-model performance prediction results corresponding to multiple compression sub-models, multiple compression sub-models with better sub-model performance prediction results can be screened out from the multiple compression sub-models corresponding to each target sub-model based on the sub-model performance prediction results as candidate compression sub-models.

[0088] The sub-model performance prediction result may include: the execution time of each operator in the compressed sub-model.

[0089] Then, when selecting candidate compression sub-models from multiple compression sub-models corresponding to each target sub-model based on the sub-model performance prediction results, the sub-model execution time corresponding to each compression sub-model can be calculated based on the sub-model performance prediction results, and the sub-model execution times can be sorted, and multiple compression sub-models with shorter sub-model execution times can be determined as candidate compression sub-models.

[0090] Accordingly, for each target sub-model, the corresponding multiple compressed sub-models and sub-model input data are sent to the model testing device, so that the model testing device uses the sub-model input data to perform inference performance testing on each compressed sub-model to obtain sub-model performance test results, including the following schemes:

[0091] For each target sub-model, the corresponding multiple candidate compression sub-models and sub-model input data are sent to the model testing device, so that the model testing device uses the sub-model input data to perform inference performance testing on each candidate compression sub-model to obtain the sub-model performance test results.

[0092] In this embodiment, since it takes a lot of time to perform inference performance testing on each compression sub-model, in order to effectively reduce the number of compression sub-models to be tested for inference performance, only multiple candidate compression sub-models and sub-model input data in each target sub-model can be sent to the model testing device, so that the model testing device can perform inference performance testing on each candidate compression sub-model.

[0093] Among them, the way in which the model compression device sends candidate sub-models and sub-model input data is similar to the way in which the compressed sub-model and sub-model input data are sent, and will not be repeated here.

[0094] The model compression method provided in this embodiment, after compressing each target sub-model using each preset compression algorithm and corresponding multiple compression parameter groups to obtain multiple compressed sub-models corresponding to each target sub-model, further includes: using a preset performance prediction model to perform performance prediction on the multiple compressed sub-models corresponding to each target sub-model to obtain sub-model performance prediction results corresponding to the multiple compressed sub-models; and for each target sub-model, screening multiple candidate compressed sub-models from the multiple compressed sub-models based on the corresponding multiple sub-model performance prediction results. Accordingly, when the corresponding multiple compressed sub-models and sub-model input data for each target sub-model are sent to the model testing device, the corresponding multiple candidate compressed sub-models and sub-model input data for each target sub-model are sent to the model testing device. Since the preset performance prediction model is used to perform performance prediction on the compressed sub-models corresponding to all target sub-models, and based on the performance prediction results, only the candidate compressed sub-models with better performance prediction results are sent to the model testing device for testing, the number of compressed sub-models inferred in the model testing device can be effectively reduced, thereby more quickly determining the optimal compressed sub-model for each target sub-model.

[0095] As an optional implementation, based on the above embodiment, the preset performance prediction model is a deep learning model.

[0096] Specifically, it may be a deep neural network model, a convolutional neural network model, etc., which is not limited in this embodiment.

[0097] Accordingly, after receiving the sub-model performance test result corresponding to the current target sub-model sent by the model testing device, the following scheme is also included:

[0098] Obtain multiple tuning samples labeled with sub-model performance test results. The tuning samples include feature data of the candidate compression sub-model; the feature data of the candidate compression sub-model includes the operator type, the type and shape of the operator input data, and the attribute parameters of the operator. Use each tuning sample to tune the preset performance prediction model to obtain a tuned preset performance prediction model.

[0099] In this embodiment, after performing performance testing on each candidate compression sub-model, the model testing device obtains a sub-model performance test result. Furthermore, for each candidate compression sub-model, the preset performance prediction model has a corresponding sub-model predicted performance result. Therefore, the parameters of the preset performance prediction model can be reversely calculated using the characteristic data of each candidate compression sub-model, the corresponding sub-model performance test result, and the predicted performance prediction result to optimize the preset performance prediction model.

[0100] Specifically, the sub-model performance test results include the execution time of each operator in the compression sub-model. Therefore, the execution time of each operator can be formed into a vector as a label for the preset performance prediction model. Together with the feature data corresponding to the candidate compression sub-model, this vector forms a tuning sample for the preset performance prediction model. Multiple tuning samples are input into the preset performance prediction model, and the parameters of the preset performance prediction model are reversely calculated to obtain the tuned parameters. The preset performance prediction model corresponding to the tuned parameters is then determined as the tuned preset performance prediction model.

[0101] It is understandable that in the feature data of the candidate compression sub-model, the attribute parameters of different types of operators may be different. For example, for a convolution operator, the attribute parameters may include: step size, convolution kernel size, etc.

[0102] The model compression method provided in this embodiment has a preset performance prediction model that is a deep learning model. After receiving the sub-model performance test results corresponding to the current target sub-model sent by the model testing device, it also includes: obtaining multiple tuning samples labeled with the sub-model performance test results, the tuning samples including: feature data of the candidate compression sub-model; the feature data of the candidate compression sub-model including: the operator type included in the candidate compression sub-model, the type and shape of the operator input data, and the attribute parameters of the operator; using each tuning sample to tune the preset performance prediction model to obtain a tuned preset performance prediction model. Since the actual sub-model performance test results corresponding to multiple candidate compression sub-models can be determined in the model testing device, the preset performance prediction model can be tuned based on the actual sub-model performance test results and the feature data of the candidate compression sub-model, so as to make full preparations for using the tuned preset performance prediction model to perform performance prediction on the next compression sub-model.

[0103] As an optional implementation, based on the above embodiment, a preset performance prediction model is used to perform performance prediction on multiple compression sub-models corresponding to each target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models, including the following technical solutions:

[0104] The tuned preset performance prediction model is used to perform performance prediction on multiple compression sub-models corresponding to the next target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models.

[0105] In this embodiment, after the preset performance prediction model is tuned using the tuning sample corresponding to the current target sub-model, the tuned preset performance prediction model is used to perform performance prediction on multiple compression sub-models corresponding to the next target sub-model. Based on the sub-model performance prediction results, a candidate compression sub-model is screened from the multiple compression sub-models corresponding to the next target sub-model and sent to a model testing device. After the model testing device determines the sub-model performance test results of the candidate compression sub-model, the preset performance prediction model is tuned based on the sub-model performance test results of the candidate compression sub-model. This process is repeated until the preset performance prediction model is fully tuned using the sub-model performance test results of the candidate compression sub-model corresponding to the last target sub-model.

[0106] The model compression method provided in this embodiment uses a preset performance prediction model to perform performance prediction on multiple compression sub-models corresponding to each target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models. Then, the tuned preset performance prediction model is used to perform performance prediction on multiple compression sub-models corresponding to the next target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models. Since the preset performance prediction model is tuned after completing the performance prediction of a compression sub-model corresponding to a target sub-model, the tuned preset performance prediction model is used to continue to perform performance prediction on multiple compression sub-models of the next target sub-model. Therefore, the preset performance prediction model can be more suitable for predicting compression sub-models, and can more accurately predict the inference performance of candidate compression sub-models on the model testing device.

[0107] As an optional implementation, based on any of the above embodiments, before using a preset performance prediction model to perform performance prediction on multiple compression sub-models corresponding to each target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models, the following technical solutions are also included:

[0108] The target model and model input data are sent to a model testing device, which uses the sub-model input data to perform inference performance testing on multiple operators in the target model to obtain operator performance test results. The performance test results for each operator sent by the model testing device are received. Multiple update samples are obtained, labeled with the operator performance test results. The update samples include: operator feature data; the operator feature data includes: operator type, operator input data type and shape, and operator attribute parameters. Each update sample is used to update the initial performance prediction model to obtain a preset performance prediction model.

[0109] In this embodiment, in order to make the preset performance prediction model applicable to the operators in the target model and the accelerated computing equipment in the model testing equipment, the initial performance prediction model needs to be updated before using the preset performance prediction model to perform performance prediction on multiple compression sub-models corresponding to each target sub-model.

[0110] The initial performance prediction model is a performance prediction model that has not been updated.

[0111] In this embodiment, since the characteristic data of the compression sub-model is used as the tuning sample to tune the preset performance prediction model, the preset performance prediction model is fundamentally tuned based on the characteristic data of each operator in the compression sub-model. Therefore, when the initial performance prediction model is updated, the initial performance prediction model can be updated with the characteristic data of each operator in the target model.

[0112] Specifically, the target model and model input data are first sent to the model testing device. After parsing and compiling the target model, the model testing device performs inference performance testing on each target operator within the target model. Specifically, the model input data is fed into the target model, the target model's inference process is executed, and the performance of each operator after inference execution is monitored to obtain performance test results for each target operator.

[0113] The target operator is an operator in the target model. The operator performance test result is the performance test result corresponding to the target operator. The operator performance test result can be the execution time of the target operator.

[0114] After the model testing device determines the operator performance test results corresponding to each target operator, the operator performance test results are sent to the model compression device. The model compression device constructs an updated sample set based on the operator performance test results and uses the updated sample set to update the initial performance prediction model.

[0115] Specifically, the update sample set includes multiple update samples, each of which is labeled with an operator performance test result and also includes feature data of the corresponding target operator.

[0116] The feature data of each target operator can be obtained based on the network structure analysis of the target model. The feature data of the target operator in the update sample is similar to the feature data of the operator in the tuning sample, and will not be repeated here.

[0117] It's understandable that the difference between update samples and tuning samples is that in update samples, operator performance test results can be represented by the execution time of a target operator, and the target operator's characteristic data is also the characteristic data of a target operator. In tuning samples, however, submodel performance test results can be represented by a vector consisting of candidate compression submodels including the operator's execution time, and the characteristic data of the candidate compression submodels is the characteristic annual data of at least one target operator.

[0118] In this embodiment, when the initial performance prediction model is updated using each update sample, each update sample can be input into the initial performance prediction model, and the parameters in the initial performance prediction model can be adjusted based on each update sample to update the parameters, and the initial performance prediction model corresponding to the updated parameters can be determined as the preset performance prediction model.

[0119] The model compression method provided in this embodiment sends the target model and model input data to a model testing device before using a preset performance prediction model to perform performance prediction on multiple compressed sub-models corresponding to each target sub-model, so that the model testing device uses the sub-model input data to perform inference performance testing on multiple target operators in the target model to obtain operator performance test results; receives the performance test results of each operator sent by the model testing device; obtains multiple update samples with the operator performance test results as labels; the update samples include: feature data of the target operator; the feature data of the target operator include: operator type, type and shape of operator input data, and attribute parameters of the operator; and uses each update sample to update the initial performance prediction model to obtain a preset performance prediction model. Before using the preset performance prediction model to predict the performance of multiple compression sub-models corresponding to each target sub-model, the performance of each target operator in the target model is tested in the test model device, and multiple update samples are constructed with the operator performance test results as labels. The initial performance prediction model is updated using the multiple update samples. The obtained preset performance prediction model can be more suitable for the target model and the accelerated computing device in the model testing device. Then, when using the preset performance prediction model to predict the performance of multiple compression sub-models corresponding to each target sub-model, the accuracy of performance prediction when the multiple compression sub-models are inferred in the model testing device is further improved.

[0120] FIG3 is a flow chart of a model compression method provided in another embodiment of the present application. As shown in FIG3 , the model compression method provided in this embodiment is described as an optional complete embodiment based on any of the above embodiments. The execution subject of this embodiment is a model compression system. The model compression method provided in this embodiment includes the following steps:

[0121] In step 301 , the model testing device tests the inference performance of each operator in the target model, and the model compression device updates the initial performance prediction model based on the operator performance test results to obtain a preset performance prediction model.

[0122] Exemplarily, as shown in FIG4 , the target model 41 includes three operators, namely, the add operator (also known as the summation operator), the Relu operator (also known as the linear activation function operator), and the Conv operator (also known as the convolution operator).

[0123] In step 302 , the model compression device obtains a target model to be compressed, and splits the target model to obtain multiple target sub-models to be compressed.

[0124] Exemplarily, as shown in FIG4 , after the target model is split, two target sub-models are obtained, namely a first target sub-model 42 and a second target sub-model 43 .

[0125] In step 303, the model compression device generates a plurality of corresponding compression parameter groups for the current target sub-model according to a plurality of preset compression algorithms and the structure of the current target sub-model; and uses each preset compression algorithm and the corresponding plurality of compression parameter groups to perform compression processing on the current target sub-model respectively to obtain a plurality of compressed sub-models corresponding to the current target sub-model respectively.

[0126] Exemplarily, as shown in FIG4 , the current target sub-model is the first target sub-model. After the first target sub-model 42 is compressed by multiple preset compression algorithms, multiple compressed sub-models are obtained, which are respectively represented as compressed sub-model 1, compressed sub-model 2, compressed sub-model 3, ... .

[0127] In step 304 , the model compression device uses a preset performance prediction model to perform performance prediction on multiple compressed sub-models corresponding to the current target sub-model to obtain sub-model performance prediction results corresponding to the multiple compressed sub-models.

[0128] Exemplarily, as shown in FIG4 , a preset performance prediction model 43 is used to perform performance prediction on multiple compression sub-models corresponding to the current target sub-model. The obtained sub-model performance prediction results can be expressed as the execution time of the target operator in the sub-model, which are T1, T2, T3, ... respectively.

[0129] In step 305 , the model compression device selects multiple candidate compression sub-models from the multiple compression sub-models according to the corresponding multiple sub-model performance prediction results for each target sub-model, and sends the corresponding multiple candidate compression sub-models and sub-model input data to the model testing device.

[0130] In step 306 , the model testing device uses the current target sub-model input data to perform an inference performance test on each candidate compression sub-model to obtain a sub-model performance test result.

[0131] Exemplarily, as shown in Figure 4, the model compression device selects the compression sub-model 3 as one of the candidate compression sub-models, and the model testing device performs an inference performance test based on the sub-model input data input and the compression sub-model 3 to obtain the sub-model performance test result, represented as T3', and can also obtain the inference result, represented as output.

[0132] In step 307 , the model compression device obtains multiple tuning samples labeled with the performance test results of the candidate compression sub-model corresponding to the current target sub-model, and uses each tuning sample to tune the preset performance prediction model to obtain the tuned preset performance prediction model.

[0133] The tuning sample includes the characteristic data of the candidate compression sub-model. As shown in FIG4 , the characteristic data Opl3 and T3′ of the candidate compression sub-model in the current target sub-model are input into the preset performance prediction model, and the preset performance prediction model is tuned to obtain the tuned preset performance prediction model 44.

[0134] In step 308 , the model compression device selects a target compression sub-model based on the performance test results of each sub-model corresponding to the current target sub-model.

[0135] Step 309 , determining whether the current target sub-model is the last target sub-model, if so, executing step 310 , otherwise returning to step 303 , and repeating steps 303 - 308 .

[0136] Step 310: The model compression device generates a target compression model based on the target compression sub-model.

[0137] As shown in FIG. 4 , illustratively, the first target sub-model is replaced by a first target compression model 45 , and the second target sub-model is replaced by a second target compression model 46 .

[0138] Figure 5 is a structural diagram of a model compression device provided in an embodiment of the present application. As shown in Figure 5, the model compression device provided in this embodiment is located in a model compression device, and the model compression device 51 provided in this embodiment includes: a sub-model splitting module 51, a compression parameter generation module 52, a sub-model compression module 53, a sending module 54, a receiving module 55 and a model generation module 56.

[0139] Among them, the sub-model splitting module 51 is used to obtain the target model to be compressed and split the target model to obtain multiple target sub-models to be compressed. The compression parameter generation module 52 is used to generate corresponding multiple compression parameter groups for each target sub-model according to multiple preset compression algorithms and target sub-model structures. The sub-model compression module 53 is used to use each preset compression algorithm and the corresponding multiple compression parameter groups to compress each target sub-model respectively to obtain multiple compressed sub-models corresponding to each target sub-model. The sending module 54 is used to send the corresponding multiple compressed sub-models and sub-model input data to the model testing device for each target sub-model, so that the model testing device uses the sub-model input data to perform inference performance testing on each compressed sub-model to obtain sub-model performance test results. The receiving module 55 is used to receive the performance test results of each sub-model sent by the model testing device. The model generation module 56 is used to screen out the target compression sub-model according to the performance test results of each sub-model, and generate a target compression model based on the target compression sub-model.

[0140] Optionally, the compression parameter generating module 52 is specifically configured to:

[0141] For each target sub-model, determine the compression parameter type and value range corresponding to each preset compression algorithm, and determine at least one target operator in the target sub-model structure; for each preset compression algorithm corresponding to each target sub-model, traverse the compression parameter values ​​of each target operator according to the corresponding compression parameter type and value range to generate multiple compression parameter groups, which include the compression parameter values ​​of each target operator.

[0142] Optionally, when selecting the target compression sub-model based on the performance test results of each sub-model, the model generation module is specifically configured to:

[0143] Determine the optimal sub-model performance test result from the performance test results of each sub-model; in response to the compression sub-model corresponding to the optimal sub-model performance test result meeting the compression accuracy requirement, determine the compression sub-model corresponding to the optimal sub-model performance test result as the target compression sub-model.

[0144] Optionally, the model compression device provided in this embodiment further includes: a prediction module and a screening module.

[0145] The prediction module is used to use a preset performance prediction model to perform performance prediction on multiple compression sub-models corresponding to each target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models. The screening module is used to screen multiple candidate compression sub-models from the multiple compression sub-models based on the corresponding multiple sub-model performance prediction results for each target sub-model;

[0146] Correspondingly, the sending module 54 is specifically used to send the corresponding multiple candidate compression sub-models and sub-model input data to the model testing device for each target sub-model, so that the model testing device uses the sub-model input data to perform inference performance testing on each candidate compression sub-model to obtain the sub-model performance test results.

[0147] Optionally, the preset performance prediction model is a deep learning model.

[0148] Correspondingly, the model compression device provided in this embodiment further includes: an optimization module.

[0149] The tuning module is configured to obtain multiple tuning samples labeled with sub-model performance test results. The tuning samples include feature data of the candidate compression sub-model; the feature data of the candidate compression sub-model includes the operator type, the type and shape of the operator input data, and the attribute parameters of the operator. Each tuning sample is used to tune the preset performance prediction model to obtain a tuned preset performance prediction model.

[0150] Optionally, the prediction module is specifically used to use the tuned preset performance prediction model to perform performance prediction on multiple compression sub-models corresponding to the next target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models.

[0151] Optionally, the model compression device provided in this embodiment further includes: an updating module.

[0152] Among them, the sending module 54 is also used to send the target model and model input data to the model testing device, so that the model testing device uses the sub-model input data to perform inference performance testing on multiple target operators in the target model to obtain operator performance test results. The receiving module 55 is also used to receive the performance test results of each operator sent by the model testing device. The update module is used to obtain multiple update samples labeled with the operator performance test results; the update samples include: the characteristic data of the target operator; the characteristic data of the target operator includes: the operator type, the type and shape of the operator input data, and the attribute parameters of the operator; each update sample is used to update the initial performance prediction model to obtain a preset performance prediction model.

[0153] Figure 6 is a structural diagram of a model compression device provided in an embodiment of the present application. As shown in Figure 6, the model compression device 60 is used to implement any of the above method embodiments. The model compression device 60 of this embodiment may include: a memory 61, a processor 62 and a transceiver 63.

[0154] Memory 61 is used to store computer programs. Memory 61 may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. It may also be a USB flash drive, a mobile hard drive, a read-only memory, a magnetic disk, or an optical disk.

[0155] The processor 62 is used to execute the computer program stored in the memory to implement the method in the above embodiment. For details, please refer to the relevant description in the above method embodiment. The processor 62 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0156] Optionally, the memory 61 may be independent or integrated with the processor 62 .

[0157] When the memory 61 is a device independent of the processor 62, the model compression device 60 may further include a bus 64. The bus 64 is used to connect the memory 61 and the processor 62. The bus 64 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of presentation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0158] The transceiver 63 can be connected to the processor 62 via a bus 64. The processor 62 can control the transceiver 63. The transceiver 63 can be used to communicate with the model test equipment and transmit and receive data.

[0159] The data is data in a broad sense, which refers to all data transmitted between two devices.

[0160] The model compression device provided in this embodiment can be used to execute the above-mentioned model compression method. Its implementation method and technical effects are similar and will not be described in detail in this embodiment.

[0161] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the methods provided in the various embodiments described above.

[0162] Among them, the computer-readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a computer-readable storage medium is coupled to a processor so that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be an integral part of the processor. The processor and the computer-readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the computer-readable storage medium can also exist in a communication device as discrete components.

[0163] Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0164] The present application also provides a computer program product, comprising a computer program stored in a computer-readable storage medium. At least one processor of a device can read the computer program from the computer-readable storage medium, and at least one processor executes the computer program so that the device implements the methods provided in the various embodiments described above.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0166] The modules may be physically separate, for example, installed in different locations on a single device, or installed on different devices, or distributed across multiple network units, or distributed across multiple processors. The modules may also be integrated, for example, installed in the same device, or integrated into a set of codes. The modules may exist in the form of hardware, or in the form of software, or may be implemented in the form of software plus hardware. The present application may select some or all of the modules according to actual needs to achieve the purpose of the present embodiment.

[0167] When each module is implemented as an integrated module in the form of a software function module, it can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0168] It should be understood that, although the various steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times, and their execution order is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of sub-steps or stages of other steps.

[0169] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0170] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A model compression method, characterized in that: The method is applied to a model compression device, and the method comprises: Obtaining a target model to be compressed, and splitting the target model to obtain multiple target sub-models to be compressed; For each target sub-model, generating corresponding multiple compression parameter groups according to multiple preset compression algorithms and the target sub-model structure; Using each preset compression algorithm and corresponding multiple compression parameter groups to compress each of the target sub-models respectively, so as to obtain multiple compressed sub-models corresponding to each target sub-model; For each target sub-model, sending the corresponding multiple compressed sub-models and sub-model input data to a model testing device, so that the model testing device uses the sub-model input data to perform an inference performance test on each of the compressed sub-models to obtain a sub-model performance test result; receiving performance test results of each sub-model sent by the model testing device; A target compression sub-model is screened out according to the performance test results of each sub-model, and a target compression model is generated based on the target compression sub-model.

2. The method according to claim 1, characterized in that For each target sub-model, a plurality of corresponding compression parameter groups are generated according to a plurality of preset compression algorithms and the target sub-model structure, including: For each target sub-model, determining the compression parameter type and value range corresponding to each of the preset compression algorithms, and determining at least one target operator in the target sub-model structure; For each preset compression algorithm corresponding to each target sub-model, the compression parameter values ​​of each target operator are traversed according to the corresponding compression parameter type and value range to generate multiple compression parameter groups, which include the compression parameter values ​​of each target operator.

3. The method according to claim 1, characterized in that The step of selecting a target compression sub-model according to the performance test results of each sub-model includes: Determining the optimal sub-model performance test result from the sub-model performance test results; In response to the compression sub-model corresponding to the optimal sub-model performance test result meeting the compression accuracy requirement, the compression sub-model corresponding to the optimal sub-model performance test result is determined as the target compression sub-model.

4. The method according to any one of claims 1 to 3, characterized in that After the target sub-models are compressed using the preset compression algorithms and the corresponding multiple compression parameter groups to obtain multiple compressed sub-models corresponding to the target sub-models, the method further includes: Using a preset performance prediction model to perform performance prediction on multiple compression sub-models corresponding to each target sub-model, so as to obtain sub-model performance prediction results corresponding to the multiple compression sub-models; For each target sub-model, multiple candidate compression sub-models are screened out from the multiple compression sub-models according to the corresponding multiple sub-model performance prediction results; For each target sub-model, the corresponding multiple compressed sub-models and sub-model input data are sent to the model testing device, so that the model testing device uses the sub-model input data to perform an inference performance test on each compressed sub-model to obtain a sub-model performance test result, including: For each target sub-model, the corresponding multiple candidate compression sub-models and sub-model input data are sent to the model testing device, so that the model testing device uses the sub-model input data to perform inference performance testing on each candidate compression sub-model to obtain the sub-model performance test results.

5. The method according to claim 4, characterized in that The preset performance prediction model is a deep learning model; After receiving the sub-model performance test result corresponding to the current target sub-model sent by the model testing device, the method further includes: Acquire multiple tuning samples labeled with sub-model performance test results, the tuning samples including: feature data of the candidate compression sub-model; the feature data of the candidate compression sub-model including: operator type included in the candidate compression sub-model, type and shape of operator input data, and attribute parameters of the operator; The preset performance prediction model is tuned using each of the tuning samples to obtain a tuned preset performance prediction model.

6. The method according to claim 5, characterized in that The method of using a preset performance prediction model to perform performance prediction on multiple compression sub-models corresponding to each target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models includes: The tuned preset performance prediction model is used to perform performance prediction on multiple compression sub-models corresponding to the next target sub-model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models.

7. The method according to claim 4, characterized in that Before performing performance prediction on the multiple compression sub-models corresponding to each target sub-model using a preset performance prediction model to obtain sub-model performance prediction results corresponding to the multiple compression sub-models, the method further includes: Sending the target model and model input data to a model testing device, so that the model testing device uses the sub-model input data to perform an inference performance test on multiple target operators in the target model to obtain operator performance test results; receiving performance test results of each operator sent by the model testing device; Acquire multiple update samples labeled with operator performance test results; the update samples include: feature data of the target operator; the feature data of the target operator includes: operator type, type and shape of operator input data, and attribute parameters of the operator; The initial performance prediction model is updated using each of the update samples to obtain a preset performance prediction model.

8. A model compression device, characterized in that: The device is located in a model compression device, and comprises: A sub-model splitting module is used to obtain a target model to be compressed and split the target model to obtain multiple target sub-models to be compressed; A compression parameter generation module, configured to generate corresponding compression parameter groups for each target sub-model according to a plurality of preset compression algorithms and the target sub-model structure; A sub-model compression module is used to compress each of the target sub-models using each preset compression algorithm and corresponding multiple compression parameter groups to obtain multiple compressed sub-models corresponding to each target sub-model; a sending module, configured to send, for each target sub-model, the corresponding plurality of compressed sub-models and sub-model input data to a model testing device, so that the model testing device performs an inference performance test on each of the compressed sub-models using the sub-model input data to obtain a sub-model performance test result; A receiving module, configured to receive the performance test results of each sub-model sent by the model testing device; The model generation module is used to screen out the target compression sub-model according to the performance test results of each sub-model, and generate a target compression model based on the target compression sub-model.

9. A model compression device, characterized in that: include: a processor, and a memory and a transceiver communicatively connected to the processor; The memory stores computer-executable instructions; the transceiver is used to transmit and receive data; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images