Model compression method deployed on hardware equipment

Abstract

The invention provides a model compression method deployed on hardware equipment, and relates to the technical field of model compression. The method comprises the steps of obtaining parameter information of to-be-deployed hardware equipment, determining a first storage space of the to-be-deployed hardware equipment and hardware operation information of the to-be-deployed hardware equipment according to the parameter information of the to-be-deployed hardware equipment, and obtaining a to-be-compressed original model and a compression method, determining a compressed model according to the to-be-compressed original model and the compression method, determining a second storage space of the compressed model, and if the first storage space is not smaller than the second storage space, deploying the to-be-deployed hardware device according to the hardware operation information of the to-be-deployed hardware device and the compressed model. And determining whether to-be-deployed hardware equipment executes the compressed model or not. By adopting the technical scheme, the size of the model which can be configured by the current electronic equipment can be quickly determined, so that a user does not need to try one by one, and the configuration efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of model compression technology, and in particular to a model compression method deployed on a hardware device. Background Art

[0002] Currently, model compression technology reduces the storage and computing overhead of the model by reducing the number of model parameters and the amount of calculation, while maintaining its performance as much as possible. This technology is crucial for deploying large models to resource-constrained devices such as mobile phones or embedded devices.

[0003] However, different electronic devices are adapted to different model sizes. The current method is to try to configure models of different sizes on the electronic device, and then select the model that best suits the current electronic device.

[0004] Therefore, there is an urgent need for a model compression method deployed on hardware devices that can quickly determine the model size that the current electronic device can be configured with, thereby improving configuration efficiency without the need for users to try one by one. Summary of the invention

[0005] The present application provides a model compression method deployed on a hardware device, which can quickly determine the model size that can be configured by the current electronic device, thereby improving the configuration efficiency without the user having to try one by one. The present application provides a model compression method deployed on a hardware device, comprising: Acquire parameter information of the hardware device to be deployed, and determine a first storage space of the hardware device to be deployed and hardware operation information of the hardware device to be deployed according to the parameter information of the hardware device to be deployed; Acquire an original model to be compressed and a compression method, determine a compressed model according to the original model to be compressed and the compression method, and determine a second storage space for the compressed model; If the first storage space is not smaller than the second storage space, determining whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model.

[0006] The present application also provides a model compression device deployed on a hardware device, including: A first acquisition module, used to acquire parameter information of a hardware device to be deployed, and determine a first storage space of the hardware device to be deployed and hardware operation information of the hardware device to be deployed according to the parameter information of the hardware device to be deployed; A second acquisition module is used to acquire an original model to be compressed and a compression method, determine a compressed model according to the original model to be compressed and the compression method, and determine a second storage space for the compressed model; A determination module is used to determine whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model if the first storage space is not smaller than the second storage space.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned model compression methods deployed on a hardware device when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned model compression methods deployed on a hardware device are implemented.

[0009] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned model compression methods deployed on a hardware device.

[0010] Through this application, the parameter information of the hardware device to be deployed is first obtained, and the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are determined according to the parameter information of the hardware device to be deployed, and then the original model to be compressed and the compression method are obtained, and the compressed model is determined according to the original model to be compressed and the compression method, and the second storage space of the compressed model is determined. Then the first storage space and the second storage space are compared. If the first storage space is not less than the second storage space, it is determined whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model. By adopting this technical solution, the model size that can be configured by the current electronic device can be quickly determined, and the user does not need to try one by one, thereby improving the configuration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 It is a flowchart of a model compression method deployed on a hardware device provided by an embodiment of the present disclosure; Figure 2 It is a flowchart of a model compression method deployed on a hardware device provided by an embodiment of the present disclosure; Figure 3is a structural diagram of a hardware device prediction module provided by an embodiment of the present disclosure; Figure 4 is a structural schematic diagram of a storage space comparison provided by an embodiment of the present disclosure; Figure 5 is a structural schematic diagram of a performance prediction module provided by an embodiment of the present disclosure; Figure 6 It is a schematic diagram of a model compression process deployed on a hardware device provided by an embodiment of the present disclosure; Figure 7 It is a structural diagram of a model compression device deployed on a hardware device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0016] The embodiment of the present application provides a model compression method deployed on a hardware device, and the method is described in detail in combination with the execution flow of the model compression method deployed on the hardware device. Specifically, Figure 1 1 is a flow chart of a model compression method deployed on a hardware device provided by an embodiment of the present disclosure. The method can be executed by an electronic device. The electronic device can be exemplarily understood as a device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc. Figure 1 As shown, the method provided in this embodiment includes the following steps: S101: Acquire parameter information of a hardware device to be deployed, and determine a first storage space of the hardware device to be deployed and hardware operation information of the hardware device to be deployed according to the parameter information of the hardware device to be deployed.

[0017] In one example, the hardware device to be deployed may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, and the like. Specifically, there may be multiple hardware devices to be deployed. The parameter information of the hardware device to be deployed may be a graphics card model, a number of graphics cards, and a network bandwidth. The first storage space of the hardware device to be deployed is used to characterize the memory space that can be used by the hardware device to be deployed.

[0018] In one example, the hardware operation information of the hardware device to be deployed includes at least one of the following: The computing speed of multiple-size tensors of hardware devices, network transmission bandwidth, network transmission latency, memory size of various types of storage, GPU interconnection transmission bandwidth, and GPU interconnection transmission latency.

[0019] In one example, in a hardware device, tensor computation speed is affected by multiple factors, including hardware architecture, tensor size, data type, and complexity of computational tasks.

[0020] In one example, network transmission bandwidth refers to the amount of data that can be transmitted through the network per unit time, usually measured in bits per second. Common units also include Kbps (kilobits per second), Mbps (megabits per second), and Gbps (gigabits per second).

[0021] In one example, network transmission latency refers to the time it takes for data to travel from the sender to the receiver, usually expressed in milliseconds. Latency primarily affects the response time of data transmission, which is the time it takes for data to travel from being sent to being received.

[0022] In one example, GPU interconnect transmission bandwidth is used to achieve high-bandwidth, low-latency communication between GPUs and GPUs, and between GPUs and CPUs. It significantly improves data transmission efficiency through dedicated hardware interconnect channels, especially in multi-GPU parallel computing and large-scale deep learning tasks.

[0023] S102: Acquire the original model to be compressed and the compression method, determine the compressed model according to the original model to be compressed and the compression method, and determine the second storage space of the compressed model.

[0024] In one example, the original model to be compressed may be a supervised learning model or an unsupervised learning model. The specific deep learning model is not limited here.

[0025] In one example, obtaining a compression method includes: Retrieve the compression method from the preset model compression database and configure the number of layers to be compressed and the quantized data format.

[0026] In one example, the preset model compression database includes at least one of the following compression methods: Pruning, quantization, distillation, low-rank decomposition, weight sharing.

[0027] In one example, pruning is to reduce the size and computation of a model by removing unimportant or redundant weights, neurons, or connections in a neural network. Its goal is to improve the model's inference speed and storage efficiency without significantly reducing model performance.

[0028] In one example, quantization works by converting high-precision floating point numbers in the model, such as FP32, to lower-precision numerical representations, such as FP16, INT8, or lower.

[0029] In one example, distillation aims to transfer the knowledge of a large, complex, and high-performing teacher model (TeacherModel) to a smaller, more computationally efficient student model (Student Model), so that the student model can be as close as possible to the performance of the teacher model while remaining small.

[0030] In one example, low-rank decomposition is a technique that decomposes a high-dimensional matrix into a low-rank matrix, which is widely used in the compression and acceleration of deep learning models.

[0031] In one example, weight sharing is a technique widely used in neural networks, the core idea of ​​which is to share the same set of parameters (weights and biases) between different parts or layers of the network. This mechanism can significantly reduce the number of parameters of the model, reduce computational complexity, and improve the generalization ability of the model.

[0032] The original models to be compressed are respectively compressed by the above compression method to obtain corresponding compressed models, and the second storage space of the plurality of compressed models is determined.

[0033] The advantage of this setting is that it can generate a combination of multiple methods, providing users with all feasible compression solutions and giving users more possibilities for using model compression technology.

[0034] S103: If the first storage space is not smaller than the second storage space, determining whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model.

[0035] In one example, if the first storage space is not less than the second storage space, it means that the hardware device to be deployed has sufficient storage space to run the compressed model, and then it is determined whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model. The performance feasibility of the compressed model refers to whether the performance of the compressed model is executable.

[0036] The present application provides a model compression method deployed on a hardware device, the method comprising: obtaining parameter information of a hardware device to be deployed, and determining a first storage space of the hardware device to be deployed and hardware operation information of the hardware device to be deployed according to the parameter information of the hardware device to be deployed, obtaining an original model to be compressed and a compression method, determining a compressed model according to the original model to be compressed and the compression method, and determining a second storage space of the compressed model, if the first storage space is not less than the second storage space, then determining whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model. By adopting this technical solution, the model size that can be configured by the current electronic device can be quickly determined, and the user does not need to try one by one, thereby improving the configuration efficiency.

[0037] Figure 2 The flowchart of a model compression method deployed on a hardware device provided by an embodiment of the present disclosure is shown. The embodiment of the present disclosure is optimized on the basis of the above embodiment, and the embodiment of the present disclosure can be combined with various optional solutions in one or more of the above embodiments.

[0038] like Figure 2 As shown, the model compression method deployed on the hardware device may include the following steps: S201. Obtain parameter information of a hardware device to be deployed, and traverse a preset hardware database; wherein the preset hardware database is used to represent a mapping relationship between parameter information of a preset hardware device, a first storage space of the preset hardware device, and hardware operation information of the preset hardware device.

[0039] In one example, the preset hardware database represents the mapping relationship between the parameter information of the preset hardware device, the first storage space of the preset hardware device, and the hardware operation information of the preset hardware device. Specifically, the parameter information of the preset hardware device, the first storage space of the preset hardware device, and the hardware operation information of the preset hardware device are one-to-one corresponding.

[0040] S202: If parameter information of the hardware device to be deployed is found in the preset hardware database, the first storage space of the hardware device to be deployed and hardware operation information of the hardware device to be deployed are retrieved from the preset hardware database.

[0041] In one example, after obtaining parameter information of the hardware device to be deployed, if the parameter information of the hardware device to be deployed can be found in a preset hardware database, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are determined in the preset hardware database according to a mapping relationship.

[0042] S203: If the parameter information of the hardware device to be deployed is not found in the preset hardware database, the parameter information of the hardware device to be deployed is added to the hardware device prediction module.

[0043] In one example, if parameter information of the hardware device to be deployed is not found in the preset hardware database, it means that the parameter information of the hardware device to be deployed has not appeared in the preset hardware database, and the parameter information of the hardware device to be deployed needs to be added to the hardware device prediction module.

[0044] S204: Based on the hardware device prediction module, according to the parameter information of the hardware device to be deployed, output the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed.

[0045] In one example, based on the hardware device prediction module, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are determined and output in real time according to the parameter information of the hardware device to be deployed.

[0046] For a clearer explanation, see Figure 3 A structural schematic diagram of a hardware device prediction module is shown.

[0047] S205: Acquire the original model to be compressed and the compression method, determine the compressed model according to the original model to be compressed and the compression method, and determine the second storage space of the compressed model.

[0048] In an example, this step may refer to the content of S102.

[0049] S206: If the first storage space is not smaller than the second storage space, determine whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model.

[0050] In one example, determining whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model includes: Inputting the compressed model into a performance prediction module to determine target performance information of the compressed model; Based on the performance prediction module, it is determined whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the target performance information of the compressed model.

[0051] In one example, the performance prediction module is used to predict target performance information of different models. The target performance information of the compressed model may be the communication transmission speed of the compressed model, the computing speed of the hardware required by the compressed model, and the energy consumption information required by the compressed model.

[0052] In one example, the hardware operation information of the hardware device to be deployed, such as: the communication volume, computing volume and power consumption value of each graphics card. The target performance information of the compressed model, such as: the communication transmission speed of the compressed model and the computing speed of the hardware required for the compressed model. Based on the performance prediction module, the transmission time is determined according to the communication volume of each graphics card and the communication transmission speed of the compressed model; and the computing time is determined according to the computing volume of each graphics card and the computing speed of the hardware required for the compressed model. The energy consumption is determined according to the energy consumption information required for the compressed model and the power consumption value required for each graphics card. Then, based on the transmission time, computing time and energy consumption, it is determined whether the hardware device to be deployed executes the compressed model.

[0053] The advantage of this setting is that users can obtain the compression implementation plan and the performance of the compressed hardware by simply selecting their own hardware device model and the original model structure to be compressed. This greatly simplifies the user operation steps, lowers the model compression threshold, and improves the universality of large model technology.

[0054] For a clearer explanation, see Figure 4 A structural schematic diagram of storage space comparison is shown.

[0055] In one example, the compressed model is input into a performance prediction module to determine target performance information of the compressed model, including: Inputting the compressed model into the performance prediction module and obtaining usage information of the compressed model; wherein the usage information of the compressed model is reasoning information or fine-tuning information; Based on the performance prediction module, target performance information of the compressed model is determined.

[0056] In one example, fine-tuning information is the information needed in the fine-tuning process. Fine-tuning refers to further training on a dataset for a specific task based on the original model to be compressed to adjust the model parameters to better suit the target task. During the fine-tuning process, most of the layers of the pre-trained model are usually frozen, and only the newly added layers are trained or a small number of key layers are adjusted. This can not only retain the features used for learning the original model to be compressed, but also quickly adapt to the specific needs of the new task. In addition, choosing the right learning rate and training rounds is also the key to successful fine-tuning. Fine-tuning refers to small-scale training for specific task objectives (downstream tasks) and task data (downstream data) based on the original model to be compressed, to achieve minor adjustments to the parameters of the pre-trained model, and finally obtain a model adapted to specific tasks and data.

[0057] In one example, if the usage information of the compressed model is reasoning information, the reasoning target performance information of the compressed model is determined based on the performance prediction module.

[0058] In one example, if the usage information of the compressed model is fine-tuning information, fine-tuning target performance information of the compressed model is determined based on the performance prediction module.

[0059] For a clearer explanation, see Figure 5 A schematic diagram of the structure of a performance prediction module is shown.

[0060] In one example, based on the performance prediction module, determining whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the target performance information of the compressed model includes: Based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is better than the target performance information of the compressed model, then determining that the hardware device to be deployed executes the compressed model; Based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is not better than the target performance information of the compressed model, it is determined that the hardware device to be deployed does not execute the compressed model.

[0061] In one example, based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is better than the target performance information of the compressed model, it means that the hardware device to be deployed can successfully execute the compressed model, and then it is determined that the hardware device to be deployed executes the compressed model.

[0062] In one example, based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is not better than the target performance information of the compressed model, it means that the hardware device to be deployed cannot successfully execute the compressed model, and it is determined that the hardware device to be deployed does not execute the compressed model.

[0063] In one example, after determining that the hardware device to be deployed executes the compressed model, the method further includes: Calculating scores of target performance information of the plurality of compressed models, and sending the corresponding scores to a user; wherein the target performance information of the plurality of compressed models is generated by the compressed models using different compression methods; Receive feedback from the user and determine a compression method used by the compressed model.

[0064] In one example, this part can be calculated by the built-in score calculation module. Specifically, the compression methods used by the compressed model that meets the target performance information of the hardware operation information of the hardware device to be deployed is better than the compressed model are scored one by one. Specifically, the score includes the time, computing power and effect of the compressed model. Different compression methods may be suitable for different scenarios. Therefore, the score of the compressed model can be sent to the user so that the user can choose by himself and then determine the compression method that meets the user's needs, so that the compression method used is more suitable for the current scenario and improves flexibility.

[0065] For a clearer explanation, see Figure 6 A schematic diagram of a model compression process deployed on a hardware device is shown.

[0066] In one example, after determining that the hardware device to be deployed executes the compressed model, the method further includes: Call a preset compression program file to compress the original model to be compressed to obtain a compressed model, and deploy the compressed model on the hardware device to be deployed.

[0067] In one example, the preset compression program file is pre-configured, and after determining that the hardware device to be deployed executes the compressed model, the original model to be compressed is quickly compressed to obtain the compressed model, and the compressed model is deployed on the hardware device to be deployed. The advantage of such a setting is that the compressed model can be put into use as soon as possible.

[0068] In one example, it also includes: If the first storage space is smaller than the second storage space, it is determined that the hardware device to be deployed does not execute the compressed model.

[0069] In one example, if the first storage space is smaller than the second storage space, it means that the storage space of the hardware device to be deployed is insufficient, and therefore, it is determined that the hardware device to be deployed does not execute the compressed model.

[0070] In one example, it also includes: Re-acquire other original models to be compressed, and re-execute the process of acquiring other original models to be compressed and compression methods, determine the compressed models according to other original models to be compressed and compression methods, and determine the third storage space of other original models to be compressed; If the first storage space is not smaller than the third storage space, it is determined whether the hardware device to be deployed executes the compressed other model according to the hardware operation information of the hardware device to be deployed and the compressed other model.

[0071] In one example, by replacing other original models to be compressed and re-performing steps S205 and S206, the process of deploying other compressed models on the hardware device to be deployed is completed. The advantage of such a setting is that it can ensure that the user configures the appropriate model and completes the user's task.

[0072] The present application provides a model compression method deployed on a hardware device, the method comprising: obtaining parameter information of a hardware device to be deployed, traversing a preset hardware database, and if the parameter information of the hardware device to be deployed is found in the preset hardware database, then retrieving the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed in the preset hardware database. If the parameter information of the hardware device to be deployed is not found in the preset hardware database, then the parameter information of the hardware device to be deployed is added to the hardware device prediction module. Based on the hardware device prediction module, according to the parameter information of the hardware device to be deployed, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are output. The original model to be compressed and the compression method are obtained, and the compressed model is determined according to the original model to be compressed and the compression method, and the second storage space of the compressed model is determined. If the first storage space is not less than the second storage space, then according to the hardware operation information of the hardware device to be deployed and the compressed model, it is determined whether the hardware device to be deployed executes the compressed model. By adopting this technical solution, multiple databases are constructed, which can avoid repeated measurements and save manpower and financial resources. At the same time, the existence of the database makes it possible to predict the computing efficiency of hardware not only by relying on theoretical peak values ​​and empirical values, but also by using the actual measured computing speed. Using the actual measured computing speed is conducive to ensuring the accuracy of the prediction results.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0074] Figure 7Schematic diagram of a model compression device deployed on a hardware device provided by an embodiment of the present disclosure. The model compression device deployed on a hardware device can be understood as the above-mentioned electronic device or some functional modules in the above-mentioned electronic device. Figure 7 As shown, the model compression device 70 deployed on the hardware device includes: The first acquisition module 701 is used to acquire parameter information of the hardware device to be deployed, and determine the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed according to the parameter information of the hardware device to be deployed; The second acquisition module 702 is used to acquire the original model to be compressed and the compression method, determine the compressed model according to the original model to be compressed and the compression method, and determine the second storage space of the compressed model; The determination module 703 is used to determine whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model if the first storage space is not smaller than the second storage space.

[0075] In one example, the first acquisition module 701 is specifically configured to: Traversing a preset hardware database; wherein the preset hardware database is used to characterize a mapping relationship between parameter information of a preset hardware device, a first storage space of a preset hardware device, and hardware operation information of the preset hardware device; If the parameter information of the hardware device to be deployed is found in the preset hardware database, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are retrieved from the preset hardware database.

[0076] In one example, the first acquisition module 701 is specifically configured to: If the parameter information of the hardware device to be deployed is not found in the preset hardware database, the parameter information of the hardware device to be deployed is added to the hardware device prediction module; Based on the hardware device prediction module, according to the parameter information of the hardware device to be deployed, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are output.

[0077] In one example, the hardware operation information of the preset hardware device includes at least one of the following: The computing speed of multiple-size tensors of hardware devices, network transmission bandwidth, network transmission latency, memory size of various types of storage, GPU interconnection transmission bandwidth, and GPU interconnection transmission latency.

[0078] In one example, the second acquisition module 702 is used to retrieve a compression method from a preset model compression database, and configure the number of layers of the model to be compressed and the quantized data format.

[0079] In one example, the preset model compression database includes at least one of the following compression methods: Pruning, quantization, distillation, low-rank decomposition, weight sharing.

[0080] In one example, the determination module 703 is used to: Inputting the compressed model into a performance prediction module to determine target performance information of the compressed model; Based on the performance prediction module, it is determined whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the target performance information of the compressed model.

[0081] In one example, the determination module 703 is used to: Inputting the compressed model into the performance prediction module and obtaining usage information of the compressed model; wherein the usage information of the compressed model is reasoning information or fine-tuning information; Based on the performance prediction module, target performance information of the compressed model is determined.

[0082] In one example, the determination module 703 is used to: based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is better than the target performance information of the compressed model, determine that the hardware device to be deployed executes the compressed model; Based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is not better than the target performance information of the compressed model, it is determined that the hardware device to be deployed does not execute the compressed model.

[0083] In one example, after determining that the hardware device to be deployed executes the compressed model, the apparatus 70 further includes: The calling module 704 is used to call a preset compression program file, compress the original model to be compressed, obtain the compressed model, and deploy the compressed model on the hardware device to be deployed.

[0084] In one example, the apparatus 70 further includes: The judgment module 705 is used to determine that the hardware device to be deployed does not execute the compressed model if the first storage space is smaller than the second storage space.

[0085] For the description of the features in the embodiment corresponding to the model compression device deployed on the hardware device, please refer to the relevant description of the embodiment corresponding to the model compression method deployed on the hardware device, which will not be repeated here.

[0086] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above-mentioned model compression method embodiments deployed on a hardware device.

[0087] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above-mentioned model compression method embodiments deployed on a hardware device when running.

[0088] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0089] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned model compression method embodiments deployed on a hardware device are implemented.

[0090] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned model compression method embodiments deployed on a hardware device.

[0091] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] The above is a detailed introduction to a model compression method deployed on a hardware device provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A model compression method deployed on a hardware device, characterized in that: include: Acquire parameter information of the hardware device to be deployed, and determine a first storage space of the hardware device to be deployed and hardware operation information of the hardware device to be deployed according to the parameter information of the hardware device to be deployed; Acquire an original model to be compressed and a compression method, determine a compressed model according to the original model to be compressed and the compression method, and determine a second storage space for the compressed model; If the first storage space is not smaller than the second storage space, determining whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model.

2. The method according to claim 1, characterized in that The determining, according to the parameter information of the hardware device to be deployed, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed comprises: Traversing a preset hardware database; wherein the preset hardware database is used to characterize a mapping relationship between parameter information of a preset hardware device, a first storage space of a preset hardware device, and hardware operation information of the preset hardware device; If the parameter information of the hardware device to be deployed is found in the preset hardware database, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are retrieved from the preset hardware database.

3. The method according to claim 2, characterized in that Also includes: If the parameter information of the hardware device to be deployed is not found in the preset hardware database, the parameter information of the hardware device to be deployed is added to the hardware device prediction module; Based on the hardware device prediction module, according to the parameter information of the hardware device to be deployed, the first storage space of the hardware device to be deployed and the hardware operation information of the hardware device to be deployed are output.

4. The method according to claim 2, characterized in that: The hardware operation information of the preset hardware device includes at least one of the following: The computing speed of multiple-size tensors of hardware devices, network transmission bandwidth, network transmission latency, memory size of various types of storage, GPU interconnection transmission bandwidth, and GPU interconnection transmission latency.

5. The method according to claim 1, characterized in that The method for obtaining compression comprises: Retrieve the compression method from the preset model compression database and configure the number of layers to be compressed and the quantized data format.

6. The method according to claim 5, characterized in that The preset model compression database includes at least one of the following compression methods: Pruning, quantization, distillation, low-rank decomposition, weight sharing.

7. The method according to claim 1, characterized in that The determining, based on the hardware operation information of the hardware device to be deployed and the compressed model, whether the hardware device to be deployed executes the compressed model includes: Inputting the compressed model into a performance prediction module to determine target performance information of the compressed model; Based on the performance prediction module, it is determined whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the target performance information of the compressed model.

8. The method according to claim 7, characterized in that The step of inputting the compressed model into a performance prediction module to determine target performance information of the compressed model includes: Inputting the compressed model into a performance prediction module, and obtaining usage information of the compressed model; wherein the usage information of the compressed model is reasoning information or fine-tuning information; Based on the performance prediction module, target performance information of the compressed model is determined.

9. The method according to claim 7, characterized in that: The determining, based on the performance prediction module, whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the target performance information of the compressed model includes: Based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is better than the target performance information of the compressed model, determining that the hardware device to be deployed executes the compressed model; Based on the performance prediction module, if the hardware operation information of the hardware device to be deployed is not better than the target performance information of the compressed model, it is determined that the hardware device to be deployed does not execute the compressed model.

10. The method according to claim 9, characterized in that After determining that the hardware device to be deployed executes the compressed model, the method further includes: A preset compression program file is called to compress the original model to be compressed to obtain the compressed model, and the compressed model is deployed on the hardware device to be deployed.

11. The method according to claim 1, characterized in that: Also includes: If the first storage space is smaller than the second storage space, it is determined that the hardware device to be deployed does not execute the compressed model.

12. A model compression device deployed on a hardware device, characterized in that: include: A first acquisition module, used to acquire parameter information of a hardware device to be deployed, and determine a first storage space of the hardware device to be deployed and hardware operation information of the hardware device to be deployed according to the parameter information of the hardware device to be deployed; A second acquisition module is used to acquire an original model to be compressed and a compression method, determine a compressed model according to the original model to be compressed and the compression method, and determine a second storage space for the compressed model; A determination module is used to determine whether the hardware device to be deployed executes the compressed model according to the hardware operation information of the hardware device to be deployed and the compressed model if the first storage space is not smaller than the second storage space.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the model compression method deployed on a hardware device as described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the model compression method deployed on a hardware device as described in any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model compression method deployed on a hardware device as described in any one of claims 1 to 11 are implemented.

Images