Compression method and device of deep learning model and electronic equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a compression method and device of a deep learning model and electronic equipment. The method comprises the following steps: acquiring a deep learning model to be compressed, and respectively determining Fisher importance and parameter association importance of model parameters; determining a collaborative importance degree based on the Fisher importance degree and the parameter association importance degree; and pruning the deep learning model based on the collaborative importance of the model parameters. The compression method of the deep learning model, provided by the invention, is better in pruning efficiency.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence technology, and specifically relates to a compression method, device and electronic device for a deep learning model. Background Art

[0002] With the successful application of deep learning models in various tasks, the contradiction between the limited hardware resources of model deployment devices and the huge size of the model has made model compression technology a hot topic. Model compression refers to reducing the number of model parameters, computational complexity, and storage space through a series of technical means, thereby maintaining or improving model performance while making it more suitable for running on resource-constrained devices.

[0003] Model pruning is a common model compression technique. Model pruning can remove parameters that contribute less to the model output, thereby significantly reducing the number of model parameters while minimizing the loss of model performance. Current model pruning methods rely on calculating the Fisher importance of parameters based on gradient information to remove parameters with less importance.

[0004] However, gradient information is essentially local information, and Fisher importance cannot evaluate the importance of parameters from a global dimension. Therefore, pruning methods based on Fisher importance often cannot achieve the global optimal pruning effect. Summary of the invention

[0005] The disclosed embodiment proposes a pruning solution for a deep learning model to solve the problem that the pruning effect of the current pruning method based on Fisher importance is often not globally optimal.

[0006] A first aspect of an embodiment of the present disclosure provides a method for compressing a deep learning model, including: Obtain a deep learning model to be compressed, and determine Fisher importance and parameter association importance of model parameters respectively; Determine the synergistic importance based on Fisher importance and parameter correlation importance; The deep learning model is pruned based on the co-importance of model parameters.

[0007] In some embodiments of the present disclosure, determining the Fisher importance of the model parameters includes: The model parameters are grouped according to their dependencies, and the Hessian matrix of the model parameters is calculated; Determine the Fisher importance from the Hessian matrix.

[0008] In some embodiments of the present disclosure, determining the Fisher importance from the Hessian matrix includes: Compute the Fisher information matrix from the Hessian matrix; The Fisher importance is determined based on the Fisher information matrix according to the following formula: ; in is the i-th eigenvalue of the Fisher information matrix, is the Fisher importance, is the quantum state angle corresponding to the i-th model parameter.

[0009] In some embodiments of the present disclosure, determining the parameter association importance of the model parameters includes: Performing quantum state mapping on the model parameters; The parameter association strength of the model parameters is determined based on the quantum state mapping result, and the parameter association importance is determined based on the parameter association strength.

[0010] In some embodiments of the present disclosure, determining the parameter association importance based on the parameter association strength includes: ; in, is the parameter association importance, is the parameter association strength between the i-th and j-th parameters, and N is the number of samples.

[0011] In some embodiments of the present disclosure, determining the collaborative importance based on the Fisher importance and the parameter association importance includes: ; in, For collaborative importance, is the Fisher importance, is the quantum correlation importance, is the quantum state angle corresponding to the i-th model parameter, is the weight coefficient, satisfying .

[0012] In some embodiments of the present disclosure, the method further includes: The compressed deep learning model is fine-tuned to restore the accuracy of the model.

[0013] A second aspect of an embodiment of the present disclosure provides a deep learning model compression device, including: An acquisition module, used for acquiring a deep learning model to be compressed, and determining Fisher importance and parameter association importance of model parameters respectively; A determination module, for determining the synergy importance based on the Fisher importance and the parameter association importance; A pruning module is used to prune the deep learning model based on the co-importance of model parameters.

[0014] A third aspect of the embodiments of the present disclosure provides an electronic device, including a memory and a processor. The memory is used to store computer programs; The processor is configured to implement the method according to the first aspect of the present disclosure when executing the computer program.

[0015] In summary, the compression method, device, and electronic device of the deep learning model provided by the embodiments of the present disclosure evaluate the parameter importance by collaboratively evaluating the Fisher importance generated based on gradient information and the parameter correlation importance measured using the quantum correlation method, and pruning is performed based on the collaborative importance, thereby integrating the advantages of the Fisher importance and the parameter correlation importance, comprehensively evaluating the parameter importance from both local and global dimensions, and improving the pruning efficiency of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings: Figure 1 It is a framework diagram of an algorithm for performing parameter pruning on a deep learning model using collaborative importance as shown in the present disclosure; Figure 2 is a flowchart of a deep learning model compression method according to some embodiments of the present disclosure; Figure 3 It is a specific process of the pruning stage and the fine-tuning stage shown in some embodiments of the present disclosure; Figure 4 is a schematic diagram of a compression device for a deep learning model according to some embodiments of the present disclosure; Figure 5 It is a schematic diagram of an electronic device shown in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0018] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.

[0019] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0020] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0021] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.

[0022] With the successful application of deep learning models in various tasks, the size and complexity of the models are also increasing rapidly. Although large models usually have higher performance, when faced with limited hardware resources, the training and reasoning of the models will be affected to a certain extent, resulting in performance loss of the models. Therefore, deep learning model compression methods have important research significance in model optimization and deployment.

[0023] Model compression includes model pruning, model quantization, knowledge distillation, etc. The compression method disclosed in the present invention refers to model pruning. Model pruning can remove parameters that contribute less to the model output, thereby significantly reducing the number of model parameters while minimizing the loss of model performance.

[0024] The current model pruning methods all rely on calculating the Hessian matrix of the model parameters, calculating the Fisher information from the Hessian matrix, and thus obtaining an evaluation of the importance of the model parameters to delete parameters with less importance. However, the Hessian matrix is ​​a second-order derivative matrix about the parameter vector, which is essentially a gradient operation, and the gradient operation can only provide local information. Therefore, the pruning method based on Fisher importance often cannot achieve the global optimal pruning effect.

[0025] In order to solve the above problems, the present disclosure introduces a quantum correlation method to determine the parameter correlation importance so that the parameter importance can be evaluated from a global dimension, and combines it with the Fisher importance that evaluates the parameter importance from a local dimension to collaboratively evaluate the parameter importance from both global and local dimensions and prune the model parameters based on the collaborative importance, thereby achieving the purpose of compressing the model. Figure 1 is an algorithm framework diagram for performing parameter pruning on a deep learning model using collaborative importance as shown in the present disclosure. In some embodiments, the compression method of the deep learning model is as follows Figure 2 As shown, the specific steps include: S210, obtaining a deep learning model to be compressed, and determining Fisher importance and parameter association importance of model parameters respectively.

[0026] Specifically, the deep learning model may be (but not limited to) VGG and ResNet series network models.

[0027] The present disclosure performs pruning based on the synergistic importance of the comprehensive Fisher importance and the parameter association importance in the pruning stage. First, the Fisher importance is calculated based on the Hessian matrix, and the specific steps include: First, calculate the Hessian matrix of the model parameters. The specific formula is as follows: ; Among them, L represents the loss function, H represents the Hessian matrix, represents the parameter vector of the neural network, is the ith output of the neural network.

[0028] Then calculate the Fisher information matrix. The specific formula is as follows: ; in is the number of samples, is the number of samples of the ith type, represents the parameter vector of the neural network, In the sample The Hessian matrix on , E represents the expectation.

[0029] Finally, the parameter importance is calculated based on the Fisher information matrix. The formula is as follows: ; in is the i-th eigenvalue of the Fisher information matrix, Fisher importance. is the quantum state angle corresponding to the i-th model parameter.

[0030] Because Fisher importance is derived from the Hessian matrix calculated based on the gradient, and gradient information is essentially local information, Fisher importance can provide an assessment of the importance of model parameters from a local level.

[0031] Then, the parameter correlation importance is obtained based on the quantum correlation method. The specific steps include: First, the model parameters are mapped to quantum states, and the calculation formula is as follows: ; in , , is the i-th parameter of the model, is the quantum state polar angle corresponding to the ith parameter, is the quantum state phase angle corresponding to the i-th parameter. Then the parameter correlation strength is calculated based on the quantum state mapping result. The calculation formula is as follows: ; The expanded form is as follows: ; Where ψ(i)> represents the quantum state corresponding to the i-th parameter. The square of the inner product of the quantum states corresponding to the i-th parameter and the i-th parameter is used to indicate their overlap. C(i,j) is the quantum correlation strength between parameter i and parameter j. The i in is an imaginary unit, that is, .

[0032] Finally, the parameter association importance is determined based on the parameter association strength: ; in, is the parameter association importance, is the parameter association strength between the i-th and j-th parameters, and N is the number of samples.

[0033] Because parameter association importance is based on the strength of association between parameters at the global level, parameter association importance can provide an assessment of the importance of model parameters from a global level.

[0034] S220, determining the collaborative importance based on the Fisher importance and the parameter association importance.

[0035] Specifically, the calculation formula of collaborative importance is as follows: ; in, For collaborative importance, is the Fisher importance, is the quantum correlation importance, is the quantum state angle corresponding to the i-th model parameter, is the quantum state angle corresponding to the i-th model parameter, satisfying .

[0036] S230, pruning the deep learning model based on the collaborative importance of model parameters.

[0037] In some embodiments of the present disclosure, after pruning the deep learning model, the method further includes: Verify the performance of the deep learning model after pruning. If the performance does not reach the preset indicator, iteratively prune the deep learning model until the performance reaches the preset indicator.

[0038] In summary, in the pruning stage, the present invention combines the Fisher importance generated based on gradient information with the parameter correlation importance measured by the quantum correlation method to produce a parameter importance result, and performs pruning based on the collaborative importance, thereby integrating the advantages of Fisher importance and parameter correlation importance, comprehensively evaluating the parameter importance from both local and global dimensions, and improving the pruning efficiency of deep learning models.

[0039] After completing parameter pruning, some embodiments of the present disclosure further include a fine-tuning phase, specifically: Select a subset of the training set as the data for fine-tuning, set the iteration rounds, learning rate and other parameters on the fine-tuning data set, and perform fine-tuning. After each round of fine-tuning, verify the results of various indicators of the fine-tuning model, and iterate fine-tune according to the results. Generally, 1-2 epochs are required.

[0040] In some embodiments of the present disclosure, the specific processes of the pruning phase and the fine-tuning phase are as follows: Figure 3 shown.

[0041] Figure 4 is a schematic diagram of a compression device for a deep learning model according to some embodiments of the present disclosure. Figure 4 As shown, the compression device 400 of the deep learning model includes an acquisition module 410, a determination module 420, and a pruning module 430. Among them: An acquisition module 410 is used to acquire a deep learning model to be compressed, and determine Fisher importance and parameter association importance of model parameters respectively; A determination module 420, configured to determine the synergy importance based on the Fisher importance and the parameter association importance; The pruning module 430 is used to prune the deep learning model based on the collaborative importance of model parameters.

[0042] Figure 5 is a schematic diagram of an electronic device according to some embodiments of the present disclosure. Figure 5 As shown, the electronic device 500 includes a memory 520 and a processor 510, wherein the memory 520 is used to store a computer program; and the processor 510 is used to implement when executing the computer program. Figure 2 The compression method of the deep learning model described in S210-S230.

[0043] In summary, the compression method, device, and electronic device of the deep learning model provided by the embodiments of the present disclosure evaluate the parameter importance by collaboratively evaluating the Fisher importance generated based on gradient information and the parameter correlation importance measured using the quantum correlation method, and pruning is performed based on the collaborative importance, thereby integrating the advantages of the Fisher importance and the parameter correlation importance, comprehensively evaluating the parameter importance from both local and global dimensions, and improving the pruning efficiency of the deep learning model.

[0044] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0045] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0046] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A compression method for a deep learning model, characterized in that: include: Obtain a deep learning model to be compressed, and determine Fisher importance and parameter association importance of model parameters respectively; Determine the synergistic importance based on Fisher importance and parameter correlation importance; The deep learning model is pruned based on the co-importance of model parameters.

2. The method according to claim 1, characterized in that: The Fisher importance of determining the model parameters includes: The model parameters are grouped according to their dependencies, and the Hessian matrix of the model parameters is calculated; Determine the Fisher importance from the Hessian matrix.

3. The method according to claim 2, characterized in that: Determining the Fisher importance from the Hessian matrix includes: Compute the Fisher information matrix from the Hessian matrix; The Fisher importance is determined based on the Fisher information matrix according to the following formula: ； in is the i-th eigenvalue of the Fisher information matrix, is the Fisher importance, is the quantum state angle corresponding to the i-th model parameter.

4. The method according to claim 1, characterized in that: The parameter-related importance of determining model parameters includes: Performing quantum state mapping on the model parameters; The parameter association strength of the model parameters is determined based on the quantum state mapping result, and the parameter association importance is determined based on the parameter association strength.

5. The method according to claim 4, characterized in that: Determining the parameter association importance based on the parameter association strength includes: ； in, is the parameter association importance, is the parameter association strength between the i-th and j-th parameters, and N is the number of samples.

6. The method according to claim 1, characterized in that: The determining of the collaborative importance based on the Fisher importance and the parameter association importance includes: ； in, For collaborative importance, is the Fisher importance, is the quantum correlation importance, is the quantum state angle corresponding to the i-th model parameter, is the weight coefficient, satisfying .

7. The method according to claim 1, characterized in that: After pruning the deep learning model, the method further includes: Verify the performance of the deep learning model after pruning, and iteratively prune the deep learning model until the performance reaches a preset indicator.

8. The method according to claim 1, characterized in that: The method further comprises: The compressed deep learning model is fine-tuned to restore the accuracy of the model.

9. A compression device for a deep learning model, characterized in that: include: An acquisition module, used for acquiring a deep learning model to be compressed, and determining Fisher importance and parameter association importance of model parameters respectively; A determination module, for determining the synergy importance based on the Fisher importance and the parameter association importance; A pruning module is used to prune the deep learning model based on the co-importance of model parameters.

10. An electronic device, characterized in that: including memory and processor, The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1-8 when executing the computer program.

Images