Method and system for computing compression strategies

Abstract

The present disclosure relates to a computing system and method for computing and using a compression strategy for a neural network, comprising the steps of: selecting a plurality of compression methods, a plurality of compression ratios associated with the plurality of compression methods, and a target global compression ratio for the neural network; compressing each layer of the plurality of layers of the neural network according to each combination of the plurality of compression methods and the plurality of compression ratios; for each layer of the plurality of layers, computing a value of a loss function and a value of a cost function; for each layer of the plurality of layers, using the value of the loss function and the value of the cost function to compute the compression strategy corresponding to the target global compression ratio; and compressing the neural network according to the compression strategy.

Description

Technical Field

[0001] This disclosure relates to methods and systems for computing and using compression strategies of neural networks. Background Technology

[0002] Neural networks are a class of machine learning models that can be used to perform a wide variety of tasks across different fields, such as healthcare, automotive, and the Internet of Things (IoT). Neural networks typically have an architecture that includes an input layer, an output layer, and one or more intermediate layers. Complex neural networks can have up to thousands of intermediate layers. Deploying such complex neural networks on edge devices presents significant challenges due to limited resource and power constraints. Summary of the Invention

[0003] The appended claims define aspects of this disclosure.

[0004] In a first aspect of this disclosure, a method for using a compression strategy of a neural network is provided. The method includes the steps of: selecting a neural network, wherein the neural network includes a plurality of layers; selecting a plurality of compression methods, a plurality of compression ratios, and a target global compression ratio of the neural network; for each of the plurality of layers of the neural network, calculating the value of a loss function and the value of a cost function for each combination of the plurality of compression methods and the plurality of compression ratios; using the value of the loss function and the value of the cost function for each combination of the plurality of compression methods and the plurality of compression ratios to calculate the compression strategy corresponding to the target global compression ratio; and compressing the neural network according to the compression strategy.

[0005] In one embodiment of the first aspect of this disclosure, the method further includes the steps of: for each of the plurality of layers of the neural network, inferring a combination that minimizes the loss function and the cost function, and using the combination to compute the compression strategy.

[0006] In one embodiment of the first aspect of this disclosure, the plurality of compression methods, the plurality of compression ratios, and the target compression ratio of the neural network are stored in a database.

[0007] In one embodiment of the first aspect of this disclosure, the value of the loss function and the value of the cost function are stored in a database.

[0008] In one embodiment of the first aspect of this disclosure, the method further includes the step of calculating the compression strategy corresponding to the target global compression ratio using an inferred combination, which is performed using a solver for a constrained optimization problem.

[0009] In one embodiment of the first aspect of this disclosure, the method further includes the step of storing a compression strategy in a database.

[0010] In one embodiment of the first aspect of this disclosure, the compression method is a quantization method, a pruning method, a weight sharing method, a tensor decomposition method, or a low-rank matrix decomposition method.

[0011] In one embodiment of the first aspect of this disclosure, further fine-tuning steps are applied on each of the compressed layers of the neural network or on the entire compressed neural network.

[0012] In a second aspect of this disclosure, a computing system is provided. The computing system includes one or more processors and one or more non-transitory computer-readable media storing instructions, which, when executed by the one or more processors, cause the computing system to: select a neural network, wherein the neural network includes a plurality of layers; select a plurality of compression methods, a plurality of compression ratios, and a target global compression ratio of the neural network; for each of the plurality of layers of the neural network, compute a loss function and a cost function for each combination of the plurality of compression methods and the plurality of compression ratios; use the loss function and the cost function for each combination of the plurality of compression methods and the plurality of compression ratios to compute a compression strategy corresponding to the target global compression ratio; and compress the neural network according to the compression strategy.

[0013] In one embodiment of the second aspect of this disclosure, the step of selecting the neural network, and / or the plurality of compression methods, the plurality of compression ratios, and / or the target global compression ratio of the neural network is performed via a user interface.

[0014] In one embodiment of the second aspect of this disclosure, the computing system further includes a database.

[0015] In one embodiment of the second aspect of this disclosure, the database is configured to store: the neural network, and / or the plurality of compression methods, and / or the plurality of compression ratios, and / or the target global compression ratio of the neural network, and / or the loss function of each layer of the neural network, and / or the cost function of each layer of the neural network.

[0016] In one embodiment of the second aspect of this disclosure, the database is configured to store the compression strategy.

[0017] In one embodiment of the second aspect of this disclosure, the database is stored in one or more non-transitory computer-readable media.

[0018] In one embodiment of the second aspect of this disclosure, the database is stored on an external server. Attached Figure Description

[0019] Embodiments of this disclosure will be described in more detail with reference to the accompanying drawings, in which:

[0020] Figure 1 A block diagram illustrating a method according to a first embodiment of the present disclosure is shown.

[0021] Figure 2 A compression method according to a first embodiment of the present disclosure is shown.

[0022] Figure 3 A database according to a first embodiment of the present disclosure is shown.

[0023] Figure 4 The calculation steps according to a first embodiment of the present disclosure are shown.

[0024] Figure 5 A system according to a second embodiment of the present disclosure is shown.

[0025] It should be noted that the figures are illustrative and not drawn to scale. For clarity and convenience in the figures, the relative dimensions and proportions of the parts have been shown by enlarging or reducing their size. The same reference numerals are generally used to refer to corresponding or similar features in modified and different embodiments. Detailed Implementation

[0026] Compression techniques can provide a solution for deploying complex neural networks on edge devices. Compression techniques can help reduce memory footprint and inference latency by using methods such as quantization, pruning, and low-rank matrix factorization. While these methods enable the deployment of machine learning models on resource-constrained edge devices, determining the optimal combination of compression methods and compression ratios for each layer remains a challenge. Specifically, it can be difficult to find a balance between maximizing compression of the entire neural network and the overall performance of the model. Furthermore, it should be noted that optimal compression can differ between layers: some layers can be significantly compressed without performance degradation, while others cannot. A common approach to finding optimal compression for each layer of a neural network involves the user using their experience or intuition to guide the compression method and compression ratio for each layer of the specified network. This approach is not time-efficient because users often invest significant time and effort in experimenting with various combinations of compression methods and compression ratios for each layer. Additionally, this approach often yields suboptimal results because users typically cannot exhaustively explore all combinations.

[0027] Failure to find the optimal compression method and compression ratio for each layer can lead to performance degradation, increased memory usage, and longer inference time, thus hindering the effectiveness of neural networks on edge devices.

[0028] This disclosure describes a fully automated method and system for compressing neural networks according to a compression strategy. The compression strategy can be defined as a policy that describes which compression method and compression ratio should be used for each layer of a given neural network to achieve a selected global target compression ratio. The global target compression ratio is defined as the compression ratio that the entire neural network should achieve. It should be noted that the global target compression ratio is not a layer-by-layer value, but a global value effective for the entire neural network. Furthermore, several global compression ratios can be defined. In this embodiment, the method of this disclosure is performed N times, where N is the number of target global compression ratios specified by the user.

[0029] Figure 1 A method according to a first embodiment of this disclosure is illustrated. In a first step 102, a neural network is selected. The neural network can be any type of neural network, as the method of this disclosure is insensitive to the neural network used. As an example, the neural network can be a one-shot neural network, a recurrent neural network, a convolutional neural network, or a feedforward neural network. The neural network can be a pre-trained neural network. In a second step 104, multiple compression methods, multiple compression ratios for these compression methods, and a global target compression ratio are selected. The neural network, the multiple compression methods, the multiple compression ratios associated with these compression methods, and the global target compression ratio can be selected by a user via a user interface, for example, by selecting the neural network, the multiple compression methods, the multiple compression ratios, and the global target compression ratio from a table or list. The compression methods can be quantization methods, pruning methods, weight sharing methods, tensor decomposition methods, or low-rank matrix factorization methods. The relevant compression ratios, for quantization methods, can be weight compression between 2 and 8 bits, and for pruning methods, can be sparsity ratios between 10%, 20%, ..., 90%. For low-rank matrix factorization methods, the weights of each layer of the neural network can be decomposed into two matrices of size M and N, respectively. The compression ratio of this method then ranges between 2 and the minimum of M or N. It should be noted that several compression methods with corresponding compression ratios can be selected. The compression ratio can be chosen as a wide or narrow range of values. This choice of compression ratio and compression method can further depend on the use case of applying the neural network.

[0030] In step 106, the method calculates the compression of each layer of the neural network based on all possible combinations of compression methods and relevant compression ratios. In an exemplary embodiment, the user may have specified quantization as the compression method, where the compression ratio is between 2 bits and 6 bits. The method of this application then creates compressed layers for each specified compression ratio. In other words, the following compressed layers of the original layer i are calculated: i compressed layers quantized to 2 bits, i compressed layers quantized to 3 bits, i compressed layers quantized to 4 bits, i compressed layers quantized to 5 bits, and i compressed layers quantized to 6 bits. In another embodiment, the layer can be compressed by a combination of a first method with a relevant compression ratio and a second method with a relevant compression ratio. For example, the layer can be compressed using a combination of a quantization method with a weight compression of 2 and a pruning method with a sparsity ratio of 10%.

[0031] In step 108, the method calculates a loss function for each combination of compression method m and compression ratio k for each compression layer i in the neural network. Values ​​and cost functions The value of the loss function. The value can be calculated using knowledge distillation (KD) loss. The KD loss is calculated relative to the original uncompressed layer and quantifies the degradation of the original uncompressed layer by applying the compression method along with its corresponding compression ratio. In other words, the KD loss quantifies the difference between each original uncompressed layer and each corresponding compressed layer. Examples of KD loss are mean squared error (MSE) and signal-to-noise ratio (NSR). Cost function Memory footprint or inference latency, for example, can be quantified. Memory footprint can be defined as the amount of memory required to run a neural network. In the case of a neural network running on a memory-constrained embedded device, memory footprint should be kept small, as such embedded devices typically do not have large amounts of memory. Inference latency can be defined as the time delay between when a neural network receives input and when the same neural network generates the corresponding output. In other words, inference latency measures the delay in which a neural network model processes input and executes its inference logic to produce a prediction or decision. Inference latency should be kept small, especially when using neural networks in security applications or time-constrained applications.

[0032] The method disclosed herein calculates the loss function between the original layer i and each compressed variant of layer i for each layer i. The value of . Additionally, the cost function for each compression layer is calculated using feedback from the hardware target. The value of . This final step can be performed in parallel with the calculation of the loss function. Repeat this procedure for all layers, and will . Figure 4 A detailed explanation will follow.

[0033] In optional step 110, the method of this disclosure can calculate the loss function layer by layer. and cost function The values ​​are stored in a database, also known as a metadata database. The database may optionally store information about the neural network used and the specific use case for requesting compression. This can have the effect of reusing historical data to speed up future applications, such as when providing the method to different customers using a similar neural network. The metadata database can be stored locally or on an external server. Storing the database on an external server can make it easier for external customers to use.

[0034] In optional step 112, the method of this disclosure can derive, for each layer i, a combination of compression method and compression ratio that minimizes the values ​​of the loss function and the cost function. This means that there is no alternative combination of compression method and compression ratio for that layer that improves one aspect (e.g., network efficiency) without sacrificing another aspect (e.g., task performance). In other words, the current combination of compression method and compression ratio represents the best achievable trade-off between network efficiency and task performance for that particular layer. These combinations can be referred to as dominant combinations. This step can be considered a preprocessing step before building the compressed neural network, filtering out suboptimal options. This preprocessing step can have the effect of significantly reducing the computation time of the next step (i.e., the computation of the compression strategy).

[0035] In step 114, the method of this disclosure calculates a compression strategy by selecting the optimal layer-by-layer combination of a compression method and a compression ratio that satisfies the selected global target compression ratio. In other words, for each layer i, a combination of compression methods and compression ratios is selected such that the global target compression ratio is achieved. This can be achieved by formulating the problem as a constrained optimization problem. An example of a constrained optimization model is the multi-choice knapsack problem. To solve the constrained optimization problem, the optimal layer-by-layer combination that minimizes the sum of the loss function and cost function is selected. In other words, for each layer, a compression method and a relevant compression ratio are selected such that the overall sum of the loss function and cost function is minimized. The compression method and compression ratio selected for each layer represent the compression strategy. Different layers can have different compression methods and different compression ratios. In step 116, the neural network is compressed according to the compression strategy. In an optional step 118, fine-tuning of the neural network can be performed. Fine-tuning of the neural network has the effect of recovering from potential losses caused by degradation effects. In one embodiment, the entire compressed neural network can be fine-tuned. In another embodiment, each compressed layer can be fine-tuned individually. Compared to alternative solutions that require fine-tuning the entire neural network, fine-tuning the compressed layers on an individual basis can reduce the overall computation time.

[0036] Figure 2The selection of a neural network, compression method, and compression ratio according to one embodiment of this disclosure is illustrated in detail. The neural network 202 is selected by the user. The neural network can be selected by the user from a pool of possible neural networks. It should be noted that this method is insensitive to the neural network used and can adapt to any type of selected neural network. As a next step, the user selects which type of compression method and associated compression ratio they wish to use for compression. Figure 2 In the example, two methods are chosen: quantization and pruning. For both methods, a range of k compression ratios is selected. It should be noted that the range of compression ratios can be different for each method: for example, the quantization method can have a compression ratio of k = 3 (i.e., 3 compression ratio options should be calculated per layer), while the pruning method can have a compression ratio of k = 4 (i.e., 4 compression ratio options should be calculated per layer). In one embodiment, the user can select a compression method and a related compression ratio from a pre-compiled table or list that includes different compression methods and different associated compression ratios. The table or list can be stored in a database. The database can be stored locally or on an external server. As a result of the user's selection, the layers (204, 206) of the neural network are compressed according to all possible combinations of compression methods and associated compression ratios.

[0037] Figure 3 The database 300 of step 110 according to an embodiment of the present disclosure is shown. For each combination of compression method m and compression ratio k, the database stores the loss function for each layer of the selected neural network 302. Calculated values ​​and cost functions for (304, 308) The calculated values ​​for (306, 310) are given. The database can store the loss function for each layer-by-layer combination of compression method and compression ratio. and cost function Different neural networks calculate the corresponding values. Storage and The value of can reduce computation time because it is possible to and The value is reused for the selected neural network. For example, if the customer needs a neural network, for said neural network, and If a value has been calculated in the past and stored in the database, it may be possible to reuse such a value. Optionally, the database may store information about use cases that have used the neural network. Information about the calculated values ​​of the selected neural network, loss function, and cost function, as well as the use cases, may be stored in the database in the form of a list or table. The database may be stored locally or on an external server / external cloud.

[0038] Figure 4 The loss function of neural network 402 is shown in detail. and cost function An example of calculating the value. In this example, a compression method is chosen, i.e., m = 1. In this example, the compression method is quantization. The selected compression ratio has a range of k = 3. Then, for each layer of the neural network 402, a total of 3 combinations of compression methods and compression ratios should be calculated. The neural network 402 has n layers represented by boxes 404, 406, and 408. Each layer is then compressed three times: the first time for m = 1 and k = 1 (boxes 410, 412, 414), the second time for m = 1 and k = 2 (boxes 416, 418, 420), and the third time for m = 1 and k = 3 (boxes 422, 424, 426).

[0039] Preferably, in parallel with such calculations, a cost function is also calculated. As described above, the cost function can quantify, for example, the memory footprint or inference latency of the compression layer: such quantities are calculated for each layer via feedback from the hardware target (428). The hardware target can be any type of CPU, processor, or computing unit on which the selected neural network can run. It should be noted that the hardware target can be the same hardware on which the compressed neural network will run, or it can be hardware with similar characteristics to the hardware on which the compressed neural network will run. For the cost function, all combinations between the compression method and the relevant compression ratio will also be calculated: in this way, for each loss function It has a corresponding cost function After calculating (and optionally storing in a database) all values ​​of the cost function and loss function for each layer of the neural network, the method of this disclosure selects a compression method for each layer with a relevant compression ratio value to achieve a global target compression ratio. It should be noted that different compression methods and / or different compression ratios can be selected for each layer of the neural network. For example, for layer i = 1, a compression method m = 1 with a relevant second compression ratio can be selected, while for layer i = 2, a compression method m = 1 with a relevant first compression ratio can be selected. The combination of all selected compression ratios and compression methods for each layer forms the compression strategy for a given neural network. The neural network is then compressed according to the compression strategy. Optionally, the compression strategy can be stored in a database along with the neural network and the use case for which the neural network is computed. This can have the effect of allowing easy retrieval of such information without needing to recalculate the compression strategy.

[0040] exist Figure 5The image shows a system according to a second embodiment of the present disclosure. System 500 includes a user computing device 502 and a server computing system 512 communicatively coupled via a network 524. The user computing device 502 can be any type of computing device, such as a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0041] User computing device 502 includes one or more processors 504 and storage unit 506. The one or more processors 504 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Storage unit 506 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, etc., and combinations thereof. Storage unit 506 includes data instructions 508, which are executed by processor 504 to cause user computing device 502 to perform operations. Specifically, the data instructions can include one or more of the method steps 102 to 118 as described above. Storage unit may optionally include database 522, configured to store one or more neural networks, multiple compression methods, and multiple compression ratios associated with such compression methods. Database may optionally store the cost function and loss function of the computation of one or more neural networks. Database may optionally store compression strategies.

[0042] Similarly, the server computing system 512 includes one or more processors 514 and a storage unit 516. The one or more processors 514 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The storage unit 516 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The storage unit 516 can store data instructions 518, which are executed by the processor to cause the server computing system to perform operations, such as those described in one or more of steps 102 to 118 of this disclosure. The server computing system may include a database 520.

[0043] User computing device 502 may further include one or more user interfaces 510 for receiving user input. User interface 510 allows the user to communicate with network 524 and server computing system 512. The user interface may be connected to one or more devices for enabling communication with the user. For example, user interface 510 may be enabled for coupling to a display, mouse, keyboard, or other input / output device. User interface 510 may be used by the user to send requests to server computing system 512, and / or to select from database 522 one or more of neural networks and / or multiple compression methods and multiple compression ratios associated with such compression methods.

[0044] In one embodiment, one or more neural networks, multiple compression methods, and multiple compression ratios associated with such compression methods can be sent by server computing system 512 to user computing device 502 via network 524 upon a user request executed through user interface 510. The data sent by server computing system is then used by data instructions 508 and processor 504 to calculate a loss function, a cost function, and a compression strategy. In this embodiment, the compression strategy is calculated locally within the user computing device.

[0045] In one embodiment, a compression strategy can be computed in server computing system 512. In this embodiment, one or more neural networks, multiple compression methods, and multiple compression ratios associated with such compression methods can be stored in database 520 or transmitted via a network by user computing device 502. Then, data instructions 518, including steps 108 to 116 of this method, are executed by processor 514 to compute a loss function, a cost function, and a compression strategy. In this embodiment, the compression strategy is computed on the server computing system, i.e., external to the user computing device.

[0046] Other variations and modifications will become apparent to those skilled in the art upon reading this disclosure. Such variations and modifications may involve equivalent and other features known in the field of semiconductor device processing and used in place of or in addition to the features described herein.

[0047] Although the appended claims relate to specific combinations of features, it should be understood that the scope of this specification also includes any novel feature or combination of novel features or any generalized form thereof explicitly or implicitly disclosed herein, regardless of whether it relates to the same specification claimed in any claim or whether it alleviates any or all of the same technical problems as those alleviated by this specification.

[0048] This disclosure relates to elements or features that are “connected” or “coupled” together. As used herein, unless otherwise expressly stated, “connected” means that one element is directly connected to (or directly communicated with) another element, and not necessarily mechanically. Similarly, unless otherwise expressly stated, “coupled” means that one element is directly or indirectly connected to (or directly or indirectly communicated with, electrically or otherwise) another element, and not necessarily mechanically. Therefore, while the schematic diagrams shown in the figures depict an exemplary arrangement of elements, additional intervening elements, devices, features, or components may be present in embodiments of the depicted subject matter.

[0049] Features described in the context of a single embodiment may be provided in combination in a single embodiment. Conversely, for the sake of brevity, the various features described in the context of a single embodiment may also be provided individually or in any suitable sub-combination. The applicant hereby reminds that new claims may be formulated based on such features and / or combinations of such features during the examination of this application or any other application derived therefrom.

[0050] For completeness, the term "comprising" does not exclude other elements or steps, and the term "a (a or an)" does not exclude the possibility that a plurality of, a single processor or other unit may implement the functions of the components described in the claims. Furthermore, reference numerals in the claims should not be construed as limiting the scope of the claims. In addition, the word "may" is used in an permissive sense (i.e., meaning the possibility of...) rather than a mandatory sense (i.e., meaning the necessity). Similarly, the words "include, including, and include" mean including but not limited to. Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish the elements described by such terms. Therefore, these terms are not necessarily intended to indicate a temporal or other priority order of such elements.

Claims

1. A method using a compression strategy employing a neural network, characterized in that, The method includes the following steps: Select the neural network, wherein the neural network comprises multiple layers; Select multiple compression methods, multiple compression ratios associated with the multiple compression methods, and the target global compression ratio of the neural network; Each of the plurality of layers of the neural network is compressed according to each combination of the plurality of compression methods and the plurality of compression ratios; For each of the plurality of layers, calculate the value of the loss function and the value of the cost function; For each of the plurality of layers, the compression strategy corresponding to the target global compression ratio is calculated using the value of the loss function and the value of the cost function; and The neural network is compressed according to the compression strategy.

2. The method according to claim 1, characterized in that, The following steps are also included: For each of the plurality of layers of the neural network, a combination that minimizes the loss function and the cost function is inferred, and the combination is used to compute the compression strategy.

3. The method according to any one of the preceding claims, characterized in that, The plurality of compression methods, the plurality of compression ratios associated with the plurality of compression methods, and the target global compression ratio of the neural network are stored in a database.

4. The method according to any one of the preceding claims, characterized in that, The values ​​of the loss function and the cost function are stored in the database.

5. The method according to any one of the preceding claims, characterized in that, The step of calculating the compression strategy corresponding to the target global compression ratio using the loss function and the cost function is performed using a solver for constrained optimization problems.

6. The method according to any one of the preceding claims, characterized in that, The additional step includes storing the compression strategy in a database.

7. The method according to any one of the preceding claims, characterized in that, The multiple compression methods include one or more of the following: quantization method, pruning method, weight sharing method, tensor decomposition method, or low-rank matrix decomposition method.

8. The method according to any one of the preceding claims, characterized in that, Further fine-tuning steps are applied to each of the plurality of layers in the neural network.

9. The method according to claims 1 to 7, characterized in that, Further fine-tuning steps were taken for the compressed neural network application.

10. A computing system, characterized in that, include: One or more processors; as well as One or more storage units storing data instructions that, when executed by the one or more processors, cause the computing system to: Choose a neural network, wherein the neural network includes multiple layers; Select multiple compression methods, multiple compression ratios associated with the compression methods, and the target global compression ratio of the neural network; Each of the plurality of layers of the neural network is compressed according to each combination of the plurality of compression methods and the plurality of compression ratios; For each of the plurality of layers, calculate the value of the loss function and the value of the cost function; For each of the plurality of layers, the value of the loss function and the value of the cost function are used to calculate the compression strategy corresponding to the target global compression ratio; as well as The neural network is compressed according to the compression strategy.

Images