Method executed by computer for neural network compression, neural network compression apparatus, computer program, and method for manufacturing compressed neural network data

Abstract

To provide a technique for compressing a neural network such that characteristics such as accuracy of the neural network after compression are guaranteed.SOLUTION: The disclosed method is a method executed by a computer for compressing a neural network including a plurality of layers, the method including: receiving setting of a desired value of a first characteristic relating to the neural network after compression; and repeatedly executing a removal step until the first characteristic reaches the desired value. Each of the repeatedly executed removal steps includes selecting a compression target layer from the plurality of layers and removing neurons in the compression target layer.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a computer-implemented method for compressing neural networks, a neural network compressor, a computer program, and a method for producing compressed neural network data. [Background technology]

[0002] Patent Document 1 discloses a method for compressing a neural network. The compression method of Patent Document 1 involves pruning selected pruning targets and reconstructing the pruned neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-022050 [Non-patent literature]

[0004] [Non-Patent Document 1] Pruning Ratio Optimization with Layer-Wise Pruning Method for Accelerating Convolutional Neural Networks, (Koji Kamma, Sarimu Inoue, and Toshikazu Wada), IEICE Transactions on Information and Systems, Vol.E105-D, No.1, 2022, pp.161-169 Summary of the Invention

[0005] Conventionally, there is no method that guarantees the accuracy and other characteristics of a neural network after compression. Therefore, a method for compressing a neural network so that the accuracy and other characteristics of the neural network after compression are guaranteed is desired (Problem 1).

[0006] Furthermore, when compressing neural networks, it is necessary to rewrite the computer program for compression to match the model of the neural network to be compressed. This is because neural network models have a variety of structures. As a result, neural network compression must be performed manually to match the model of the neural network to be compressed, making it difficult to automate the compression. Therefore, technology that makes it easier to automate the compression of neural networks is desired (Problem 2).

[0007] The present disclosure relates to solving either one of the above problems 1 and 2.

[0008] One aspect of the present disclosure is a computer-implemented method for compressing a neural network having multiple layers. The disclosed method includes accepting a desired value for a first characteristic of the compressed neural network and repeatedly performing a removal process until the first characteristic reaches the desired value, where each of the repeated removal processes includes selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed.

[0009] Another aspect of the present disclosure is an apparatus for compressing a neural network having multiple layers. The disclosed apparatus is configured to perform a process including accepting a desired value of a first characteristic for the compressed neural network and repeatedly performing a removal process until the first characteristic reaches the desired value, where each of the repeated removal processes includes selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed.

[0010] Another aspect of the present disclosure is a computer program that causes a computer to execute a process for compressing a neural network having multiple layers, the process including accepting a desired value of a first characteristic of the compressed neural network and repeatedly executing a elimination step until the first characteristic reaches the desired value, wherein each of the repeated elimination steps includes selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed.

[0011] Another aspect of the present disclosure is a manufacturing method. The disclosed manufacturing method is for manufacturing neural network data in which a neural network having multiple layers is compressed. The disclosed manufacturing method includes accepting a desired value for a first characteristic of the compressed neural network, and repeatedly executing a removal step until the first characteristic reaches the desired value, where each of the repeatedly executed removal steps includes selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed. The manufacturing method is executed by a computer program.

[0012] Further details will be described in the following embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing the configuration of a neural network compression device and a neural network utilization device. [Figure 2] FIG. 2 is a flowchart of the process in the neural network compression device. [Figure 3] FIG. 3 is a flowchart of the compression process. [Figure 4] FIG. 4 shows an example of neuron removal. [Figure 5] FIG. 5 is a graph showing the relationship between the amount of calculation reduced and the amount of accuracy reduced. [Figure 6] FIG. 6 is a flowchart of the identification process. [Figure 7] FIG. 7 shows another example of neuron removal. [Figure 8] FIG. 8 shows another example of neuron removal. [Figure 9] FIG. 9 shows another example of neuron removal. [Figure 10] FIG. 10 is an explanatory diagram of neuron labeling. [Figure 11] FIG. 11 is an explanatory diagram of neuron labeling. [Figure 12] FIG. 12 is a diagram illustrating an example of a part of a computation graph. [Figure 13] FIG. 13 is a diagram illustrating an example of parsing an expression in a computation graph. [Figure 14] FIG. 14 is a diagram illustrating an example of parsing an expression in a computation graph. [Figure 15] FIG. 15 is a graph showing the experimental results. DETAILED DESCRIPTION OF THE INVENTION

[0014] 1. Overview of a computer-implemented method for compressing neural networks, a neural network compression device, a computer program, and a method for producing compressed neural network data

[0015] (1) A method according to an embodiment may be a computer-implemented method for compressing a neural network having multiple layers. The method according to an embodiment may include accepting a desired value for a first characteristic of the compressed neural network and repeatedly performing a removal step until the first characteristic reaches the desired value. Each of the repeated removal steps may include selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed. In this case, the neural network can be compressed so that the first characteristic of the compressed neural network is guaranteed.

[0016] (2) In each of the repeatedly executed removal steps, the layer to be compressed may be selected based on a second characteristic of the neural network when a neuron in the layer to be compressed is removed, and the second characteristic may include a relationship between the amount of calculation reduced by removing the neuron and the amount of change in the first characteristic due to the removal of the neuron.

[0017] (3) The first characteristic may include the accuracy of the neural network. In this case, the neural network can be compressed so that the accuracy of the neural network after compression is guaranteed. Note that the first characteristic may be the computational complexity of the neural network or the removal rate of neurons in the neural network.

[0018] (4) In each of the repeatedly executed removal steps, the layer to be compressed may be selected based on a second characteristic of the neural network when neurons in the layer to be compressed are removed. The first characteristic may include accuracy of the neural network. The second characteristic may include a relationship between the amount of calculation reduced by removing the neuron and the amount of loss in accuracy due to the removal of the neuron.

[0019] (5) In each of the repeatedly performed removing steps, the compression target layer may be selected based on a plurality of second characteristics, each of which may be a second characteristic obtained for a different compression target layer.

[0020] (6) Each of the repeatedly executed removal steps may include removing neurons in the selected layer to be compressed and removing neurons in other layers that are dependent on the neurons in the layer to be compressed.

[0021] (7) According to an embodiment, the method may include identifying the dependencies in the neural network. The identifying the dependencies may be performed by structurally analyzing the computation graph of the neural network using a parser. Identifying the dependencies facilitates automated compression of the neural network.

[0022] (8) An apparatus according to an embodiment may be an apparatus for compressing a neural network having multiple layers. The apparatus according to an embodiment may be configured to execute a process including accepting a desired value of a first characteristic of the compressed neural network and repeatedly executing a elimination step until the first characteristic reaches the desired value. Each of the repeatedly executed elimination steps may include selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed.

[0023] (9) A computer program according to an embodiment may cause a computer to execute a process for compressing a neural network having multiple layers. The process may include accepting a desired value for a first characteristic of the compressed neural network and repeatedly executing a elimination step until the first characteristic reaches the desired value. Each of the repeatedly executed elimination steps may include selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed.

[0024] (10) A method according to an embodiment may be a method for producing neural network data in which a neural network having multiple layers is compressed. The method according to an embodiment may include receiving a desired value for a first characteristic of the compressed neural network, and repeatedly performing a removal step until the first characteristic reaches the desired value. Each of the repeatedly performed removal steps may include selecting a layer to be compressed from the multiple layers and removing neurons in the layer to be compressed. The production method may be executed by a computer program.

[0025] 2. Examples of computer-implemented methods for compressing neural networks, neural network compression devices, computer programs, and methods for producing compressed neural network data

[0026] Hereinafter, the embodiments will be described in more detail with reference to the drawings.

[0027] Fig. 1 shows devices 10, 100 according to the embodiment. The devices 10, 100 shown in Fig. 1 include a neural network compression device (hereinafter referred to as "compression device") 10 and a neural network utilization device (hereinafter referred to as "utilization device") 100.

[0028] The compression device 10 according to the embodiment executes a compression process 21 for compressing the neural network N1 to reduce its size. The method (neural network compression method) implemented by executing the compression process 21 is also a method for producing a compressed neural network or a method for producing compressed neural network data.

[0029] A neural network is an artificial computation mechanism in which multiple artificial neurons (also referred to as "nodes") are connected. The neural network is, for example, a deep neural network (DNN). The DNN may be, for example, a fully connected neural network (FCN) or a convolutional neural network (CNN). Hereinafter, the neural network N1 to be compressed will be referred to as the "original neural network," and the compressed neural network N2 will be referred to as the "compressed neural network." Note that the compression device 10 according to the embodiment can also perform processing for machine learning (deep learning) of the original neural network N1. The compression device 10 compresses the trained original neural network N1.

[0030] The compression device 10 is configured by a computer having one or more processors 20 and a storage device 30. The one or more processors 20 include, for example, a graphic processing unit (GPU). The one or more processors 20 may further include a CPU. A massively parallel computing mechanism such as a GPU is suitable for large-scale calculations for performing processing related to a large-scale neural network.

[0031] The storage device 30 stores a computer program 31 that is executed by the processor 20. The processor 20 executes the computer program 31 to perform a compression process 21. The compression process 21 is a process for compressing and downsizing the original neural network N1 to obtain a compressed neural network N2. The compression includes pruning and reconstruction (see "REAP" in Patent Document 1).

[0032] The processor 20 also executes a computer program 31 to perform a classification process 22. The classification process 22 is a process for identifying dependencies between multiple layers included in the original neural network N1.

[0033] As described above, the method according to the embodiment includes two steps: an identification process 22 for identifying dependencies between layers in the original neural network N1, and a compression process 21 for compressing the original neural network N1.

[0034] As shown in Fig. 2, the compression device 10 may execute an identification process 22 after acquiring an original neural network N1 and a compression dataset and a validation dataset. The compression dataset and the validation dataset are datasets to be input to a neural network (the original neural network N1, or a neural network during or after compression). The compression dataset and the validation dataset may be different from each other or may be the same.

[0035] The compression process 21 is executed after the identification process 22. Prior to the compression process 21, a desired value of a desired first characteristic (e.g., the accuracy of the neural network) is set by a user or the like (see step S301 in FIG. 3 described later). As an example, the compression device 10 according to the embodiment (1) automatically identifies layers to be compressed (layers to be compressed and other layers that are dependent on the layers to be compressed) among the layers constituting the original neural network N1, (2) automatically performs compression so as to guarantee the desired value of the first characteristic (e.g., the desired accuracy), and (3) outputs the compressed neural network.

[0036] In the compression process 21, if a neuron in a certain layer is deleted, it is necessary to delete weight parameters and neurons in other layers that are dependent on that layer. In other words, the compression process 21 must perform compression on a certain layer and other layers that are dependent on that layer as a whole. However, the types of dependencies that exist depend on the structure of the neural network, and therefore may differ from neural network to neural network. Automating compression requires automatically dealing with dependencies that may differ depending on the structure of the neural network.

[0037] To compress neural networks of various structures, it is desirable to identify a layer to be compressed (first layer) and other layers (second layers) that are dependent on the layer to be compressed as a group of layers that need to be compressed together for each neural network to be compressed. The identification process 22 automatically identifies such a group of dependent layers. Then, in the subsequent compression process 21, if a layer to be compressed (first layer) has a dependent relationship with other layers (second layers), the group of dependent layers is compressed.

[0038] As described below, the compression process 21 includes a removal process (see steps S303 and S304 in FIG. 3) that is repeatedly executed multiple times (multiple cycles). In one cycle of the repeatedly executed removal process, compression of a selected compression target layer is executed. By repeating the removal process, compression gradually progresses and the compression ratio increases. In each cycle of the repeatedly executed removal process, the selected compression target layer may be different. Details of the identification process 22 and the compression process 21 will be described later.

[0039] Returning to Fig. 1, the storage device 30 can store data (compressed neural network data) N20 representing the compressed neural network N2 produced by the compression process 21. The compressed neural network data N20 is data consisting of various parameters (weight parameters, connection relationships, etc.) that represent the compressed neural network N2.

[0040] The storage device 30 can store dependency data 32 indicating dependencies identified by the classification process 22. The storage device 30 can also store data 33 used in the classification process 22. The data 33 used in the classification process 22 is, for example, a list of syntax (e.g., functions) indicating dependencies between layers in a neural network. The storage device 30 can also store a data set 34 for compression or accuracy verification. The storage device 30 can also store other data as needed.

[0041] The compression device 10 can output the compressed neural network data N20 to a neural network engine, etc. The compressed neural network data N20 is read into the neural network engine, causing the neural network engine to function as the compressed neural network N2.

[0042] The neural network compression performed by the neural network compression device 10 of the embodiment can be provided to a user (client terminal) as, for example, a web service (SaaS). For example, a server operating as the neural network compression device 10 can acquire an original neural network N1 from a client terminal via a network such as the Internet, and compress the original neural network N1 to generate a compressed neural network N2. Data N20 of the generated compressed neural network N2 is provided to the client terminal (e.g., the utilization device 100) via the network.

[0043] A user can set a desired value (desired value) as a first characteristic, such as the accuracy of the compressed neural network. The neural network compression device 10 can accept, for example, a desired value for the accuracy of the compressed neural network N2 as the first characteristic. The desired value for accuracy is, for example, 90%. In this case, if the accuracy of the original neural network N1 before compression is 100%, the user is willing to tolerate a reduction in accuracy of up to 10% due to compression. Note that in this case, setting an accuracy of 90% as the desired value is equivalent to setting an allowable accuracy reduction of 10% as the desired value.

[0044] The neural network compression device 10 repeatedly removes neurons until the first characteristic, such as the accuracy of the neural network after compression, reaches the desired value. The accuracy of the neural network gradually decreases as neurons are removed. If an accuracy of 90% is set as the desired value, however, the neural network will repeatedly remove neurons as long as the accuracy does not fall below 90%. In other words, the neural network will repeatedly remove neurons as long as the accuracy does not drop below 10%. Therefore, compression will be performed until the accuracy reaches the desired value of approximately 90%.

[0045] As described above, the compression device 10 of the embodiment can automatically compress a neural network and can guarantee the characteristics (e.g., accuracy) after compression. Therefore, the compression device 10 of the embodiment can operate as, for example, an automatic neural network compression device with guaranteed accuracy, and can provide users with a service of automatic neural network compression with guaranteed accuracy via a network.

[0046] Conventionally, to compress a neural network, it was necessary to manually identify the layer to be compressed and its dependent layers, associate the indices of the neurons contained in those layers, and write a program to execute the compression. Performing this work on a large neural network model with hundreds to thousands of layers requires significant effort, making it extremely difficult to do manually. Furthermore, adjusting the compression ratios for the many layers to minimize loss of accuracy was also a time-consuming task. By using the compression device 10 according to the embodiment, the above series of steps are automated, simplifying the use of compression technology.

[0047] The utilization device 100 has a neural network engine that reads the compressed neural network data N20 and functions as a compressed neural network N2. The neural network engine includes, for example, a processor 200 and a storage device 300. The processor 200 may be, for example, a low-power CPU in an embedded system. The compressed neural network data N20 is smaller in size than the data of the original neural network N1, and therefore can be processed by a low-power CPU.

[0048] An embedded system is not a general-purpose computer system, but a computer system for a specific purpose, such as computer systems in household devices such as smartphones and home appliances, industrial devices such as industrial robots, various medical devices, vehicles such as automobiles and drones, and other devices. Embedded systems often use low-power CPUs as processors, but the compressed neural network data N20 is easy to execute due to its small data size.

[0049] The compressed neural network N2 is used for applications such as image and voice conversion, segmentation, and classification. More specifically, it can be used to extract necessary information from images of objects, such as measuring the number of customers in a store, analyzing gender and age demographics, counting vehicles, and analyzing vehicle types. The original neural network N1 is large-scale and requires high computational costs, making it difficult to run on embedded systems. However, the compressed neural network N2 is smaller in size, making it easier to run on embedded systems.

[0050] FIG. 3 shows an example of the procedure of the compression process 21 according to the embodiment. In the compression process 21 according to the embodiment, pruning is performed as an example. Pruning is the removal (deletion) of unnecessary neurons (neurons that may be deleted). In the following description, "compression" means "pruning" as an example, but is not limited to "pruning."

[0051] Prior to explaining Fig. 3, neural network pruning (neuron deletion) will be explained based on Fig. 4. The neural network shown in Fig. 4 is, as an example, a densely connected DNN (i.e., FCN). For ease of understanding, the explanation will be given mainly on the case where the fully connected layer (layer A and layer B in Fig. 4(A)) is the compression target, but other layers (for example, convolutional layers and attention layers) can also be the compression target.

[0052] In FIG. 4(A), circles represent neurons, and weight parameters are assigned to lines representing connections from neurons in one layer to neurons in the next layer.

[0053] The neural network shown in Figure 4(A) comprises layers A and B. Here, a "layer" refers to a "collection of neurons" where some processing is performed in parallel in the calculation flow inside the neural network model. In addition, in a certain layer, the weight parameters given to the connections extending from the neurons belonging to that layer to the next layer also belong to that layer. For example, the weight parameter W given to the connection extending from layer A to layer B is a also belongs to layer A and will be called the "weight parameters of layer A" or the "weight matrix of layer A."

[0054] For example, layer A in FIG. 4(A) includes three neurons a1, a2, and a3, and layer B, which is the next layer to layer A, includes three neurons b1, b2, and b3. Layer A also includes weight parameters W given to the connections extending from the neurons a1, a2, and a3 in layer A to the neurons b1, b2, and b3 in layer B. a Furthermore, layer B includes weight parameters W given to connections extending from each neuron b1, b2, and b3 in layer B to each neuron c1, c2, and c3 in the next layer. b In Figure 4(A), the vector that lists the response values ​​of all neurons a1, a2, and a3 in layer A is shown as vector a, and the vector that lists the response values ​​of all neurons b1, b2, and b3 in layer B is shown as vector b. In the neural network of Figure 4(A), b=W a a,c=W b The calculation is carried out as in b.

[0055] For example, as shown in Figure 4(B), if we remove neuron b1 in layer B of the neural network in Figure 4(A), then c = W b In the calculation of b, the corresponding element in vector b (the element corresponding to b1) is deleted, and the weight matrix W b The corresponding line is deleted in b=Wa In the calculation of a, W a The line corresponding to b1 is deleted from

[0056] In this way, when compression is performed in layer B, the weight matrix W b In addition, the weight matrix W of the layer A just before a In this way, in order to automate the compression of a neural network, it is necessary to not only identify the layer B (first layer) to be compressed, but also to identify other layers A (second layers) that are dependent on it. Other examples of inter-layer dependencies will be described later.

[0057] Now, returning to FIG. 3, the compression process 21 according to the embodiment will be described.

[0058] The neural network N1 is composed of multiple layers. In such a neural network N1, to perform compression so as to obtain a desired characteristic (first characteristic), it is necessary to perform compression while adjusting the compression rate in each layer (each layer to be compressed). For example, if the first characteristic is the accuracy of the neural network, to perform compression so as to obtain the desired accuracy, it is necessary to repeatedly perform compression while adjusting the compression rate in each layer (each layer to be compressed). The compression rate refers to the proportion of neurons that are deleted or the proportion of neurons that are left untouched.

[0059] In this embodiment, as an example of a method for selecting and compressing a layer to be compressed, a layer (layer to be compressed) that is determined to have little effect on the accuracy (first characteristic) of the entire neural network even if compressed is selected, and a small number (one or more) of neurons are deleted from that layer to be compressed. This procedure is repeated until just before the accuracy falls below the desired accuracy (first characteristic), thereby adjusting the compression rate of each layer. Note that instead of accuracy, the computational amount of the neural network (the amount of computation reduced by compression) or the neuron removal rate (compression rate) in the neural network may be used as the first characteristic, and the above procedure may be repeated until the desired computational amount is achieved or the desired removal rate (compression rate) is reached. Here, a smaller computational amount is preferable, and a higher removal rate is preferable.

[0060] FIG. 3 shows an example of a method for adjusting the compression rate of each layer by repeatedly applying the above-described procedure until the compression rate falls below the desired accuracy (first characteristic).

[0061] 3, the compression device 10 receives from the user a setting of a desired accuracy s for the compression neural network N2. The compression device 10 also receives a setting of a threshold value (threshold value for step S304) for a permissible loss of accuracy u in one cycle of repetition. For example, if the permissible loss of accuracy u in one cycle is 2%, compression accompanied by a loss of accuracy of about 2% is repeated multiple times until the accuracy falls below the desired accuracy s.

[0062] After step S301, the compression device 10 automatically performs all steps. In step S302, the compression device 10 calculates the accuracy of the original neural network N1 using the accuracy verification data set. The accuracy is calculated from the output when the accuracy verification data set is input to the original neural network N1.

[0063] In steps S303 and S304, the compression device 10 executes the removal step.

[0064] In step S303, the compression rate r is set to an arbitrary value for one of the layers that can be compressed (let's call this "layer A"), and compression is performed. The accuracy of the compressed accuracy verification dataset and the amount of calculation reduction due to compression are recorded. A similar operation is performed for one layer A at multiple compression rates. For example, compression is performed at each of three compression rates r = 0.1, 0.2, and 0.3 (see Figure 5). This results in a graph showing the relationship between the "reduced amount of calculation" and the "amount of accuracy reduction" in layer A, as shown by the solid line in the graph in Figure 5.

[0065] The same process is performed for each layer that can be compressed (e.g., layer A, layer B, C, etc.). As a result, a graph showing the relationship between the "reduced amount of calculation" and the "amount of loss of accuracy" for each of layers A and B is obtained, as shown in the graph in Figure 5. The "reduced amount of calculation" and the "amount of loss of accuracy" are examples of the second characteristic used to select the layer to be compressed.

[0066] In step S304, when compression is performed until the threshold u (see step S301) set in advance by the user is reached, the layer with the largest "reduced amount of calculation" (the layer where the amount of calculation can be reduced the most) is selected as the layer to be compressed (the layer from which neurons are to be deleted). In the case of Figure 5, layer A has a larger "reduced amount of calculation" than layer B, so layer A is selected as the layer to be compressed. The selected layer to be compressed is compressed.

[0067] Compression is performed not only on the selected layer to be compressed, but also on other layers that have dependencies on the layer to be compressed. In other words, neuron and weight parameters in other layers that have dependencies are also deleted. This also applies to the compression in step S304.

[0068] As a compression method, for example, a method of minimizing the "reconstruction error" in the layer next to each layer to be compressed ("REAP" described in Patent Document 1) can be used. However, the compression method is not limited to "REAP" and other methods may be used.

[0069] In step S305 after the removal process (steps S303 and S304), the compression device 10 uses the accuracy verification data set to determine the accuracy of the compressed neural network. In step S306, the compression device 10 determines whether the accuracy of the compressed neural network is below a desired accuracy value s.

[0070] If the accuracy of the compressed neural network is below the desired accuracy value s ("YES" in step S306), the compression device 10 returns the compressed neural network to the state it was in immediately before the last execution of step S304, and terminates the process (step S307). That is, if "YES" in step S306, the compression device 10 restores the neurons deleted when S304 was last executed, and also restores the weight parameters deleted, and terminates the process.

[0071] If the accuracy of the neural network after compression is not below the desired accuracy value s ("NO" in step S306), the process returns to step S303. Therefore, the elimination process of steps S303 and S304 is repeated until the accuracy reaches the desired accuracy value s. The above procedure is advantageous because the compression rate of each layer is adjusted based on the accuracy.

[0072] Fig. 6 shows an example of the procedure of the above-mentioned identification process 22. In the removal process (steps S303 and S304), not only the selected compression target layer but also other layers that have a dependency relationship with the compression target layer are compressed. Prior to a detailed explanation of Fig. 6, examples of dependency relationships between layers other than those shown in Fig. 4 will be explained with reference to Figs. 7 to 9.

[0073] In Figure 7, layer P exists between layer A and layer B, which are fully connected layers. The neurons in layer P and layer B are connected one-to-one. In this case, for example, if b1 is deleted, p1 also becomes unnecessary and should be deleted. Therefore, the corresponding row of the weight matrix Wa of layer A connected to layer P should also be deleted. Also, if layer P has some parameter W p If you have W p The corresponding elements of should also be deleted. Examples of layers that have one-to-one connections between neurons, such as layer P, include batch regularization layers and depth-wise convolutional layers.

[0074] Figure 8 shows an example of compression in a layer where addition processing is performed in addition to matrix operations. In this example, the response value in layer B is calculated using not only the response value of the immediately preceding layer A but also the response value of another layer P, as follows: b = W a It is calculated as a+p. Here, we are adding vectors together, so we use three vectors (b,W a The number of dimensions of layers a and p) must be the same. Therefore, when deleting b1, not only the corresponding element of Wa but also neuron p1 must be deleted at the same time. This additive dependency between vectors such as those in layers B and P often appears in well-known models (neural networks) such as ResNet. Note that this type of additive dependency can also be considered an example of the dependency due to one-to-one connections between neurons shown in Figure 7.

[0075] Figure 9 shows an example where a process is included to concatenate the response values ​​of neurons in multiple layers. Vector d, which lists the response values ​​of layer D, is the concatenation of c and b. Therefore, d1 and d2 are connected one-to-one with c1 and c2, and d3 and d4 are connected one-to-one with b1 and b2. In the example in Figure 9, if b1 is deleted as in the example in Figure 7, d3 must also be deleted at the same time. Also, d3 is located third from the top in layer D, while b1 is located first from the top in layer B. In this way, even when neurons in different layers correspond one-to-one, their indices (the ordinal number of each neuron in each layer) may differ. In such cases, it is necessary to associate the neuron indices.

[0076] 10 and 11 show an example of a method for identifying a layer to be compressed and its dependent layers, and for associating neuron indices, which is automatically performed by the compression device 10.

[0077] FIG. 10 shows an example in which a process for connecting multiple layers is included, as in FIG.

[0078] First, as shown in the initial state of Figure 10(A), labels are assigned to two neurons in layer A, which is a fully connected layer. In this example, the labels are 1 and 2.

[0079] Next, as shown in Figure 10(B), labeling is performed in layers B and C. Because layers A and B, and layers B and C, are tightly connected, new labels (here, 3, 4 and 5, 6) are given to the neurons in layers B and C so as not to overlap with already labeled neurons.

[0080] In Figure 10(C), labeling is performed in layer D. The top two neurons in layer D are connected one-to-one to neurons in layer C, and the bottom two neurons in layer D are connected one-to-one to neurons in layer B. In this case, the neurons in layer D are given the same labels as the neurons in layers B and C that they are connected to.

[0081] Figure 10(D) shows the process of assigning a label to the neuron in layer E. Because layers D and E are tightly connected, a new label (7,8) is assigned to the neuron in layer E.

[0082] Consider the case where compression is performed on a neural network labeled as above, with layer B as the target layer. If a neuron labeled "3" is deleted in layer B, the neuron labeled "3" in layer D will also be deleted at the same time. In other words, neurons with the same label are deleted. At this time, elements corresponding to the deleted neuron are removed not only from the weight matrix of layer B, but also from the weight matrices of layers A and D. Therefore, if layer B is the target layer for compression, it can be seen that the layers that are dependent on it are layers A and B. Furthermore, by looking at the labels assigned to each neuron, it is possible to determine the index correspondence.

[0083] Next, let us look at the example in Figure 11. Figure 11 shows an example in which the response value of a neuron in layer B is calculated based on the response values ​​of two layers (layer A and layer P), as also shown in Figure 8. In this case, the neurons in layer B are labeled twice, which causes a problem of label conflict, and we need to address this.

[0084] FIG. 11(A) shows the initial state, where labeling in layers A and P has already been completed.

[0085] Figure 11(B) shows labeling of layer B. Here, because layers A and B have dense connections, new labels (7, 8, 9) are given to the neurons in layer B that do not overlap with other neurons.

[0086] Here, a label conflict occurs. Because neurons in layer B and neurons in layer P are connected one-to-one, the labels of the connected neurons should be the same. However, the labels of the neurons in layer P are 1, 2, and 3, while the neurons in layer B have already been given the labels 7, 8, and 9. Therefore, as shown in Figure 11(C), the labels of the neurons in layer P are changed from 1, 2, and 3 to 7, 8, and 9, respectively. If there are neurons in layers other than layer P that are labeled 1, 2, or 3, their labels are also changed to 7, 8, and 9, respectively. In this way, label conflicts are avoided.

[0087] Figure 11(D) shows the process of assigning labels to neurons in layer C. Because layers B and C are tightly connected, new labels (10, 11, 12) are assigned to neurons in layer C.

[0088] 10 and 11 are the basic procedures for establishing inter-layer dependencies and labeling neurons. The compression device 10 scans each layer of the model of the original neural network N1, and if a certain layer A is a fully connected layer, it selects layer A as the layer to be compressed. At the same time, it identifies layers that are dependent on layer A using the above procedures and labels each neuron.

[0089] When performing compression, the compression device 10 (1) selects one layer to be compressed, (2) selects one neuron to be deleted in that layer, (3) searches for and deletes neurons with the same label as the selected neuron throughout the model, (4) deletes the weight parameters corresponding to the deleted neuron in all layers from which neurons have been deleted, and (5) similarly deletes weight parameters in the layer immediately preceding each layer from which neurons have been deleted. In this way, by using the neuron labels, it is possible to identify layers that depend on each layer to be compressed and associate neuron indices.

[0090] 6, based on the above explanation, a procedure for dependency relationship identification processing 22 will be described. Here, dependency relationships are identified using a syntax analyzer.

[0091] First, the compression device 10 generates a computation graph of the original neural network N1 (step S601). The computation graph describes the flow of computation within the neural network. The compression device 10 writes out the computation graph as, for example, a text file. The computation graph is analyzed by a parser.

[0092] Figure 12 shows (part of) the computation graph of VGG-16, a DNN model, written out using the functions of Pytorch, a DNN framework. Note that although Pytorch is used here, it is not limited to Pytorch, and other frameworks such as Tensorflow and ONNX can also be used as long as they have the function to write out the computation graph of a neural network.

[0093] The compression device 10 analyzes the computation graph using a parser (step S602). The parser analyzes the content of the text describing the computation flow of the neural network to detect what functions are included in the computation graph, etc. The parser can identify dependencies through the parsing. Neurons are labeled based on the dependencies (see FIGS. 10 and 11). Dependency data indicating the dependencies in the neural network is generated through the labeling (step S603). The dependency data is stored in a storage device (step S604).

[0094] The parser is generated using PLY, a parser generator that runs on Python. PLY is a Python library that is a port of Lex, which generates lexical analyzers, and Yacc, a parser generator, to run on Python. However, other parser generators can be used as long as they support LALR(1) grammars.

[0095] 13 and 14 show an example of dependency identification using a parser.

[0096] FIG. 13 shows an example of identifying a one-to-one correspondence. FIG. 13 shows a computation graph for determining the response value of a layer named "%input.33" from the response value of a layer named "%input.31" in a certain neural network. When this computation graph is parsed, it is found that it contains a maximum pooling function called "aten::max_pool2." The "aten::max_pool2" function is registered in the syntax list 33 as one of the functions corresponding to a one-to-one correspondence. Therefore, based on the "aten::max_pool2" function detected by the syntax analysis and the syntax list 33, the compression device 10 can identify that there is a one-to-one correspondence between the neuron in the layer named "%input.31" and the neuron in "%input.33."

[0097] FIG. 14 shows an example of identifying a dense connection. FIG. 14 shows a computation graph for determining the response value of a "%200" neuron from the response value of a "%input" neuron in a certain neural network. When this computation graph is parsed, it is found that it contains a matrix multiplication function called "aten::linear." The "aten::linear" function is registered in the syntax list 33 as one of the functions corresponding to a dense connection. Therefore, based on the "aten::linear" function detected in the syntax analysis and the syntax list 33, the compression device 10 can identify that the "%input" neuron and the "%200" neuron have a dense connection.

[0098] In this way, by registering functions corresponding to each type of dependency relationship in the list 33, the compression device 10 can identify the type of dependency relationship between neurons from the functions obtained by syntax analysis.

[0099] In the formula of the computation graph in Figure 13, %113 to %13 are hyperparameters of the maximum pooling function, and in the formula of the computation graph in Figure 14, %weight and %bias are a weight matrix and a bias vector, which are not necessary for finding the correspondence between neurons.

[0100] Using the compression device 10 according to the embodiment, compression was performed on a DNN model for image classification called Mobilenet v2. FIG. 15 shows the results. 5,000 pieces of data randomly extracted from ImageNet were used as the data to be used for compression. The same 5,000 pieces of data were also used for accuracy verification. Mobilenet v2 is a model trained using a large-scale image classification dataset called ImageNet.

[0101] Mobilenet v2 has 35 layers to be compressed, and these and the layers that depend on them could be identified using the identification process 22 according to the embodiment.

[0102] In addition, the accuracy of the above 5,000 sheets of data was compared in the following cases: (1) when compression was performed while adjusting the compression rate of each layer using the compression device 10 of the embodiment (Experimental Example 1); (2) when compression was performed by manually fixing the compression rates of all layers to a constant value (Experimental Example 2); and (3) when compression was performed using the compression rate adjustment method of Non-Patent Document 1 and the compression method of Patent Document 1 (Experimental Example 3).

[0103] While the accuracy of MobileNet v2 before compression was 0.8244, in Experimental Example 1, the desired accuracy value t (guaranteed accuracy t) was set to three values: t = 0.7244, 0.6244, and 0.5244. The results are shown in Figure 15. In Figure 15, the horizontal axis represents FLOPs (the number of multiplications required to process one piece of data), which indicates the computational complexity of the model, and the vertical axis represents the accuracy for the accuracy verification data. It can be seen that the accuracy of the compressed model is greatly improved by using the compression device according to the embodiment.

[0104] The present invention is not limited to the above-described embodiment, and various modifications are possible. [Explanation of symbols]

[0105] 10: Neural network compression device 20: Processor 21: Compression processing 22: Identification process 30: Storage device 31: Computer Program 32: Dependency data 33: List 34: Dataset 100: Equipment used 200: Processor 300: Storage device N1: Original neural network N2: Compressed Neural Network N20: Compressed neural network data

Claims

1. 1. A computer-implemented method for compressing a neural network having multiple layers, comprising: Accepting a desired value of a first characteristic for the compressed neural network; performing the removing step repeatedly until the first characteristic reaches the desired value. Prepared for this, Each of the repeatedly executed removal steps includes selecting a layer to be compressed from the plurality of layers, and removing neurons in the layer to be compressed. A computer-implemented method for compressing neural networks.

2. In each of the repeatedly executed removal steps, the layer to be compressed is selected based on a second characteristic of the neural network when neurons in the layer to be compressed are removed; the second characteristic includes a relationship between a calculation amount reduced by removing the neuron and an amount of change in the first characteristic due to removing the neuron. The method of claim 1.

3. the first characteristic includes accuracy of the neural network; The method of claim 1.

4. In each of the repeatedly executed removal steps, the layer to be compressed is selected based on a second characteristic of the neural network when neurons in the layer to be compressed are removed; the first characteristic includes accuracy of the neural network; the second characteristic includes a relationship between a calculation amount reduced by removing the neuron and an amount of loss in accuracy due to removing the neuron. The method of claim 1.

5. In each of the repeatedly performed removing steps, the compression target layer is selected based on a plurality of second characteristics; The plurality of second characteristics are second characteristics obtained for different compression target layers, 5. The method according to any one of claims 2 to 4.

6. each of the repeatedly executed removal steps includes removing a neuron in the selected layer to be compressed and removing a neuron in another layer that is dependent on the neuron in the layer to be compressed; The method of claim 1.

7. further comprising identifying the dependencies in the neural network; The identifying of the dependencies is performed by structurally analyzing the computation graph of the neural network using a parser. The method of claim 6.

8. 1. A compressor for a neural network having multiple layers, comprising: Accepting a desired value of a first characteristic for the compressed neural network; performing the removing step repeatedly until the first characteristic reaches the desired value. configured to perform a process comprising: Each of the repeatedly executed removal steps includes selecting a layer to be compressed from the plurality of layers, and removing neurons in the layer to be compressed. Neural network compressor.

9. A computer program that causes a computer to execute a process for compressing a neural network having multiple layers, the computer program comprising: The process comprises: Accepting a desired value of a first characteristic for the compressed neural network; performing the removing step repeatedly until the first characteristic reaches the desired value. Prepared for this, Each of the repeatedly executed removal steps includes selecting a layer to be compressed from the plurality of layers, and removing neurons in the layer to be compressed. Computer program.

10. A method for producing compressed neural network data for a neural network having multiple layers, comprising: Accepting a desired value of a first characteristic for the compressed neural network; performing the removing step repeatedly until the first characteristic reaches the desired value. Prepared for this, Each of the repeatedly executed removal steps includes selecting a layer to be compressed from the plurality of layers, and removing neurons in the layer to be compressed; The manufacturing method is carried out by a computer program. A method for producing compressed neural network data.

Images