MODEL GENERATION METHOD, MODEL GENERATION PROGRAM, MODEL GENERATION DEVICE, AND DATA PROCESSING DEVICE

Abstract

To provide a method for generating a model which enhances processing speed of a convolutional neural network.SOLUTION: A model generation method, which is executed by a processor and generates a machine learning model by replacing a convolutional layer in a convolutional neural network with a matrix decomposed decomposition layer, includes: rearranging a weighting parameter constituting an original layer defined in the convolutional layer before replacement so as to constitute an equivalent weighting matrix equivalent to a weighting matrix product being matrix product of a weighting parameter constituting the decomposition layer; extracting a plurality of ranks by matrix decomposing the equivalent weighting matrix; and selecting at least one rank and constructing a decomposed layer on the basis of convolution in a weighting matrix product corresponding to the selected rank.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to a model generation technique for generating a machine learning model of a convolutional neural network. [Background technology]

[0002] The model generation technology disclosed in Patent Document 1 reduces a machine learning model by decomposing a weight matrix composed of weight parameters in a convolutional layer of a convolutional neural network and then lowering its rank. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-155010 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the model generation technology disclosed in Patent Document 1 performs matrix decomposition and rank reduction while maintaining the layer structure of the original convolutional layer. In this case, there is a limit to increasing the processing speed of the convolutional neural network, which is becoming increasingly complex as a machine learning model.

[0005] An object of the present disclosure is to provide a model generation method for increasing the processing speed of a convolutional neural network. Another object of the present disclosure is to provide a model generation program for increasing the processing speed of a convolutional neural network. Another object of the present disclosure is to provide a model generation device for increasing the processing speed of a convolutional neural network. Another object of the present disclosure is to provide a data processing device having a high processing speed of a convolutional neural network. [Means for solving the problem]

[0006] The technical means of the present disclosure for solving the problems will be described below. Note that the claims and the reference characters in parentheses in this section indicate the corresponding relationship with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0007] A first aspect of the present disclosure is A model generation method executed by a processor (12) for generating a machine learning model by replacing a convolutional layer in a convolutional neural network with a decomposition layer obtained by matrix decomposition, comprising: Rearrange weight parameters constituting the original layer defined in the convolution layer before the replacement so as to form an equivalent weight matrix equivalent to a weight matrix product which is a matrix product of weight parameters constituting the decomposition layer; Decomposing the equivalent weight matrix to extract a plurality of ranks; and filtering at least one rank and constructing a decomposition layer based on convolution with a weight matrix product corresponding to the filtered rank.

[0008] A second aspect of the present disclosure is A model generation program stored in a storage medium (10) and including instructions to be executed by a processor (12) for generating a machine learning model by replacing a convolutional layer in a convolutional neural network with a decomposition layer obtained by matrix decomposition, the model generation program comprising: The command is, Rearrange weight parameters constituting the original layer defined in the convolutional layer before the replacement so as to form an equivalent weight matrix equivalent to a weight matrix product which is a matrix product of weight parameters constituting the decomposition layer; Decomposing the equivalent weight matrix to extract a plurality of ranks; and filtering at least one rank and constructing a decomposition layer based on a convolution with a weight matrix product corresponding to the filtered rank.

[0009] A third aspect of the present disclosure is A model generation device that generates a machine learning model by replacing a convolution layer in a convolutional neural network with a decomposition layer obtained by matrix decomposition, comprising: The processor Rearrange weight parameters constituting the original layer defined in the convolution layer before the replacement so as to form an equivalent weight matrix equivalent to a weight matrix product which is a matrix product of weight parameters constituting the decomposition layer; Decomposing the equivalent weight matrix to extract a plurality of ranks; and constructing a decomposition layer based on a convolution with a weight matrix product corresponding to the selected rank.

[0010] According to the first to third aspects, the weight parameters constituting the original layer defined in the convolution layer before replacement are rearranged to form an equivalent weight matrix equivalent to the weight matrix product of the weight parameters constituting the decomposition layer after replacement. According to this, the decomposition layer is constructed based on the convolution with the weight matrix product corresponding to at least one selection rank selected from the multiple ranks extracted by the matrix decomposition of the equivalent weight matrix, so that the number of weight parameters in the decomposition layer can be reduced as much as possible. Therefore, it is possible to increase the processing speed of the convolution neural network.

[0011] A fourth aspect of the present disclosure is a data processing device, A storage medium (10) that stores a machine learning model of a convolutional neural network generated by the model generation method of the first aspect; The system further includes a processor (12) that executes data processing based on a machine learning model stored in a storage medium.

[0012] According to the fourth aspect, in data processing based on the machine learning model of the convolutional neural network generated by the model generation method of the first aspect, it is possible to realize a high processing speed through the decomposition layer in which the number of weight parameters is reduced as much as possible. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an overall configuration of a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram for explaining a machine learning model according to the first embodiment. [Diagram 3] FIG. 4 is a schematic diagram for explaining an initial layer according to the first embodiment. [Figure 4] FIG. 2 is a schematic diagram for explaining decomposition layers according to the first embodiment. [Diagram 5] FIG. 4 is a schematic diagram for explaining an initial layer according to the first embodiment. [Figure 6] FIG. 2 is a schematic diagram for explaining decomposition layers according to the first embodiment. [Figure 7] 1 is a block diagram showing a functional configuration of a model generating device according to a first embodiment. [Figure 8] 4 is a flowchart showing a model generation flow according to the first embodiment. [Figure 9] FIG. 4 is a schematic diagram for explaining a sorting process according to the first embodiment. [Figure 10] FIG. 4 is a schematic diagram for explaining a sorting process according to the first embodiment. [Figure 11] FIG. 4 is a schematic diagram for explaining a sorting process according to the first embodiment. [Figure 12] FIG. 4 is a schematic diagram for explaining a sorting process according to the first embodiment. [Figure 13] FIG. 4 is a schematic diagram for explaining a rank extraction process according to the first embodiment. [Figure 14] FIG. 4 is a schematic diagram for explaining a layer construction process according to the first embodiment. [Figure 15] FIG. 4 is a schematic diagram for explaining a layer construction process according to the first embodiment. [Figure 16] FIG. 11 is a schematic diagram for explaining decomposition layers according to the second embodiment. [Figure 17] FIG. 11 is a schematic diagram for explaining decomposition layers according to the second embodiment. [Figure 18] 10 is a flowchart showing a model generation flow according to a second embodiment. [Figure 19] FIG. 11 is a schematic diagram for explaining a sorting process according to a second embodiment. [Figure 20] FIG. 11 is a schematic diagram for explaining a layer construction process according to a second embodiment. [Figure 21] FIG. 11 is a schematic diagram for explaining a layer construction process according to a second embodiment. [Figure 22] FIG. 13 is a schematic diagram for explaining a secondary decomposition layer according to the third embodiment. [Figure 23] FIG. 13 is a schematic diagram for explaining a primary decomposition layer according to the third embodiment. [Figure 24] FIG. 13 is a schematic diagram for explaining a secondary decomposition layer according to the third embodiment. [Diagram 25] 13 is a flowchart showing a model generation flow according to a third embodiment. [Figure 26] FIG. 13 is a schematic diagram for explaining a sorting process according to a third embodiment. [Figure 27] FIG. 13 is a schematic diagram for explaining a layer construction process according to a third embodiment. [Figure 28] FIG. 13 is a schematic diagram for explaining a layer construction process according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. In addition, by assigning the same reference numerals to corresponding components in each embodiment, duplicated descriptions may be omitted. In addition, when only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other parts of the configuration. Furthermore, in addition to the combination of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments can be partially combined together even if not explicitly stated, as long as there is no particular problem with the combination.

[0015] First embodiment The model generating device 1 of the first embodiment shown in Fig. 1 generates a machine learning model ML by replacing a convolution layer in a convolution neural network with a decomposition layer obtained by matrix decomposition. To this end, the model generating device 1 includes at least one dedicated computer. The dedicated computer constituting the model generating device 1 includes at least one memory 10 and one processor 12.

[0016] The memory 10 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, an optical medium, etc., that non-temporarily stores computer-readable programs or data, etc. The processor 12 includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), a reduced instruction set computer (RISC)-CPU, etc.

[0017] As shown in Fig. 2, the machine learning model ML provides a convolutional neural network having multiple convolutional layers Lm as intermediate layers between an input layer Li and an output layer Lo. As shown in Fig. 3 and Fig. 4, the convolutional layer Lm performs a convolution operation on a feature map n with a channel number c to output a feature map n+1 with a channel number o.

[0018] As shown in Fig. 3, the initial layer Lm0, which is the initial structure of the convolution layer Lm, is constructed from the convolution filters F for the number of output channels o, with a three-dimensional tensor of size h x w x c as a normal convolution filter (i.e., kernel) F. Here, in the initial layer Lm0, the convolution filter F for each output channel of the number of channels o is constructed from the weight parameters w ochwThe layer structure of the initial layer Lm0 can be expressed by the combination formula shown in Figure 5. o is the bias parameter for each output channel.

[0019] As shown in Fig. 4, a decomposition layer Lmd that is replaced from an initial layer Lm0 by matrix decomposition of a convolution layer Lm is constructed based on convolution with a weight matrix product that is a matrix product of weight parameters that constitute the layer Lmd. In particular, the decomposition layer Lmd of the first embodiment is constructed based on convolution with a weight matrix product that is obtained by matrix decomposing the initial layer Lm0 (see Fig. 3) into a depth-wise (DW) convolution filter Fdw and a point-wise (PW) convolution filter Fpw.

[0020] Here, the DW convolution filter Fdw for the number of input channels c in the decomposition layer Lmd is a two-dimensional tensor of size h×w×1 shown in FIG. 4, and has h×w weight parameters w′ chw On the other hand, in the decomposition layer Lmd, the PW convolution filters Fpw for the number of output channels o are one-dimensional tensors of size 1×1×c shown in FIG. 4, and are defined by weighting parameters w′ shown in FIG. oc The decomposition layer Lmd is defined by a weight matrix with matrix elements as follows. For these reasons, the decomposition layer Lmd can be expressed by the combination formula shown in FIG. 6. Note that b o is the bias parameter for each output channel.

[0021] The machine learning model ML including the decomposition layer Lmd replaced from the initial layer Lm0 for each convolution layer Lm is stored in the memory 10 as shown in FIG. 1. The processor 12 of the model generation device 1 executes data processing based on the machine learning model ML stored in the memory 10, thereby functioning as a data processing device. Here, the data processing by the model generation device 1 is at least one of machine learning processing of the machine learning model ML using teacher data, and analysis processing of input data passed through the machine learning model ML. The teacher data and input data are data related to at least one of digital data such as image data, voice data, text data, sensing data, vehicle motion data, vehicle driving data, and environmental data.

[0022] In the model generating device 1, the processor 12 executes a plurality of instructions included in the model generating program stored in the memory 10 in order to generate the machine learning model ML to be used in this way. As a result, the model generating device 1 constructs a plurality of functional blocks for generating the machine learning model ML by replacing the convolution layer Lm from the initial layer Lm0 to the decomposition layer Lmd. In this manner, in the model generating device 1, the matching program stored in the memory 10 causes the processor 12 to execute a plurality of instructions, thereby realizing each function of the plurality of functional blocks. These plurality of functional blocks include a sorting block 100, a rank extraction block 200, and a layer construction block 300, as shown in FIG. 7.

[0023] By the cooperation of these blocks 100, 200, and 300, the model generation device 1 replaces the convolution layer Lm from the initial layer Lm0 to the decomposition layer Lmd, and the model generation method for generating the machine learning model ML is executed according to the model generation flow of Fig. 8. Note that each "S" in the model generation flow means a plurality of steps executed by a plurality of instructions included in the generation program.

[0024] In the model generation flow of the first embodiment, steps S101 to S103 are executed as shown in Fig. 8. Specifically, in step S101, the rearrangement block 100 defines the initial layer Lm0 input to the model generation device 1 as the convolutional layer Lm before permutation as the original layer, and sets the weight parameters w ochw At this time, the rearrangement block 100 rearranges the weight parameters w′ constituting the decomposition layer Lmd after replacement. chw ,w” oc The weighting parameters w of the initial layer Lm0 are set to construct an equivalent weighting matrix WMe equivalent to the weighting matrix product of the two matrices as shown in FIG. ochw Rearrange the items.

[0025] Specifically, the rearrangement block 100 rearranges the weight parameters w ochw 10 for each input channel of the number of channels c. At the same time, the sorting block 100 sorts the weight parameters w′ of the DW convolution filter Fdw constituting the decomposition layer Lmd that is replaced from the initial layer Lm0. chw and the weight parameter w of the PW convolution filter Fpw. oc 11 and 12, respectively, are allocated to each input channel.

[0026] Under such allocation, the sorting block 100 uses the weight parameter w′ shown on the right side of FIG. chw ,w” oc In order to ensure that the equality holds for the weight matrix product of ochw In this case, in the first embodiment in particular, the DW weighting matrix, which is a one-dimensional tensor in one row, is generated by rearranging the weight parameters w′ of the DW convolution filter Fdw. chw In the first embodiment, the PW weighting matrix, which is a one-dimensional tensor with one row that forms a weighting matrix product with the DW weighting matrix, is assumed to be the weight parameter w" of the PW convolution filter Fpw. ocFrom these assumptions, in the first embodiment, a weight matrix that is a two-dimensional tensor of size (h×w)×o is defined as an equivalent weight matrix WMe.

[0027] 8, the rank extraction block 200 extracts a plurality of ranks r by matrix decomposing the equivalent weight matrix WMe obtained by the sorting block 100 in S101. At this time, the rank extraction block 200 of the first embodiment extracts a plurality of ranks r by using a weight parameter w′ chw The decomposition matrix U associated with the DW weight matrix of the singular value diagonal matrix Σ and the weight parameter w” oc In the singular value decomposition for each input channel, the rank extraction block 200 decomposes each singular value ω r In the example of FIG. 13, the rank extraction block 200 extracts indices (subscript suffixes 0, 1, and 2 of the code ω) for identifying each rank r. At the same time, the rank extraction block 200 extracts columns of the decomposition matrix U and rows of the decomposition matrix V as matrix elements corresponding to each rank r. Furthermore, based on the results of these extractions, the rank extraction block 200 extracts the DW weighting matrix for each rank r by dividing the columns of the decomposition matrix U and the singular values ​​ω r and the PW weighting matrix is ​​obtained from the matrix product of the decomposition matrix V itself, or the DW weighting matrix is ​​obtained from the column of the decomposition matrix U itself and the PW weighting matrix is ​​obtained from the row of the decomposition matrix V and the singular value ω r These are obtained by matrix multiplication with

[0028] In S103 shown in Fig. 8, the layer construction block 300 selects at least one rank rs from the multiple ranks r extracted by the rank extraction block 200 in S102, and constructs a decomposition layer Lmd based on convolution with a weight matrix product corresponding to the selected rank rs. In this case, the layer construction block 300 of the first embodiment in particular also selects weight matrix products corresponding to at least two selection ranks rs that are smaller than the total number of ranks r (i.e., the number of ranks of the singular value diagonal matrix Σ) as the matrix product of the DW weight matrix and the PW weight matrix obtained by decomposing the equivalent weight matrix WMe for each input channel with the number of channels c shown in Fig. 14. Here, the selection rank rs is the number of singular values ​​ω r In other words, the singular values ​​ω r Ranks r with small values ​​may be excluded from the selection ranks rs.

[0029] Under such sorting, the layer construction block 300 obtains a decomposition layer Lmd by performing element-by-element addition across the sorting ranks rs of feature maps obtained as a result of convolution of the DW weight matrix and the PW weight matrix corresponding to each sorting rank rs, as shown in Figures 14 and 15. Specifically, for each sorting rank rs, the layer construction block 300 convolves the feature map hxwxc obtained by convolving the feature map n with the DW weight matrix, and then convolves the feature map hxwxo with the PW weight matrix to obtain a feature map hxwxo, and then performs element-by-element addition to output the feature map n+1 of hxwxo. Here, Figure 14 shows the weight parameters w', which are matrix components corresponding to each sorting rank rs. chw ,w” oc The connections between the weight parameters w′ are expressed as the structure of the decomposition layer Lmd for each input channel. However, in FIG. 14, in order to clarify the correspondence with the selection rank rs, for the sake of convenience, each weight parameter w′ chw ,w” oc The superscript suffix on represents the corresponding selection rank rs.

[0030] As a result, the layer construction block 300 replaces the initial layer Lm0 of the original layer stored in the memory 10 in response to the input with the decomposition layer Lmd constructed based on the selection rank rs. At this time, even in the case of a combination of DW convolution and PW convolution that requires machine learning, for example, the replacement from the convolution layer Lm can be realized without machine learning while suppressing degradation and maintaining accuracy.

[0031] (Action and effect) The effects of the first embodiment described above will be described below.

[0032] According to the first embodiment, the weight parameters w constituting the initial layer Lm0 as the original layer defined in the convolutional layer Lm before replacement are ochw is the weight parameter w' that constitutes the decomposition layer Lmd after replacement. chw ,w” oc The weight matrix WMe is rearranged to form an equivalent weight matrix WMe equivalent to the weight matrix product of the weight matrix WMe. According to this, the decomposition layer Lmd is constructed based on the convolution with the weight matrix product corresponding to at least one selection rank rs selected from the multiple ranks r extracted by the matrix decomposition of the equivalent weight matrix WMe, so that the number of weight parameters in the decomposition layer Lmd can be reduced as much as possible. Therefore, it is possible to increase the processing speed of the convolutional neural network. In addition, it is possible to reduce the amount of calculation in the convolutional neural network and to unify the layer structure after replacement, thereby miniaturizing the model generating device 1 as hardware.

[0033] According to the first embodiment, the decomposition layer Lmd is constructed based on convolution with weight matrix multiplication corresponding to a smaller number of selection ranks rs than the total number of ranks r, so that the effect of reducing the number of weight parameters can be enhanced. Therefore, the first embodiment is advantageous in increasing the processing speed of the convolutional neural network. The first embodiment is also advantageous in reducing the size of the model generating device 1.

[0034] According to the first embodiment, the decomposition layer Lmd is generated by adding elements of the convolution results of the weight matrix multiplication corresponding to at least two selection ranks rs, so that the replacement accuracy can be improved. Here, particularly in the first embodiment, the selection ranks rs are smaller than the total number of ranks r, so that the replacement accuracy by low-rank approximation can be improved. Therefore, the first embodiment is advantageous in improving the processing speed of the convolutional neural network as well as the processing accuracy. The first embodiment is also advantageous in miniaturizing the model generation device 1 with high processing accuracy.

[0035] According to the first embodiment, the equivalent weight matrix WMe is calculated by multiplying the weight parameters w in the initial layer Lm0 so that the equivalent weight matrix WMe is equivalent to the weight matrix product of the decomposition layer Lmd that is decomposed into the DW convolution filter Fdw and the PW convolution filter Fpw. ochw The combination of DW convolution and PW convolution, in conjunction with layer construction based on convolution with a weight matrix product corresponding to the sorting rank rs, can enhance the effect of reducing the number of weight parameters in the decomposition layer Lmd. Therefore, the first embodiment is advantageous in increasing the processing speed of the convolutional neural network. The first embodiment is also advantageous in reducing the size of the model generating device 1.

[0036] According to the first embodiment, in data processing based on the machine learning model ML of the convolutional neural network generated by the model generation method, a high processing speed can be achieved through the decomposition layer Lmd in which the number of weight parameters can be reduced as much as possible. In addition, since the amount of calculation for data processing in the convolutional neural network is reduced and the layer structure is unified, it is possible to miniaturize the model generation device 1, which is also hardware that functions as a data processing device.

[0037] Second Embodiment The second embodiment is a modification of the first embodiment.

[0038] In the second embodiment, the decomposition layer Lmd is constructed based on convolution using a weight matrix product obtained by decomposing the initial layer Lm0 into a weight-sharing DW convolution filter Fdws and a PW convolution filter Fpw, as shown in Fig. 16. In particular, in the decomposition layer Lmd of the second embodiment, a single DW convolution filter Fdws is shared for the PW convolution filters Fpw for the number of output channels o defined according to the first embodiment.

[0039] Here, the weight sharing DW convolution filter Fdws is a two-dimensional tensor of size h×w×1 shown in FIG. 16, and has h×w weight parameters w′ shown in FIG. hw The decomposition layer Lmd of the second embodiment can be expressed by the combination formula shown in FIG. o is the bias parameter for each output channel.

[0040] 18 of the second embodiment, steps S201 to S203 are executed instead of steps S101 to S103 of the first embodiment. Specifically, in step S201, the rearrangement block 100 rearranges the weight parameters w′ constituting the decomposition layer Lmd. hw ,w” oc For the weight matrix multiplication of these layers, the weight parameters w ochw At this time, the rearrangement block 100 of the second embodiment rearranges the weight parameter w′ shown on the right side of FIG. hw ,w” oc In order to ensure that the equality holds for the weight matrix product of ochw to generate the equivalent weight matrix WMe.

[0041] Here, the weighting parameter w of the PW convolution filter Fpw is oc Regarding the PW weighting matrix assumed to be a one-dimensional tensor with one row according to the first embodiment, the weight parameter w' of the DW convolution filter Fdws is hwFor the above, a DW weight matrix is ​​assumed to be a one-dimensional tensor of one column. Therefore, in the second embodiment, a weight matrix that is a two-dimensional tensor of size (h×w)×(o×c) is defined as an equivalent weight matrix WMe that is equivalent to the matrix product of the DW weight matrix and the PW weight matrix.

[0042] In the model generation flow of the second embodiment shown in Fig. 18, in S202, the rank extraction block 200 extracts multiple ranks r by matrix decomposing the equivalent weight matrix WMe obtained by the sorting block 100 in S201. In particular, the rank extraction block 200 of the second embodiment extracts multiple ranks r by using a weight parameter w' hw The decomposition matrix U associated with the DW weight matrix of the singular value diagonal matrix Σ and the weight parameter w” oc The equivalent weight matrix WMe is decomposed into a matrix product with a decomposition matrix V associated with the PW weight matrix of r , and the corresponding columns of the decomposition matrix U and rows of the decomposition matrix V. Based on these extraction results, the rank extraction block 200 of the second embodiment extracts the DW weight matrix for each rank r by dividing the DW weight matrix into columns of the decomposition matrix U and singular values ​​ω r and the PW weighting matrix is ​​obtained from the matrix product of the decomposition matrix V itself, or the DW weighting matrix is ​​obtained from the column of the decomposition matrix U itself and the PW weighting matrix is ​​obtained from the row of the decomposition matrix V and the singular value ω r These are obtained by matrix multiplication with

[0043] Furthermore, in the model generation flow of the second embodiment, in S203, the layer construction block 300 constructs a decomposition layer Lmd based on convolution with a weight matrix product corresponding to a selection rank rs selected from the multiple ranks r extracted by the rank extraction block 200 in S202. In particular, at this time, the layer construction block 300 of the second embodiment also selects weight matrix products corresponding to at least two selection ranks rs that are smaller than the total number of ranks r as the matrix product of the DW weight matrix and the PW weight matrix obtained by decomposing the equivalent weight matrix WMe as shown in FIG.

[0044] Under such selection, the layer construction block 300 of the second embodiment obtains the decomposition layer Lmd by adding elements of the feature map obtained as a result of convolution of the weight-sharing DW weight matrix and the PW weight matrix corresponding to each selection rank rs, over the selection rank rs, as shown in Figures 20 and 21. Here, Figure 20 shows the weight parameter w', which is a matrix element corresponding to each selection rank rs. hw ,w” oc In FIG. 20, in order to clarify the correspondence with the selection rank rs, for the sake of convenience, each weight parameter w′ hw ,w” oc The superscript suffix on represents the corresponding selection rank rs. As described above, the layer construction block 300 of the second embodiment also replaces the initial layer Lm0 of the original layer stored in the memory 10 in response to the input with the decomposition layer Lmd constructed based on the selection rank rs.

[0045] According to the second embodiment described above, the weight parameters w constituting the initial layer Lm0 as the original layer defined in the convolutional layer Lm before replacement are ochw is the weight parameter w' that constitutes the decomposition layer Lmd after replacement. hw ,w” oc The weight matrices are rearranged to form an equivalent weight matrix WMe equivalent to the product of the weight matrices of the weight matrices. According to this, based on the same principle as in the first embodiment, it is possible to reduce the number of weight parameters in the decomposition layer Lmd as much as possible and increase the processing speed of the convolutional neural network. In addition, it is possible to reduce the amount of calculation in the convolutional neural network and unify the layer structure after replacement, thereby reducing the size of the model generating device 1.

[0046] Furthermore, according to the second embodiment, the equivalent weight matrix WMe is calculated by multiplying the weight parameters w in the initial layer Lm0 so that the equivalent weight matrix WMe is equivalent to the weight matrix product of the decomposition layer Lmd decomposed into the weight sharing type DW convolution filter Fdws and the PW convolution filter Fpw. ochw In this way, the weight parameter w' for the PW convolution is hwAccording to the DW convolution in which the weights are shared, in combination with the layer construction based on the convolution with the weight matrix product corresponding to the selection rank rs, the effect of reducing the number of weight parameters in the decomposition layer Lmd can be enhanced. Therefore, the second embodiment is advantageous in increasing the processing speed of the convolutional neural network. The second embodiment is also advantageous in reducing the size of the model generating device 1.

[0047] Third embodiment The third embodiment is a modification of the second embodiment.

[0048] As the convolution layer Lm of the third embodiment, the primary decomposition layer Lmd replaced from the previous original layer, the initial layer Lm0, according to the second embodiment is redefined as the next original layer, and is replaced with a secondary decomposition layer Lmd2 that is further matrix decomposed. Therefore, as shown in Fig. 22, the secondary decomposition layer Lmd2 is constructed based on convolution with a weight matrix product obtained by further matrix decomposing the weight sharing type DW convolution filter Fdws of the primary decomposition layer Lmd into a pair of one-dimensional DW convolution filters Fdw2.

[0049] In the following description, for the weight sharing DW convolution filter Fdws of the primary decomposition layer Lmd that is the original layer after redefinition, for convenience, the weight parameter w′ hw is the weight parameter w hw It is assumed that the equation is redefined as follows: In addition, b in the coupling equation in FIG. 23 is a bias parameter.

[0050] One of the pair of DW convolution filters Fdw2 is a one-dimensional tensor of size 1×w×1 shown in FIG. 22, and has w weight parameters w′ shown in FIG. w On the other hand, the other DW convolution filter Fdw2 is a one-dimensional tensor of size h×1×1 shown in FIG. 22 and is defined by a weighting matrix having h weight parameters w′ shown in FIG. hAs a result, the secondary decomposition layer Lmd2 of the third embodiment can be expressed by the combination formula shown in Fig. 24. Note that b in the combination formula in Fig. 24 is a bias parameter.

[0051] In the model generation flow of the third embodiment shown in Fig. 25, steps S201 to S203 are executed, and then steps S301 to S303 are executed. Specifically, in step S301, the rearrangement block 100 rearranges the weighting parameters w' w ,w” h For the weight matrix multiplication of the two layers, the weight parameters w of the DW convolution filter Fdws in the primary decomposition layer Lmd redefined in the original layer are hw At this time, the rearrangement block 100 of the third embodiment rearranges the weight parameter w′ shown on the right side of FIG. w ,w” h In order to ensure that the equality holds for the weight matrix product of hw to generate the equivalent weight matrix WMe.

[0052] Here, one of the weight parameters w' of each DW convolution filter Fdws is w The weighting matrix DW is a one-dimensional tensor in a row, and the weighting parameter w” h For each of the above, a DW weight matrix is ​​assumed to be a one-dimensional tensor with one row. Therefore, in the third embodiment, a weight matrix that is a two-dimensional tensor with a size of h×w is defined as the equivalent weight matrix WMe.

[0053] In the model generation flow of the third embodiment shown in Fig. 25, in S302, the rank extraction block 200 re-extracts multiple ranks r by matrix decomposing the equivalent weight matrix WMe obtained by the sorting block 100 in S301. In particular, the rank extraction block 200 of the third embodiment extracts one weight parameter w' w The decomposition matrix U associated with the DW weight matrix of the singular value diagonal matrix Σ and the other weight parameter w” hThe equivalent weight matrix WMe is decomposed into a matrix product with a decomposition matrix V associated with the DW weight matrix of . Then, the rank extraction block 200 of the third embodiment calculates the singular value ω r , and the corresponding columns of the decomposition matrix U and rows of the decomposition matrix V. Based on these extraction results, the rank extraction block 200 of the third embodiment extracts, for each rank r, one of the DW weight matrices as the columns of the decomposition matrix U and the singular values ​​ω r and the other DW weight matrix is ​​obtained from the matrix product of the decomposition matrix V itself, or one DW weight matrix is ​​obtained from the column of the decomposition matrix U itself and the other DW weight matrix is ​​obtained from the row of the decomposition matrix V and the singular value ω r are obtained from the matrix multiplication of

[0054] Furthermore, in the model generation flow of the third embodiment, in S303, the layer construction block 300 constructs a secondary decomposition layer Lmd2 based on convolution with a weight matrix product corresponding to the selection rank rs selected from the multiple ranks r extracted by the rank extraction block 200 in S302. In particular, at this time, the layer construction block 300 of the third embodiment also selects weight matrix products corresponding to at least two selection ranks rs that are fewer than the total number of ranks r as the matrix product of a pair of DW weight matrices obtained by decomposing the equivalent weight matrix WMe as shown in FIG.

[0055] Under such selection, the layer construction block 300 of the third embodiment obtains a secondary decomposition layer Lmd2 by adding elements of feature maps obtained as a result of convolution with a pair of one-dimensional DW weight matrices corresponding to each selection rank rs, across the selection rank rs, as shown in Figures 27 and 28. Here, Figure 27 shows the weight parameters w', which are matrix elements corresponding to each selection rank rs. w ,w” h In FIG. 27, in order to clarify the correspondence with the selection rank rs, for the sake of convenience, each weight parameter w′ w ,w” hThe superscript suffix added to represents the corresponding selection rank rs. As described above, the layer construction block 300 of the third embodiment replaces the layer structure related to the weight sharing DW convolution filter Fdws in the primary decomposition layer Lmd as the original layer stored in the memory 10 by S201 to S203 with the secondary decomposition layer Lmd2 constructed based on the selection rank rs.

[0056] According to the third embodiment described above, the secondary decomposition layer Lmd2 that is replaced from the primary decomposition layer Lmd, which is the previous original layer, is redefined as the next original layer. As a result, the weight parameters w hw is the weight parameter w' that constitutes the secondary decomposition layer Lmd2 after the permutation. w ,w” h The weights are rearranged to form an equivalent weight matrix WMe equivalent to the product of the weight matrices of the weights. According to this, based on the same principle as in the first embodiment, a secondary decomposition layer Lmd2 in which the number of weight parameters is further reduced as much as possible from the primary decomposition layer Lmd of the previous permutation can be constructed by the next permutation following the previous one. Therefore, the third embodiment is advantageous in increasing the processing speed of the convolutional neural network. In addition, the third embodiment is advantageous in reducing the amount of calculation in the convolutional neural network and in unifying the layer structure after permutation, thereby miniaturizing the model generating device 1.

[0057] According to the third embodiment, the equivalent weight matrix WMe is equivalent to the weight matrix product of the secondary decomposition layer Lmd2 decomposed into a pair of one-dimensional DW convolution filters Fdw2, and is expressed as the weight parameters w hw The combination of such one-dimensional DW convolutions, coupled with layer construction based on convolution with a weight matrix product corresponding to the sorting rank rs, can enhance the effect of reducing the number of weight parameters in the secondary decomposition layer Lmd2. Therefore, the third embodiment is advantageous in increasing the processing speed of the convolutional neural network. The third embodiment is also advantageous in reducing the size of the model generating device 1.

[0058] (Other embodiments) Although several embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope not departing from the gist of the present disclosure.

[0059] In a modified example, the dedicated computer constituting the model generating device 1 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), CPLD (Complex Programmable Logic Device), etc. Furthermore, such a digital circuit may have a memory that stores a program.

[0060] In a modified example, the order of the filters Fdw, Fpw forming the weighting matrix product may be interchanged from the order described in the first embodiment. In a modified example, the order of the filters Fdws, Fpw forming the weighting matrix product may be interchanged from the order described in the second embodiment. In a modified example, the order of the pair of filters Fdw2, Fdw2 forming the weighting matrix product may be interchanged from the order described in the third embodiment.

[0061] In a modified example, matrix decomposition may be realized by a decomposition method other than singular value decomposition, such as principal component analysis or eigenvalue decomposition. In a modified example, the number of selection ranks rs may be adjusted by a trade-off between processing speed and processing accuracy. In a modified example, the number of selection ranks rs may be reduced as much as possible, so that the weight parameters of the decomposition layers Lmd and Lmd2 may be machine-learned after replacement.

[0062] The selection rank rs of the modified example is a single rank r that is smaller than the total number, preferably the maximum singular value ωr In this case, the decomposition layers Lmd, Lmd2 may be constructed based on convolution using only a weight matrix multiplication corresponding to a single selection rank rs. In the modified example, the selection rank rs may be selected for all ranks r. In this case, the decomposition layers Lmd, Lmd2 may be constructed by element addition between the convolution results using weight matrix multiplication corresponding to all selection ranks rs.

[0063] In a modified example, the decomposition layer Lmd of the third embodiment may be the initial layer Lm0 of the convolution layer Lm. In this case, in the model generation flow of the third embodiment, the execution of S201 to S203 may be omitted, and only S301 to S303 may be executed, so that the layer Lmd as the original layer may be replaced with a matrix decomposed layer Lmd2.

[0064] In the modified example, the model generating device 1 may not have the function of a data processing device. In addition to the above, each of the embodiments and modified examples described above may be implemented as a semiconductor device (e.g., a semiconductor chip) having at least one each of the processor 12 and the memory 10 of the model generating device 1. [Explanation of symbols]

[0065] 1: model generating device, 10: memory, 12: processor

Claims

1. A model generation method executed by a processor (12) for generating a machine learning model by replacing a convolutional layer in a convolutional neural network with a decomposition layer obtained by matrix decomposition, comprising: Rearrange weight parameters constituting an original layer defined in the convolutional layer before permutation so as to form an equivalent weight matrix equivalent to a weight matrix product which is a matrix product of weight parameters constituting the decomposition layer; decomposing the equivalent weight matrix to extract a plurality of ranks; and constructing the decomposition layer based on convolution with the weight matrix product corresponding to the selected rank.

2. Constructing the decomposition layers comprises:

2. The method of claim 1, comprising constructing the decomposition layers based on convolution with the weight matrix products corresponding to a smaller number of the culled ranks than the total number of ranks.

3. Constructing the decomposition layers comprises: The method for generating a model according to claim 1 or 2, further comprising: generating the decomposition layer by adding elements of convolution results of the weight matrix multiplication corresponding to at least two of the sorting ranks.

4. The rearrangement of the weight parameters of the original layer includes: The method for generating a model according to claim 1 , further comprising: obtaining, by rearrangement, the equivalent weight matrix equivalent to the weight matrix product of the decomposition layer decomposed into a depth-wise convolution filter and a point-wise convolution filter.

5. The rearrangement of the weight parameters of the original layer includes: The model generating method according to claim 1 , further comprising: obtaining, by rearrangement, the equivalent weight matrix equivalent to the weight matrix product of the decomposition layer decomposed into a weight-sharing depth-wise convolution filter and a point-wise convolution filter.

6. The rearrangement of the weight parameters of the original layer includes: The method of claim 1 , further comprising: obtaining, by permutation, the equivalent weight matrix equivalent to the weight matrix product of the decomposition layer decomposed into a pair of one-dimensional depth-wise convolution filters.

7. The rearrangement of the weight parameters of the original layer includes: The model generating method according to any one of claims 1 to 6, further comprising: redefining the decomposition layer replaced from the previous original layer as the next original layer.

8. A model generation program stored in a storage medium (10) and including instructions to be executed by a processor (12) for generating a machine learning model by replacing a convolutional layer in a convolutional neural network with a decomposition layer obtained by matrix decomposition, the model generation program comprising: The instruction: Rearrange weight parameters constituting an original layer defined in the convolutional layer before permutation so as to form an equivalent weight matrix equivalent to a weight matrix product which is a matrix product of weight parameters constituting the decomposition layer; decomposing the equivalent weight matrix to extract a plurality of ranks; and selecting at least one of the ranks and constructing the decomposition layer based on a convolution with the weight matrix product corresponding to the selected rank.

9. A model generation device that generates a machine learning model by replacing a convolution layer in a convolutional neural network with a decomposition layer obtained by matrix decomposition, comprising: The processor, Rearrange weight parameters constituting an original layer defined in the convolutional layer before permutation so as to form an equivalent weight matrix equivalent to a weight matrix product which is a matrix product of weight parameters constituting the decomposition layer; decomposing the equivalent weight matrix to extract a plurality of ranks; culling at least one of the ranks; and constructing the decomposition layer based on a convolution with the weight matrix product corresponding to the culled rank.

10. A storage medium (10) that stores the machine learning model of the convolutional neural network generated by the model generation method according to any one of claims 1 to 7; A data processing device comprising: a processor (12) that executes data processing based on the machine learning model stored in the storage medium.

Images