Evaluation method, evaluation device, and program

Abstract

To provide an evaluation method capable of obtaining a neural network architecture of relatively high actual inference performance using an NAS (Network Attached Storage).SOLUTION: The evaluation method includes the steps of: acquiring a neural network architecture to be evaluated (S101); calculating a weighted sum of a first index value that represents the amount of change in output values when the input value to be input to a neural network that has the acquired neural network architecture (S102) and one or more second index values indicating the variance of one or more normalization layers included in the neural network (S103-S104) as an evaluation value; and outputting the same (S105).SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an evaluation method, an evaluation apparatus, and a program.

Background Art

[0002] There is a technology of neural architecture search (NAS (Neural Architecture Search)) for searching for a neural network architecture with relatively high inference performance.

[0003] One of the NAS technologies is Zero-Shot NAS. One of the Zero-Shot NAS technologies is Zen-NAS. Zen-NAS contributes to obtaining a neural network architecture with relatively high performance by evaluating the neural network architecture without learning the neural network.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, there is a problem that a neural network architecture with relatively high actual inference performance may not be obtained by NAS.

[0006] Therefore, the present invention provides an evaluation method and the like that contribute to obtaining a neural network architecture with relatively high actual inference performance by NAS.

Means for Solving the Problems

[0007] An evaluation method according to one aspect of the present invention is an evaluation method for evaluating a neural network architecture using a processor, the method comprising: acquiring the neural network architecture that is the subject of the evaluation; calculating a weighted sum of a first index value indicating the amount of change in the output value when the input value input to a neural network model having the acquired neural network architecture is changed, and one or more second index values ​​indicating the variance of one or more normalization layers included in the neural network model, as an evaluation value; and outputting the calculated evaluation value.

[0008] These comprehensive or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0009] The evaluation method of the present invention can contribute to obtaining a neural network architecture with relatively high actual inference performance using NAS. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram illustrating the layers included in a block in a neural network architecture. [Figure 2] This is an explanatory diagram illustrating an example of performance evaluation values ​​and actual measured performance values ​​for a neural network architecture. [Figure 3] This is an explanatory diagram schematically showing the configuration of the evaluation device in the embodiment. [Figure 4] This is a schematic diagram illustrating the processing performed by the acquisition unit of the evaluation device in the embodiment. [Figure 5]This is a schematic diagram illustrating the processing performed by the calculation unit of the evaluation device in the embodiment. [Figure 6] This is a flowchart showing the processes performed by the evaluation device in the embodiment. [Figure 7] This is an explanatory diagram illustrating an example of the relationship between the performance evaluation value calculated by the evaluation device in the comparative example and the actual performance measurement value. [Figure 8] This is an explanatory diagram showing an example of the relationship between the performance evaluation value calculated by the evaluation device in the embodiment and the actual performance measurement value. [Figure 9] This is an explanatory diagram illustrating examples of inference accuracy and inference time for neural network architectures obtained by exploring evaluation devices in comparative examples and embodiments. [Modes for carrying out the invention]

[0011] (Knowledge that formed the basis of this invention) The inventors have found that the following problems arise with respect to the techniques for exploring neural network architectures, as described in the "Background Art" section.

[0012] When performing processing using AI (Artificial Intelligence) models in in-vehicle or embedded devices, it is desirable to use a neural network architecture that requires relatively small computational resources and enables high-speed and high-accuracy inference. This is because the computational resources of in-vehicle devices and similar systems are limited.

[0013] General-purpose neural network architectures may not be able to fully utilize the performance of SoCs (System-on-a-chip) used in automotive and other applications. Specifically, the inference speed or accuracy of AI models with neural network architectures varies depending on the SoC. One reason for this is that the optimal values ​​for the design parameters of the neural network architecture differ for each SoC. Examples of design parameters include the number of blocks, the number of channels, the stride, and the kernel size.

[0014] By the way, there is a technology called Neural Architecture Search (NAS) for exploring neural network architectures with relatively high inference performance. The NAS technology can contribute to exploring neural network architectures that can fully utilize the performance of the SoC.

[0015] According to NAS, generally, by evaluating the performance while changing the design parameters included in the neural network architecture, the process of obtaining a neural network architecture with relatively high performance is repeated. The longer the search time, the more times the repetition occurs, and the higher the probability of obtaining a neural network architecture with higher performance.

[0016] One of the NAS technologies is Zero-Shot NAS. One of the Zero-Shot NAS technologies is Zen-NAS. Zen-NAS contributes to obtaining a neural network architecture with relatively high performance by evaluating the neural network architecture without learning the neural network.

[0017] However, there may be a situation where the evaluation value, which is the performance value of the neural network architecture obtained during the exploration of the neural network architecture by Zen-NAS, does not match the measured value, which is the value indicating the performance of the inference actually performed by the neural network having the neural network architecture.

[0018] When the evaluation value and the measured value do not match, there is a problem that even if the neural network architecture is explored using the evaluation value, it is impossible to obtain a neural network architecture with relatively high actual inference performance.

[0019] Therefore, the present invention provides an evaluation method and the like that contributes to obtaining a neural network architecture with relatively high actual inference performance using NAS.

[0020] The following describes examples of inventions that can be obtained from the disclosures in this specification, and explains the effects and other benefits that can be obtained from such inventions.

[0021] (1) An evaluation method for evaluating a neural network architecture using a processor, the method comprising: acquiring the neural network architecture that is the subject of the evaluation; calculating a weighted sum of a first index value indicating the amount of change in the output value when the input value input to a neural network model having the acquired neural network architecture is changed, and one or more second index values ​​indicating the variance of one or more normalization layers included in the neural network model, as an evaluation value, and outputting the calculated evaluation value.

[0022] According to the above embodiment, the evaluation value of the neural network architecture is calculated using a weighted sum of the first index value and one or more second index values. This allows for the calculation of a more appropriate evaluation value than when the neural network architecture is calculated using the sum (in other words, simple sum) of the first index value and one or more second index values. This is because the weights of the weighted sum allow for adjustment of the contribution of the first and second index values ​​to the evaluation value, or the contribution of one or more second index values ​​to the evaluation value. Therefore, the above evaluation method contributes to obtaining a neural network architecture with relatively high actual inference performance using NAS.

[0023] (2) The evaluation method according to (1), wherein when calculating the evaluation value, for each of the one or more normalization layers, the weighted sum is calculated using the weights determined according to the type of convolutional layer included in the preceding stage of the normalization layer in the neural network model, and the evaluation value is calculated using the calculated weighted sum.

[0024] According to the above embodiment, in particular, the magnitude of the contribution of each of the one or more second index values ​​to the evaluation value can be adjusted according to the type of normalization layer by the weights of the weighted sum, so that a more appropriate evaluation value can be calculated and the evaluation result can be made more appropriate. Therefore, the above evaluation method contributes to obtaining a more appropriate neural network architecture with relatively high actual inference performance using NAS.

[0025] (3) The evaluation method described in (2), wherein the weight for each of the one or more normalization layers is the same as when the layer preceding the normalization layer is a Conv (Convolution) layer if the layer preceding the normalization layer is a PWConv (Pointwise Convolution) layer, and greater than when the layer preceding the normalization layer is a Conv (Depthwise Convolution) layer if the layer preceding the normalization layer is a DWConv (Depthwise Convolution) layer.

[0026] According to the above embodiment, the weights for the second index value can be more easily adjusted depending on whether the layer preceding the normalization layer is a PWConv layer, a DWConv layer, or a Conv layer. Specifically, since the DWConv layer has fewer parameters compared to the PWConv layer or Conv layer, the behavior of the evaluation value can be made more appropriate by increasing the weight for the DWConv layer. Therefore, the above evaluation method contributes to more easily obtaining a neural network architecture with relatively high actual inference performance using NAS.

[0027] (4) The evaluation method according to any one of (1) to (3), wherein when acquiring the neural network architecture, the neural network architecture is acquired by changing at least one of the multiple design parameters of one of the multiple neural network architectures stored in the memory, and when changing at least one of the multiple design parameters, the number of blocks of a predetermined type among the multiple design parameters is fixed and the other design parameters are changed.

[0028] According to the above embodiment, it is possible to efficiently modify design parameters among multiple design parameters that can contribute to obtaining a neural network architecture with relatively high inference performance. Some of the multiple design parameters are known to enable the neural network architecture to exhibit relatively high inference performance when set to specific values. Fixing such design parameters can contribute to efficiently obtaining a neural network architecture with relatively high inference performance. Therefore, the above evaluation method contributes to more efficiently obtaining a neural network architecture with relatively high actual inference performance using NAS.

[0029] (5) The evaluation method according to (4), wherein when the evaluation value is calculated for the acquired neural network architecture, the storage unit is updated to update the storage unit so that a predetermined number of neural network architectures with higher evaluation values ​​are stored in the storage unit from among the group of neural network architectures including the acquired neural network architecture and a plurality of neural network architectures stored in the storage unit, and acquiring the neural network architecture includes acquiring the neural network architecture from the storage unit on which the update control has been performed.

[0030] According to the above embodiment, the memory unit can be maintained in a state where a neural network architecture with a relatively high evaluation value is stored, and the stored neural network architecture can be provided, as well as contribute to generating a neural network architecture with an even higher evaluation value. Therefore, the above evaluation method contributes to obtaining a neural network architecture with relatively high actual inference performance using NAS.

[0031] (6) An evaluation device for evaluating a neural network architecture using a processor, comprising: an acquisition unit for acquiring the neural network architecture that is the subject of evaluation; and a calculation unit for calculating and outputting as an evaluation value a weighted sum of a first index value indicating the amount of change in the output value when the input value input to the neural network model having the neural network architecture acquired by the acquisition unit is changed, and one or more second index values ​​indicating the variance of one or more normalization layers included in the neural network model.

[0032] According to the above embodiment, the same effect as the above evaluation method is achieved.

[0033] (7) A program that causes a computer to execute the evaluation method described in (1).

[0034] According to the above embodiment, the same effect as the above evaluation method is achieved.

[0035] These comprehensive or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, or recording medium.

[0036] The embodiments will be described in detail below with reference to the drawings.

[0037] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0038] (Embodiment) In this embodiment, we will describe evaluation methods and other aspects that contribute to obtaining a neural network architecture with relatively high actual inference performance using NAS.

[0039] Figure 1 is a schematic diagram illustrating the layers included in a block in a neural network architecture.

[0040] Figure 1(a) shows the blocks included in neural network architecture A (also called architecture A).

[0041] In the block shown in Figure 1(a), the PWConv (Pointwise Convolution) layer L11, the Conv (Convolution) layer L13, and the PWConv layer L15 are arranged in that order. Furthermore, normalization layers L12, L14, and L16 are placed after the PWConv layer L11, the Conv layer L13, and the PWConv layer L15, respectively.

[0042] Figure 1(b) shows the blocks included in neural network architecture B (also called architecture B).

[0043] In the block shown in Figure 1(b), the PWConv layer L21, the DWConv (Depthwise Convolution) layer L23, and the PWConv layer L25 are arranged in that order. Furthermore, normalization layers L22, L24, and L26 are placed after the PWConv layer L21, the DWConv layer L23, and the PWConv layer L25, respectively.

[0044] Figure 2 is an explanatory diagram illustrating examples of evaluated and measured values ​​for a neural network architecture. Figure 2 shows examples of performance evaluation values ​​from NAS and actual measured performance values ​​during inference for each of the neural network models with architectures A and B shown in Figure 1.

[0045] The performance evaluation values ​​shown in Figure 2 represent the Zen-NAS score for a neural network architecture in which one or more blocks included in the architecture are connected. The score includes the sum of a term (also called the Δ term) and a term (also called the σ term) calculated by Zen-NAS, which are based on the index value Δ. The Δ term represents the change in the output value when the input value to the neural network model is changed by a small amount (generally also called a perturbation). The σ term represents the sum of the variances of one or more normalization layers included in the neural network model. The Δ term and σ term can be said to be indicators of the complexity of the neural network architecture, respectively. The number of connected blocks is 1, 4, 7, and 10. The more connected blocks there are, the more pronounced the performance differences between neural network architectures may become.

[0046] Furthermore, the measured performance values ​​shown in Figure 2 represent the accuracy rate when inference is performed using a pre-trained model that was actually trained on a dataset, for a neural network model obtained through neural architecture search using only the architecture in question, under the same inference speed constraints.

[0047] Comparing architecture A and architecture B, architecture A has higher performance evaluation values, while architecture B has higher measured performance values. Furthermore, regarding performance evaluation values, the contribution of the σ term is greater as the number of connected blocks increases.

[0048] Thus, there are instances where the performance evaluation values ​​from Zen-NAS and the actual measured performance values ​​for neural network architectures do not match.

[0049] Figure 3 is a schematic diagram illustrating the configuration of the evaluation device 10 in this embodiment. Figure 4 is a schematic diagram illustrating the processing performed by the acquisition unit 12 of the evaluation device 10 in this embodiment. Figure 5 is a schematic diagram illustrating the processing performed by the calculation unit 13 of the evaluation device 10 in this embodiment.

[0050] The evaluation device 10 is an information processing device that uses a processor to evaluate neural network architectures. Furthermore, the evaluation device 10 can use the results of the above evaluation to search for neural network architectures with relatively high inference performance.

[0051] As shown in Figure 3, the evaluation device 10 comprises a storage unit 11, an acquisition unit 12, a calculation unit 13, and a storage unit 14. Note that the evaluation device 10 does not necessarily have to include a storage unit 14.

[0052] The memory unit 11 stores multiple neural network architectures from the evaluation device 10. The multiple neural network architectures stored in the memory unit 11 are the targets of evaluation by the evaluation device 10. The memory unit 11 may have the neural network architectures to be evaluated pre-stored in it, or it may store the neural network architectures in the storage unit 14.

[0053] The acquisition unit 12 acquires the neural network architecture to be evaluated by the evaluation device 10 from the storage unit 11. The acquisition unit 12 provides the acquired neural network architecture to the calculation unit 13. The processing performed by the acquisition unit 12 is described in detail here (see Figure 4).

[0054] The acquisition unit 12 selects one neural network architecture from among the multiple neural network architectures stored in the memory unit 11 (step S21). The selection of one neural network architecture can be done in any way; for example, it may be done by randomly selecting one neural network architecture from among the multiple neural network architectures stored in the memory unit 11.

[0055] Furthermore, the acquisition unit 12 modifies at least one of the design parameters of the neural network architecture selected in step S21 (step S22). Examples of design parameters include the number of blocks, the number of channels, the stride, and the kernel size. In this way, the acquisition unit 12 acquires a neural network architecture that has been slightly modified from the neural network architecture selected in step S21. This acquisition of a neural network architecture by the acquisition unit 12 can also be described as acquiring a neural network architecture using a genetic algorithm.

[0056] If the evaluation value for the neural network architecture selected in step S21 is unknown, then in step S22, at least one of the design parameters of the neural network architecture may not be changed. If the evaluation value for the neural network architecture is unknown, for example, this corresponds to the case where the calculation unit 13 has not yet calculated an evaluation value for that neural network architecture. In this case, the calculation unit 13 will calculate an evaluation value for the neural network architecture selected in step S21.

[0057] Furthermore, the acquisition unit 12 may fix the number of blocks and change other design parameters in the processing of step S22 (step S23). In other words, the acquisition unit 12 may prohibit the change of the number of blocks in the processing of step S22. More specifically, the acquisition unit 12 may fix the number of blocks so that the ratio of the number of blocks of predetermined types, for example, blocks with data sizes of 56×56 pixels, 28×28 pixels, 14×14 pixels, and 7×7 pixels, is 1:1:3:1. As described above, the acquisition unit 12 can acquire neural network architectures that have been slightly changed with respect to various design parameters from the neural network architecture stored in the memory unit 11 by changing the design parameters. However, some of the design parameters are known to exhibit relatively high inference performance when set to specific values. By prohibiting the change of such design parameters in step S22 (i.e., fixing them), it is possible to efficiently obtain a neural network architecture that exhibits relatively high inference performance.

[0058] The calculation unit 13 calculates the evaluation value of the neural network architecture acquired by the acquisition unit 12 and outputs the calculated evaluation value. The destination to which the calculation unit 13 outputs the evaluation value is, for example, the storage unit 14. This is explained as an example, but the destination to which the evaluation value is output may be an external device of the evaluation device 10.

[0059] Here, the processing performed by the calculation unit 13 will be explained in detail (see Figure 5).

[0060] First, the calculation unit 13 calculates an index value Δ (also called the first index value) that indicates the amount of change in the output value when the input value input to the neural network model having a neural network architecture acquired by the acquisition unit 12 is changed (step S31).

[0061] Furthermore, the calculation unit 13 calculates one or more index values ​​σ (also called the second index value) that represent the variance of one or more normalization layers included in the neural network model acquired by the acquisition unit 12 (step S32). Here, the one or more normalization layers may be one or more layer normalization layers or one or more instance normalization layers.

[0062] Furthermore, the calculation unit 13 obtains the weight for the index value Δ and the weight for each of the index values ​​σ that are 1 or more, as weights to be used when calculating the weighted sum in step S34 (step S33). Here, the case where the weight for the index value Δ is 1 is explained as an example. If the weight for the index value Δ is not 1, the weight for the index value Δ and the weight for each of the index values ​​σ that are 1 are divided by the index value Δ to reduce it to the case where the weight for the index value Δ is 1. Note that the case where the weight for each of the index values ​​σ that are 1 is excluded.

[0063] The weights have the function of adjusting the contribution of the Δ term and the σ term, respectively, to the evaluation value used in the calculation of the evaluation value in step S34. Furthermore, the weights have the function of adjusting the contribution of each of the 1 or more index values ​​σ to the σ term.

[0064] For example, the weights are determined for each of the one or more normalization layers, according to the type of convolutional layer included before that normalization layer in the neural network model.

[0065] More specifically, the weights for each of the one or more normalization layers can be the same as when the layer preceding the normalization layer is a Conv layer, if the layer preceding the normalization layer is a PWConv layer. Also, for example, if the layer preceding the normalization layer is a DWConv layer, the weights can be larger than when the layer preceding the normalization layer is a Conv layer. When the layer preceding the normalization layer is a DWConv layer, the DWConv layer passes relatively few parameters to the subsequent layer, so the contribution of the σ term is relatively small compared to the Δ term. Therefore, in the above case, by increasing the weights for each of the one or more normalization layers, the contributions of the Δ term and the σ term can be made to be roughly equal.

[0066] The calculation unit 13 uses the weights obtained in step S33 to calculate the weighted sum of the index value Δ and each of the index values ​​σ that are 1 or greater as the evaluation value (step S34). For example, the calculation unit 13 calculates the evaluation value E by the following (Equation 1). In (Equation 1), σi corresponds to each of the index values ​​σ that are 1 or greater, and λi corresponds to the weight for each of the index values ​​σ that are 1 or greater. i is an index value that takes an integer value in the range of 1 or more and N or less, and N is the number of normalization layers of the neural network architecture that is the subject of evaluation. In (Equation 1), the first term corresponds to the Δ term, and the second term corresponds to the σ term.

[0067]

number

[0068] The storage unit 14 updates the storage unit 11 by storing the neural network architecture for which the calculation unit 13 has calculated an evaluation value, along with that evaluation value, in the storage unit 11.

[0069] Specifically, the storage unit 14 acquires the evaluation value output by the calculation unit 13. In update control, the storage unit 14 performs update control (also called update control) to update the storage unit 11 so that a predetermined number of neural network architectures with higher evaluation values ​​are stored in the storage unit 11 from a group of neural network architectures that includes the neural network architecture for which the calculation unit 13 calculated the evaluation value and multiple neural network architectures stored in the storage unit 11.

[0070] Specifically, the storage unit 14 compares the evaluation value of the neural network architecture with the lowest evaluation value among the multiple neural network architectures stored in the memory unit 11 with the evaluation value calculated by the calculation unit 13. Then, the storage unit 14 stores the neural network architecture with the higher evaluation value in the memory unit 11 (or maintains it if it is already stored there), and discards the neural network architecture with the lower evaluation value.

[0071] Figure 6 is a flowchart showing the processes performed by the evaluation device 10 in this embodiment.

[0072] In step S101, the acquisition unit 12 acquires the neural network architecture that is the subject of evaluation.

[0073] In step S102, the calculation unit 13 calculates an index value Δ for the neural network architecture obtained in step S101.

[0074] In step S103, the calculation unit 13 calculates the variance σi of each normalization layer for the neural network architecture obtained in step S101. The variance σi of each normalization layer corresponds to an index value σ of 1 or more.

[0075] In step S104, the calculation unit 13 obtains the weights of the variance σi of each normalization layer for the neural network architecture obtained in step S101.

[0076] In step S105, the calculation unit 13 calculates an evaluation value using the index value Δ calculated in step S102, the variance σi calculated in step S103, and the weight calculated in step S104. The calculation unit 13 also outputs the calculated evaluation value.

[0077] Figure 7 is an explanatory diagram illustrating an example of the relationship between the performance evaluation value calculated by the evaluation device in the comparative example and the actual performance measurement value. The evaluation device in the comparative example is, as an example, an evaluation device that calculates the performance evaluation value using Zen-NAS.

[0078] Figure 7 is a graph plotting points 31, 32, 33, 34, and 35, which represent the performance evaluation values ​​calculated by the evaluation device in the comparative example and the measured performance values, for neural network architectures obtained through neural architecture search under the same inference speed constraints using only five different architectural block structures. In the graph, the horizontal axis represents the performance evaluation values, and the vertical axis represents the measured performance values.

[0079] Points 31-35 are scattered on the graph without any apparent regularity. In other words, there is no correlation between the performance evaluation values ​​and the measured performance values ​​for the five neural network architectures; in other words, the performance evaluation values ​​and the measured performance values ​​are inconsistent.

[0080] Figure 8 is an explanatory diagram showing an example of the relationship between the performance evaluation value calculated by the evaluation device 10 in this embodiment and the actual performance measurement value.

[0081] Figure 8 is a graph plotting points 41, 42, 43, 44, and 45, which represent the performance evaluation values ​​calculated by the evaluation device 10 and the measured performance values, for five neural network architectures similar to those in Figure 7. In the graph, the horizontal axis represents the performance evaluation values, and the vertical axis represents the measured performance values.

[0082] Points 41-45 all lie near the upward-sloping line L on the graph. This means that there is a correlation between the performance evaluation values ​​and the measured performance values ​​of the five neural network architectures, or in other words, the performance evaluation values ​​and the measured performance values ​​are generally consistent.

[0083] Figure 9 is an explanatory diagram showing examples of the inference accuracy and inference time of the neural network architecture obtained by the evaluation device in the comparative example and the evaluation device 10 in this embodiment.

[0084] The graph shown in Figure 9 plots points 51, 52, 53, and 54 for four existing neural network architectures, representing the time required for inference (also called inference time) and the inference accuracy of the neural network model possessing that architecture. The horizontal axis of the graph represents inference time, and the vertical axis represents inference accuracy.

[0085] Points 51, 52, 53, and 54 are examples using MobileNetV1, MobileNetV2, MobileNetV3, and EfficientNetB0 as neural network architectures, respectively.

[0086] In general, there is a trade-off relationship between inference time and inference accuracy: longer inference times generally correlate with higher inference accuracy, while shorter inference times correlate with lower accuracy. In other words, on a graph with inference time on the horizontal axis and inference accuracy on the vertical axis, the points representing inference time and inference accuracy for each neural network architecture are located near an upward-sloping straight line.

[0087] Points 51-54, shown in Figure 9, are located near the upward-sloping straight line L1 on the graph, indicating the aforementioned trade-off.

[0088] Furthermore, the graph shown in Figure 9 plots points 51A and 52A, which represent the inference time and inference accuracy for two neural network architectures obtained by searching using evaluation values ​​calculated by the evaluation device 10, based on two of the four existing neural network architectures mentioned above (the neural network architectures related to points 51 and 52).

[0089] Points 51A and 52A, shown in Figure 9, are located near line L2, which is above line L1 on the graph. A higher position on the graph indicates higher inference accuracy.

[0090] Therefore, it can be said that by using the evaluation values ​​calculated by the evaluation device 10, a neural network architecture with higher inference accuracy can be obtained than the typical trade-off relationship between inference time and inference accuracy.

[0091] In the above embodiment, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, the software that implements the evaluation device, etc., in the above embodiment is the following program.

[0092] In other words, this program is a program that causes a computer to execute an evaluation method for evaluating a neural network architecture using a processor, which involves acquiring the neural network architecture that is the subject of evaluation, calculating a weighted sum of a first index value indicating the amount of change in the output value when the input value input to a neural network model having the acquired neural network architecture is changed, and one or more second index values ​​indicating the variance of one or more normalization layers included in the neural network model, and outputting the calculated evaluation value.

[0093] Although the evaluation methods and other aspects relating to one or more embodiments have been described above based on the embodiments, the present invention is not limited to these embodiments. Without departing from the spirit of the present invention, various modifications that a person skilled in the art could conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments. [Industrial applicability]

[0094] This invention can be used in evaluation devices for evaluating neural network architectures. [Explanation of Symbols]

[0095] 10 Evaluation device 11 Storage section 12 Acquisition Department 13 Calculation Section 14 Storage Unit 31, 32, 33, 34, 35, 41, 42, 43, 44, 45, 51, 51A, 52, 52A, 53, 54 points L, L1, L2 straight line L11, L15, L21, L25 PWConv layer L12, L14, L16, L22, L24, L26 Normalization Layer L13 Conv layer L23 DWConv layer

Claims

1. An evaluation method for evaluating a neural network architecture using a processor, The neural network architecture that is the subject of the evaluation is obtained, The evaluation value is calculated as a weighted sum of a first index value, which indicates the amount of change in the output value when the input value to the neural network model having the acquired neural network architecture is changed, and one or more second index values, which indicate the variance of one or more normalization layers included in the neural network model. Output the calculated evaluation value, When calculating the aforementioned evaluation value, For each of the one or more normalization layers, the weighted sum is calculated using weights determined according to the type of convolutional layer included in the preceding stage of the neural network model. The evaluation value is calculated using the weighted sum that was calculated. Evaluation method.

2. The weights for each of the one or more normalization layers are: If the layer preceding the normalization layer is a PWConv (Pointwise Convolution) layer, it is the same as if the layer preceding the normalization layer was a Conv (Convolution) layer. When the layer preceding the normalization layer is a DWConv (Depthwise Convolution) layer, the result is greater than when the layer preceding the normalization layer is a Conv layer. The evaluation method according to claim 1.

3. When obtaining the aforementioned neural network architecture, The neural network architecture is obtained by changing at least one of the multiple design parameters of one of the multiple neural network architectures stored in the memory unit. When changing at least one of the aforementioned multiple design parameters, Of the aforementioned multiple design parameters, the number of blocks of a predetermined type is fixed, and the other design parameters are changed. The evaluation method according to claim 1 or 2.

4. When the evaluation value is calculated for the acquired neural network architecture, The update control is performed to update the storage unit so that a predetermined number of neural network architectures, selected from the acquired neural network architecture and the plurality of neural network architectures stored in the storage unit, are stored in the storage unit, starting with those with the highest evaluation values. Obtaining the aforementioned neural network architecture is This includes obtaining the neural network architecture from the storage unit on which the update control has been performed. The evaluation method described in claim 3.

5. An evaluation device that uses a processor to evaluate neural network architectures, An acquisition unit that acquires the neural network architecture that is the subject of the evaluation, The system includes a calculation unit that calculates and outputs, as an evaluation value, a weighted sum of a first index value, which indicates the amount of change in the output value when the input value input to the neural network model having the neural network architecture acquired by the acquisition unit is changed, and one or more second index values, which indicate the variance of one or more normalization layers included in the neural network model, respectively. When calculating the evaluation value, the calculation unit shall For each of the one or more normalization layers, the weighted sum is calculated using weights determined according to the type of convolutional layer included in the preceding stage of the neural network model. The evaluation value is calculated using the weighted sum that was calculated. Evaluation device.

6. A program that causes a computer to execute the evaluation method described in claim 1.

Images