Information processing method and information processing system

Abstract

This provides an information processing method that can find the appropriate network. [Solution] In the information processing method, the first network is trained by distilling the second network, which is larger in size than the first network (S609), the third network is generated by reducing the size of the second network (S612), the third network is trained by distilling the first network, or the third network is trained by distilling the second network (S613, S614).

Description

Technical Field

[0001] The present disclosure relates to an information processing method for causing a computer to execute, etc.

Background Art

[0002] In Non-Patent Document 1, a method for searching a network structure has been proposed for a neural network model.

Prior Art Document

Non-Patent Document

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with the method proposed in Non-Patent Document 1, it is difficult to find an inference model in which the loss caused by quantization is suppressed.

[0005] Therefore, the present disclosure provides an information processing method, etc. that can find an appropriate network.

Means for Solving the Problems

[0006] An information processing method according to an aspect of the present disclosure is an information processing method for causing a computer to execute, distilling a second network larger in size than a first network to train the first network, generating a third network by reducing the size of the second network, distilling the first network to train the third network, or distilling the second network to train the third network.

[0007] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0008] The information processing method, etc., relating to one aspect of this disclosure makes it possible to find an appropriate network. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a conceptual diagram showing the degradation of detection accuracy in the first reference example. [Figure 2] Figure 2 is a conceptual diagram showing a comparison between the reference network and the network for embedded environments in the first example. [Figure 3] Figure 3 is a conceptual diagram showing a comparison between the reference network and the network for embedded environments in Embodiment 1. [Figure 4] Figure 4 is a block diagram showing the configuration of the information processing system in the second reference example. [Figure 5] Figure 5 is a flowchart illustrating the operation of the information processing system in the second reference example. [Figure 6] Figure 6 is a block diagram showing an example configuration of the information processing system in Embodiment 1. [Figure 7] Figure 7 is a flowchart showing a first example of operation of the information processing system in Embodiment 1. [Figure 8] Figure 8 is a flowchart showing a second example of operation of the information processing system in Embodiment 1. [Figure 9] Figure 9 is a graph showing the size difference evaluation function in Embodiment 2. [Figure 10]FIG. 10 is a flowchart showing a first operation example of the information processing system in Embodiment 2. [ [ [Figure 11] FIG. 11 is a flowchart showing a second operation example of the information processing system in Embodiment 2. [ [ [Figure 12] FIG. 12 is a block diagram showing a configuration example of the information processing system in Embodiment 3. [ [ [Figure 13] FIG. 13 is a flowchart showing a first phase of an operation example of the information processing system in Embodiment 3. [ [ [Figure 14] FIG. 14 is a flowchart showing the second and third phases of an operation example of the information processing system in Embodiment 3. [ [ [Figure 15] FIG. 15 is a block diagram showing a basic implementation example of the information processing system in a plurality of embodiments. [ [ [Figure 16] FIG. 16 is a flowchart showing a basic operation example of the information processing system in a plurality of embodiments. [ [ [Figure 17] FIG. 17 is a block diagram showing another basic implementation example of the information processing system in a plurality of embodiments. [ [ [Embodiments of the Invention] [ [ [

[0010] [ [(Knowledge underlying the present disclosure)] [ In some cases, an inference function based on deep learning may be incorporated into IoT (Internet of Things) devices. Also, from the perspectives of cost and privacy, inference processing may be performed by a processor on the device rather than in a cloud or GPU environment. In these cases, methods such as quantization are used to lightweight the network (NW). Thereby, inference processing based on deep learning is performed by a processor with limited computing resources such as computing power and memory capacity. [ [

[0011] [ Here, the network means an inference model such as a neural network model for performing inference processing including inference processing.

[0012] However, for example, in quantization, a reference network (RefNW) with a floating-point representation is converted into a network (IntNW) for an embedded environment with a fixed-point representation. Such quantization may cause a loss in inference performance. Specifically, there is a possibility that the accuracy deteriorates or that there is a discrepancy in the inference results between the reference network and the network for the embedded environment.

[0013] In Non-Patent Document 1, a method for searching a network structure for a neural network model has been proposed. In the method proposed in Non-Patent Document 1, a network structure with high inference performance and high inference speed is searched. The network structure corresponds to the number of layers, the number of nodes in each layer, and the connection form between nodes, etc. That is, in the method proposed in Non-Patent Document 1, a network structure having the number of layers, the number of nodes in each layer, and the connection form between nodes, etc., with high inference performance and high inference speed is searched.

[0014] However, the method proposed in Non-Patent Document 1 does not consider the loss caused by quantization. Therefore, even if the method proposed in Non-Patent Document 1 is used, there is a possibility that the inference performance will be lost due to quantization for network lightweighting.

[0015] Therefore, for example, an information processing method according to an aspect of the present disclosure is an information processing method to be executed by a computer, which acquires a first inference model as a reference, calculates a second inference model having a larger model size than the first inference model based on the first inference model, quantizes the calculated second inference model to generate a third inference model, trains the third inference model using machine learning, determines whether the performance of the trained third inference model satisfies a condition, and outputs the trained third inference model when the performance satisfies the condition.

[0016] This results in the quantization of the second inference model, which has a larger model size than the first inference model. It is assumed that the performance of the second inference model, which has a larger model size, will not degrade significantly even after quantization. In other words, it is assumed that the loss caused by quantization will be relatively small in the third inference model generated by the quantization of the second inference model, which has a larger model size than the first inference model. Therefore, it becomes possible to find an inference model in which the loss caused by quantization is suppressed.

[0017] Furthermore, for example, the information processing method further acquires setting information indicating the quantization settings for the second inference model, and sets initial values ​​in the calculation of the second inference model based on the setting information and the first inference model.

[0018] This initiates the calculation of the second inference model based on the quantization settings and the first inference model. Consequently, it becomes possible to quickly find the third inference model corresponding to the quantization and the first inference model.

[0019] Furthermore, for example, the information processing method further acquires difficulty information indicating the inference difficulty of at least one of the first inference model, the second inference model, and the third inference model, and sets initial values ​​in the calculation of the second inference model based on the difficulty information and the first inference model.

[0020] This initiates the calculation of the second inference model based on the inference difficulty information and the first inference model. Therefore, it becomes possible to quickly find a third inference model that corresponds to the inference difficulty and the first inference model.

[0021] Furthermore, for example, the calculation of the second inference model is a search for the second inference model using a loss function, wherein the output value of the loss function decreases as the difference between the inference result of the first inference model and the inference result of the third inference model decreases, and the output value decreases as the model size of the second inference model becomes relatively larger than that of the first inference model, and the search for the second inference model is performed in such a way that the output value of the loss function decreases.

[0022] This makes it possible to find an inference model that suppresses the losses caused by quantization, based on the loss function.

[0023] Furthermore, for example, the information processing method may acquire setting information indicating the quantization settings for the second inference model and modify the loss function based on the setting information.

[0024] This makes it possible to find an inference model that suppresses the losses caused by quantization, based on a loss function that corresponds to the quantization settings.

[0025] Furthermore, for example, the loss function is modified such that the output value of the loss function increases as the degree of quantization increases in the settings indicated by the setting information, and the search for the second inference model is performed so that the output value of the loss function is less than or equal to a threshold.

[0026] As a result, the output value of the loss function increases with increasing quantization, but the second inference model is searched for such that the output value of the loss function falls below a threshold. In other words, even when the loss is large due to a high degree of quantization, the second inference model is searched for such that certain conditions for suppressing the loss are met. Therefore, it becomes possible to find an inference model in which the loss is suppressed at a certain level.

[0027] Furthermore, for example, the information processing method further acquires difficulty information indicating the inference difficulty of at least one of the first inference model, the second inference model, and the third inference model, and modifies the loss function based on the difficulty information.

[0028] This makes it possible to find inference models that suppress the losses caused by quantization, based on a loss function that corresponds to the inference difficulty.

[0029] Furthermore, for example, the loss function is modified such that the output value of the loss function increases as the inference difficulty, as indicated by the difficulty information, increases, and the search for the second inference model is performed so that the output value of the loss function is less than or equal to a threshold.

[0030] As a result, the output value of the loss function increases with higher inference difficulty, but the second inference model is searched for so that the output value of the loss function falls below a threshold. In other words, even when the loss is large due to high inference difficulty, the second inference model is searched for so that certain conditions for suppressing the loss are met. Therefore, it becomes possible to find an inference model in which the loss is suppressed at a certain level.

[0031] Furthermore, for example, the information processing method may change the quantization settings for the second inference model if the performance does not satisfy the conditions.

[0032] This may allow the quantization settings to be modified to satisfy the performance requirements. This then makes it possible to find an inference model that meets those performance requirements.

[0033] Furthermore, for example, the conditions include the accuracy or precision of the inference of the third inference model on the inference results of the first inference model or on the reference data, and the change in the setting reduces the degree of quantization if the accuracy or precision of the inference of the third inference model is below a threshold.

[0034] This means that if the accuracy or precision of the third inference model's inference is below a threshold, the quantization applied to the second inference model is narrowed to improve the accuracy or precision of the third inference model's inference. This makes it possible to find an inference model that satisfies the conditions for accuracy or precision of the inference.

[0035] Furthermore, for example, the conditions include the inference processing speed of the third inference model, and the change in the setting increases the degree of quantization if the inference processing speed is below a threshold.

[0036] This means that if the inference processing speed of the third inference model is below a threshold, the degree of quantization for the second inference model is increased so that the inference processing speed of the third inference model becomes faster. This makes it possible to find an inference model that satisfies the inference processing speed condition.

[0037] Furthermore, for example, the information processing method further inputs data into the first inference model to obtain the inference result of the first inference model, inputs the data into the second inference model to obtain the inference result of the second inference model, and trains the first inference model based on the difference between the inference result of the first inference model and the inference result of the second inference model.

[0038] This allows the first and third inference models to be constructed based on the same second inference model. Therefore, it becomes possible to minimize the difference between the inference results of the first inference model and the inference results of the third inference model.

[0039] Furthermore, for example, an information processing system according to one aspect of the present disclosure comprises at least one processor and at least one memory, wherein the at least one processor uses the at least one memory to acquire a reference first inference model, calculates a second inference model having a larger model size than the first inference model based on the first inference model, quantizes the calculated second inference model to generate a third inference model, trains the third inference model using machine learning, determines whether the performance of the trained third inference model satisfies a condition, and outputs the trained third inference model if the performance satisfies the condition.

[0040] This results in the quantization of the second inference model, which has a larger model size than the first inference model. It is assumed that the performance of the second inference model, which has a larger model size, will not degrade significantly even after quantization. In other words, it is assumed that the loss caused by quantization will be relatively small in the third inference model generated by the quantization of the second inference model, which has a larger model size than the first inference model. Therefore, it becomes possible to find an inference model in which the loss caused by quantization is suppressed.

[0041] Furthermore, these comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0042] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the scope of the claims. Furthermore, components in the following embodiments that are not described in an independent claim will be described as optional components.

[0043] Furthermore, in this disclosure, elements may be assigned ordinal numbers such as 1st, 2nd, and 3rd. These ordinal numbers are assigned to elements to identify them and do not necessarily correspond to a meaningful order. These ordinal numbers may be rearranged, newly assigned, or removed as appropriate.

[0044] Furthermore, in this disclosure, inference includes detection, recognition, and identification. Furthermore, in this disclosure, computing includes decision processing, search processing, acquisition processing, derivation processing, and extraction processing.

[0045] Furthermore, in this disclosure, distilling network NW1 to train network NW2 means, for example, using network NW1 as a training network to train network NW2. Also, in this disclosure, training a network means, for example, adjusting the network parameters. Training a network may be rephrased as learning the network. Also, a network may be rephrased as an inference model.

[0046] (Embodiment 1) Figure 1 is a conceptual diagram illustrating the degradation of detection accuracy in the first reference example. Specifically, it shows the difference between the detection results in the reference network and the detection results in the network for the embedded environment.

[0047] For example, a reference network is intended for use in a cloud or GPU environment and is constructed using floating-point representation. On the other hand, a network for embedded environments is intended for use in IoT devices, etc., and is constructed using fixed-point representation. Basically, the reference network is first constructed by training in a cloud or GPU environment. After that, the reference network is converted into a network for embedded environments.

[0048] In embedded environments, resources are limited, so when a reference network is converted to an embedded network, it undergoes a lightweighting process. This lightweighting includes quantization to convert floating-point representation to fixed-point representation. Such lightweighting degrades detection accuracy.

[0049] Specifically, as shown in Figure 1, the reference network detects dogs, people, and horses from the image. In the embedded network, dogs and people are detected from the image, but horses are not. In other words, the embedded network exhibits a degradation in detection accuracy compared to the reference network. That is, the embedded network has an increased number of false negatives (FN) and false positives (FP) compared to the reference network.

[0050] Figure 2 is a conceptual diagram showing a comparison between the reference network and the embedded network in the first example. The embedded network is constructed by making the reference network lighter. This lightening process results in a degradation of inference accuracy. In other words, the inference accuracy of the embedded network is lower than that of the reference network.

[0051] Therefore, the desired performance may not be achieved in networks designed for embedded environments. Furthermore, inconsistencies in inference results may occur between the reference network and the embedded network, potentially increasing the effort required for evaluating and verifying the embedded network.

[0052] Figure 3 is a conceptual diagram showing a comparison between the reference network and the embedded environment network in this embodiment. In this embodiment, an embedded environment network is constructed that has inference accuracy close to that of the reference network.

[0053] This may enable the desired performance to be achieved in networks for embedded environments. Furthermore, it reduces inconsistencies in inference results between reference networks and networks for embedded environments, thereby decreasing the effort required for evaluating and verifying networks for embedded environments.

[0054] Figure 4 is a block diagram showing the configuration of the information processing system in the second reference example. Specifically, it shows the configuration of the information processing system 100 assumed from Non-Patent Document 1. The information processing system 100 comprises a network search unit 101, an evaluation value calculation unit 103, and a learning processing unit 104.

[0055] The network search unit 101 searches for a second network 112 that is likely to improve inference accuracy and inference speed by distilling the first network 111, which is the reference network. This process corresponds to obtaining an evaluation value from the evaluation value calculation unit 103 and searching for a second network 112 that will improve the evaluation value.

[0056] The first network 111 and the second network 112 are inference models, such as neural network models, for performing inference processing. The second network 112 is expected to have a faster inference speed than the first network 111. Therefore, the size of the second network 112 to be explored is assumed to be smaller than the size of the first network 111.

[0057] The size of a network corresponds to the number of nodes, layers, parameters, and connections between nodes included in the network. The more nodes, layers, parameters, and connections between nodes included in the network, the larger the network size. Alternatively, the size of a network may correspond to any one of the following: the number of nodes, layers, parameters, and connections between nodes included in the network.

[0058] The evaluation value calculation unit 103 calculates an evaluation value. Specifically, the evaluation value calculation unit 103 acquires the inference results of the first network 111 and the inference results of the second network 112, and calculates the output value of a loss function related to inference accuracy and inference speed as an evaluation value. In this loss function, the larger the difference between the inference results of the first network 111 and the inference results of the second network 112, the larger the output value, and the larger the processing delay of the second network 112. A smaller evaluation value corresponding to the output value of the loss function is a better value.

[0059] The evaluation value calculation unit 103 may input the same data to the first network 111 and the second network 112 in order to obtain inference results from both networks. Alternatively, data input may be performed by the learning processing unit 104, or by an input unit or network control unit (not shown).

[0060] The learning processing unit 104 updates the second network 112 to improve inference accuracy and inference speed. Specifically, the learning processing unit 104 obtains evaluation values ​​from the evaluation value calculation unit 103 and updates the second network 112 to improve the evaluation values.

[0061] In particular, the learning processing unit 104 updates the second network 112 to improve the evaluation score, thereby reducing the difference between the inference result of the first network 111 and the inference result of the second network 112. In other words, the learning processing unit 104 distills the first network 111 to train the second network 112.

[0062] The network search unit 101, the evaluation value calculation unit 103, and the learning processing unit 104 repeat the above-described processes to obtain a second network 112 with good inference accuracy and inference speed. Furthermore, the search for the second network 112 in the information processing system 100 is an automatically performed search based on the loss function, and is a search for the network structure. This search is also called automatic search or NAS (Neural Architecture Search).

[0063] Figure 5 is a flowchart showing the operation of the information processing system 100 in the second reference example. First, the network search unit 101 sets the first network 111 as the initial value for searching the second network 112 (S101). In other words, the network search unit 101 sets the first network 111 as the starting position for searching the second network 112.

[0064] Next, the learning processing unit 104 trains the second network 112 (S102). Specifically, the learning processing unit 104 trains the second network 112 so that the evaluation value obtained from the evaluation value calculation unit 103 improves. This process corresponds to distilling the first network 111 and training the second network 112.

[0065] Next, the network search unit 101 searches for a second network 112 that will improve the evaluation value obtained based on the training results (S103).

[0066] Then, if the performance of the second network 112 meets the requirements (Yes in S104), the process ends. For example, if the inference accuracy and inference speed of the second network 112 meet the requirements, the process ends. On the other hand, if the performance of the second network 112 does not meet the requirements (No in S104), the training of the second network 112 (S102) and the exploration of the second network 112 (S103) are repeated.

[0067] In the second example, the search for the second network 112 seeks a structure for the second network 112 that has inference accuracy close to that of the first network 111 and faster inference speed than the first network 111. Since the second network 112 has faster inference speed than the first network 111, it is assumed that the size of the second network 112 is smaller than the size of the first network 111.

[0068] However, the second reference example does not consider optimization for embedded networks. In particular, quantization is not considered in the second reference example. Therefore, the second network 112 may not be applicable to embedded networks. Furthermore, optimization including quantization may result in degradation compared to the reference network.

[0069] This embodiment describes an information processing method, etc., that can find an inference model that suppresses the above-mentioned losses. Note that suppressing losses may be rephrased as compensating for losses.

[0070] Figure 6 is a block diagram showing an example configuration of the information processing system in this embodiment. Specifically, it shows the configuration of the information processing system 200. The information processing system 200 includes a network search unit 201, a lightweighting unit 202, an evaluation value calculation unit 203, and a learning processing unit 204, etc.

[0071] The network search unit 201 is an information processing unit that performs information processing. It sets the first network 211 as the initial value for the search and searches for the second network 212, which is likely to suppress the loss caused by the optimization.

[0072] For example, the network search unit 201 acquires the first network 211 as a reference network from an external device of the information processing system 200. Then, the network search unit 201 sets the first network 211 as the starting position for searching for the second network 212. After that, the network search unit 201 acquires an evaluation value from the evaluation value calculation unit 203 and searches for a second network 212 that will improve the evaluation value.

[0073] Furthermore, the number of nodes and the kernel type may be set as initial values ​​for the search. The network search unit 201 may also acquire configuration information indicating the optimization settings and difficulty information indicating the difficulty of inference, and determine the initial values ​​for the search based on the optimization settings and the difficulty of inference.

[0074] In other words, the network search unit 201 may adjust the initial values ​​of the search based on the optimization settings and the difficulty of inference. Specifically, the network search unit 201 may increase the number of nodes if the optimization is stronger than the standard. Also, the network search unit 201 may increase the number of nodes if the difficulty of inference is higher than the standard. Furthermore, a table may be used to determine the initial values ​​of the search based on the optimization settings, the difficulty of inference, or a combination thereof.

[0075] The first network 211 and the second network 212 are inference models, such as neural network models, for performing inference processing. In particular, the network size of the second network 212 is larger than that of the first network 211. This is expected to suppress the losses caused by optimization.

[0076] Furthermore, the first network 211 is the foundational network and the reference network. For example, floating-point representation is used in the first network 211. The second network 212 is an intermediate network that is different from both the reference network and the network for embedded environments. For example, floating-point representation is also used in the second network 212.

[0077] The lightweighting unit 202 is an information processing unit that performs information processing, and it lightweights the second network 212 to generate the third network 213. The lightweighting is performed for several purposes, as follows.

[0078] One of the objectives is to reduce the execution latency of the network. Execution latency is expressed in milliseconds (ms) or microseconds (μs).

[0079] Another objective is to reduce the required computational complexity. This complexity is expressed in terms of the number of operations (or operations).

[0080] Another objective is to reduce the amount of memory required for weight storage memory, intermediate feature storage memory, etc. The required amount of memory is expressed in units of bits.

[0081] Another objective is to reduce the amount of memory that needs to be transferred. The amount of memory that needs to be transferred mainly represents the amount of data transferred between the processor of the device performing the inference processing and the external DRAM, and is expressed in units of bits per second. However, the memory that the processor interacts with is not limited to the external DRAM.

[0082] Another objective is to reduce power consumption and energy consumption. Power consumption is expressed in watts (W) or milliwatts (mW), and energy consumption is expressed in watt-hours (Wh). Power consumption and energy consumption are determined by a combination of factors, including the hardware on which the inference processing is performed, the required computational power, and the required memory transfer amount.

[0083] Another objective is to miniaturize devices that incorporate neural network models, deep learning models, or machine learning models. The instrument size indicator for miniaturization is cubic centimeters (cm). 3 ) or cubic millimeters (mm 3 It is expressed as follows: Equipment size is determined by a combination of factors such as the equipment's power consumption, thermal capacity, required network execution latency, and the size of the equipment's components.

[0084] Furthermore, not all of the above are for the purpose of weight reduction, and some of the above may be for the purpose of weight reduction.

[0085] The optimizations performed for the above purposes basically involve quantization. For example, the quantization may be one that changes floating-point representation to fixed-point representation. Furthermore, the quantization is not limited to changing floating-point representation to fixed-point representation; it may also be one that changes the representation format to a representation format with fewer bits.

[0086] For example, network quantization changes the representation format of network parameters, input data values, intermediate data values, and output data values ​​to a representation format with fewer bits. It is also possible that only some, or not all, of the representation formats of network parameters, input data values, intermediate data values, and output data values ​​are changed to a representation format with fewer bits.

[0087] Furthermore, optimization may include reducing the network size, such as by reducing the number of layers, the number of nodes, and the number of connections between nodes. Alternatively, optimization may include only quantization. In other words, optimization may consist of quantization. Or, optimization that does not include quantization may be used.

[0088] Furthermore, for example, the lightweighting unit 202 acquires lightweighting setting information and lightweights the second network 212 based on the setting information. The setting information may include the number of quantization bits (in other words, the degree of quantization), the amount of reduction in the number of layers, the amount of reduction in the number of nodes, and the amount of reduction in the number of connections between nodes. Here, a small number of quantization bits (in other words, a large degree of quantization) corresponds to a wide quantization width, and a large number of quantization bits (in other words, a small degree of quantization) corresponds to a narrow quantization width. The lightweighting setting information may also be stored in memory or the like as lightweighting setting 231.

[0089] The third network 213 is an inference model, such as a neural network model, for performing inference processing. Generally, the network size of the third network 213 is larger than that of the first network 211. However, this configuration is not limited to this; the network size of the third network 213 may be smaller than that of the first network 211.

[0090] Furthermore, the third network 213 is a network designed for embedded environments. Specifically, the third network 213 is a network that achieves the goal of lightweight design. For example, the third network 213 uses fixed-point representation.

[0091] The evaluation value calculation unit 203 is an information processing unit that performs information processing and calculates an evaluation value. Specifically, the evaluation value calculation unit 203 calculates the inference accuracy of the third network 213 as an evaluation value.

[0092] For example, the evaluation value calculation unit 203 obtains the inference result of the first network 211 and the inference result of the third network 213 for the same input data. The evaluation value calculation unit 203 may then calculate the difference between the inference result of the first network 211 and the inference result of the third network 213 as the evaluation value. Alternatively, the evaluation value calculation unit 203 may calculate the difference between the correct answer data 232 stored in memory or the like and the inference result of the third network 213 as the evaluation value. In such cases, a smaller evaluation value is better.

[0093] The evaluation value calculation unit 203 may input the same data to the first network 211 and the second network 212 in order to obtain inference results from both networks. Alternatively, data input may be performed by the learning processing unit 204, or by an input unit or network control unit (not shown).

[0094] Furthermore, for example, the evaluation value calculated by the evaluation value calculation unit 203 is used in the network search unit 201 and the learning processing unit 204. The evaluation value calculation unit 203 may calculate the evaluation value used in the network search unit 201 and the evaluation value used in the learning processing unit 204 using different criteria.

[0095] Specifically, for example, the evaluation value calculation unit 203 may calculate the difference between the inference result of the first network 211 and the inference result of the third network 213 as the evaluation value used in the network search unit 201. Alternatively, the evaluation value calculation unit 203 may calculate the difference between the ground truth data 232 and the inference result of the third network 213 as the evaluation value used in the learning processing unit 204.

[0096] The learning processing unit 204 is an information processing unit that performs information processing and updates the third network 213 to improve its inference accuracy. Specifically, the learning processing unit 204 obtains evaluation values ​​from the evaluation value calculation unit 203 and updates the third network 213 to improve the evaluation values.

[0097] For example, the learning processing unit 204 may update the inference results of the first network 211 and the third network 213 in accordance with an evaluation value corresponding to the difference between the inference results of the first network 211 and the third network 213, so as to reduce the difference between them. In other words, the learning processing unit 204 may distill the first network 211 and train the third network 213.

[0098] Alternatively, the learning processing unit 204 may perform adversarial learning on the third network 213. Adversarial learning may be performed based on a comparison between the inference results of the first network 211 and the inference results of the third network 213, or based on a comparison between the ground truth data 232 and the inference results of the third network 213. Alternatively, the learning processing unit 204 may perform distance learning on the third network 213.

[0099] Furthermore, the learning processing unit 204 may change the optimization setting 231 according to evaluation values, etc. That is, the learning processing unit 204 may change the optimization setting 231 according to the inference accuracy of the third network 213, etc. For example, the learning processing unit 204 may reduce the optimization as the inference accuracy of the third network 213 decreases.

[0100] The network search unit 201, the optimization unit 202, the evaluation value calculation unit 203, and the learning processing unit 204 repeat the above-described processes to obtain a third network 213 with good inference accuracy as a network for embedded environments. Furthermore, because optimization has been performed, the inference speed of the third network 213 is fast. The learning processing unit 204 or other components may output the final third network 213 as a network for embedded environments.

[0101] The information processing system 200 may also include a difficulty calculation unit 205. The difficulty calculation unit 205 is an information processing unit that performs information processing and calculates the difficulty of the inference based on the dataset 233. The difficulty may also be stored in memory or the like as a task difficulty 234.

[0102] For example, the difficulty calculation unit 205 may calculate the difficulty of the inference based on the amount and number of types of data in the dataset 233 for inference. The difficulty calculation unit 205 may also calculate the difficulty based on the type of inference, regardless of the dataset 233. For example, the difficulty of the detection process may be higher than the difficulty of the identification process.

[0103] Furthermore, the information processing system 200 may include a learning processing unit 206. This learning processing unit 206 updates the second network 212 to improve its inference accuracy. Specifically, the learning processing unit 206 may train the second network 212 using the correct answer data 232 as training data. Alternatively, the learning processing unit 206 may train the second network 212 by distilling the first network 211.

[0104] Alternatively, the learning processing unit 206 may perform adversarial learning on the second network 212. Adversarial learning may be performed based on a comparison between the inference results of the first network 211 and the inference results of the second network 212, or based on a comparison between the ground truth data 232 and the inference results of the second network 212. Alternatively, the learning processing unit 206 may perform distance learning on the second network 212.

[0105] Figure 7 is a flowchart showing a first example of operation of the information processing system 200 in this embodiment.

[0106] First, the network search unit 201 acquires the first network 211 and sets the first network 211 as the initial value for searching the second network 212 (S201). Next, the network search unit 201 acquires configuration information indicating the optimization settings and difficulty information indicating the difficulty of inference (S202). Then, the network search unit 201 determines the initial value for searching the second network 212 based on the optimization settings and the difficulty of inference (S203).

[0107] For example, the network search unit 201 designates the first network 211 as the second network 212. Then, the network search unit 201 adjusts the second network 212 based on the optimization settings and the difficulty of inference, and determines the second network 212. This determines the second network 212 in the initial stages of the search.

[0108] Next, the learning processing unit 206 trains the second network 212 (S204). This step may be omitted.

[0109] Next, the lightweighting unit 202 lightens the second network 212 based on the lightweighting settings to generate the third network 213 (S205).

[0110] Next, the learning processing unit 204 trains the third network 213 (S206). Specifically, the learning processing unit 204 trains the third network 213 so that the evaluation value obtained from the evaluation value calculation unit 203 improves. This process may correspond to training the third network 213 by distilling the first network 211, or it may correspond to other training methods.

[0111] Then, if the performance of the third network 213 meets the requirements (Yes in S207), the process ends. For example, if the inference accuracy and inference speed of the third network 213 meet the requirements, the process ends. On the other hand, if the performance of the third network 213 does not meet the requirements (No in S207), the network search unit 201 changes the number of nodes in each layer or a specific layer (S208). Then, the process is repeated starting from training the second network 212 (S204).

[0112] The above inference accuracy may include both precision and accuracy. Furthermore, as accuracy, the correctness may be expressed as the rate of agreement between the ground truth data 232 and the inference results of the third network 213, or as the reference agreement rate, which is the rate of agreement between the inference results of the first network 211 and the inference results of the third network 213. In addition, the above inference speed may correspond to the processing time of the inference.

[0113] For example, the above requirement may also be that the inference accuracy of the third network 213 is above a certain standard, or that the inference speed of the third network 213 is above a certain standard. Specifically, the above requirement may also be that the correct answer match rate is 90% or higher. Furthermore, the above requirement may also be that the reference match rate is 98% or higher. Furthermore, the above requirement may also be that the processing time is 20ms or less. Furthermore, the above requirement may also be a combination of these.

[0114] The network exploration unit 201 may determine whether the performance of the third network 213 meets the requirements based on the evaluation value obtained from the evaluation value calculation unit 203. The network exploration unit 201 may also change the number of nodes in each layer or a specific layer to improve the evaluation value obtained from the evaluation value calculation unit 203.

[0115] For example, the evaluation value obtained from the evaluation value calculation unit 203 may indicate the inference accuracy of the third network 213. The inference accuracy of the third network 213 may indicate the difference between the inference result of the third network 213 and the inference result of the first network 211, or it may indicate the difference between the inference result of the third network 213 and the ground truth data 232. If the inference accuracy of the third network 213 is poor, the network search unit 201 may increase the number of nodes in each layer or a specific layer.

[0116] Then, the training of the second network 212 (S204), the optimization of the second network 212 (S205), the training of the third network 213 (S206), and the change in the number of nodes (S208) are repeated until the performance of the third network 213 meets the requirements. This results in a third network 213 whose performance meets the requirements. The learning processing unit 204 or other components may output the final third network 213.

[0117] Figure 8 is a flowchart showing a second example of operation of the information processing system 200 in this embodiment.

[0118] First, the network search unit 201 acquires the first network 211 and sets the first network 211 as the initial value for searching the second network 212 (S301). Next, the network search unit 201 acquires setting information indicating the optimization settings and difficulty information indicating the difficulty of inference (S302). Then, the network search unit 201 determines the initial value for searching the second network 212 based on the optimization settings and the difficulty of inference (S303).

[0119] Next, the learning processing unit 206 optionally trains the second network 212 (S304). Then, the optimization unit 202 optimizes the second network 212 based on the optimization settings to generate the third network 213 (S305). Next, the learning processing unit 204 trains the third network 213 (S306). The processing up to this point is the same as the processing in the first operation example in this embodiment.

[0120] Next, if the performance of the third network 213 does not meet the first requirement (No in S307), the network discovery unit 201 changes the number of layers (S308). Then, the process is repeated starting from training the second network 212 (S304).

[0121] The network discovery unit 201 may determine whether the performance of the third network 213 meets the first requirement, according to the evaluation value obtained from the evaluation value calculation unit 203. The network discovery unit 201 may also change the number of layers to improve the evaluation value obtained from the evaluation value calculation unit 203.

[0122] For example, the evaluation value obtained from the evaluation value calculation unit 203 may indicate the inference accuracy of the third network 213. The inference accuracy of the third network 213 may indicate the difference between the inference result of the third network 213 and the inference result of the first network 211, or it may indicate the difference between the inference result of the third network 213 and the ground truth data 232. If the inference accuracy of the third network 213 is poor, the network search unit 201 may increase the number of layers.

[0123] Furthermore, if the performance of the third network 213 satisfies the first requirement but does not satisfy the second requirement (Yes in S307 and No in S309), the network discovery unit 201 changes the number of nodes in each layer or a specific layer (S310). This process is the same as the process in the first operation example in this embodiment. Then, the process is repeated starting from training the second network 212 (S304).

[0124] Then, the training of the second network 212 (S304), the optimization of the second network 212 (S305), the training of the third network 213 (S306), the change in the number of layers (S308), and the change in the number of nodes (S310) are repeated until the first and second requirements are met. When the performance of the third network 213 meets the first and second requirements (Yes in S307 and Yes in S309), the process ends. This results in a third network 213 whose performance meets the first and second requirements.

[0125] For example, the second requirement is higher than the first requirement. In other words, the second requirement is stricter than the first requirement. It is assumed that changing the number of layers has a greater impact on performance than changing the number of nodes. Therefore, the number of layers is changed to satisfy the less stringent first requirement, and then the number of nodes is changed to satisfy the stricter second requirement. This is expected to allow for the early discovery of a third network 213 that has the performance to satisfy the strict second requirement.

[0126] As described above, the information processing system 200 in this embodiment searches for a second network 212 that is larger than the first network 211. Then, the information processing system 200 reduces the size of the second network 212 to generate a third network 213 and trains the third network 213.

[0127] This could potentially lead to the discovery of a third network 213 that suppresses the losses caused by quantization included in the weight reduction.

[0128] The information processing system 200 may include some of the components shown in this embodiment, or it may perform some of the processes shown in this embodiment. Furthermore, at least some of the components and processes shown in this embodiment may be combined with at least some of the components and processes shown in other embodiments.

[0129] (Embodiment 2) The configuration example in this embodiment is the same as the configuration example shown in Figure 6. However, in this embodiment, network automatic search is performed using a loss function. This loss function is used to find an intermediate network (i.e., a second network 212) and an embedded network (i.e., a third network 213) that suppresses the loss caused by quantization included in the lightweighting process. This allows for the efficient discovery of an embedded network. Specifically, the following L(x) is used as the loss function.

[0130] L(x) = CE(x, TargetNetQuant) ·α·log(LAT(TargetNetQuant)) β ·γ·Diff(RefNet(x), TargetNetQuant(x)) δ +λ·R_size(SizeDiff(RefNet, TargetNet))

[0131] Here, x represents the input to the network. RefNet represents the reference network (i.e., the first network 211). TargetNet represents the intermediate network (i.e., the second network 212). TargetNetQuant represents the embedded network (i.e., the third network 213).

[0132] CE(x, TargetNetQuant) is the cross-entropy term, representing the difference between the inference result and the ground truth data (232) in an embedded network. CE is the cross-entropy function. This term has the same properties as normal training.

[0133] α·log(LAT(TargetNetQuant)) β This is the latency term, which relates to the execution speed of the embedded network. LAT is a function that represents the amount of latency. LAT(TargetNetQuant) represents the amount of latency of the embedded network. Also, α and β are coefficients used to adjust the output value of the latency term.

[0134] γ Diff(RefNet(x), TargetNetQuant(x)) δ `RefNet(x), TargetNetQuant(x)` is a reference embedded equivalent term that represents the difference between the reference network and the embedded network. `Diff` is a function that represents the difference. `Diff(RefNet(x), TargetNetQuant(x))` represents the difference between the inference result of the reference network and the inference result of the embedded network.

[0135] The difference may be an absolute difference, a squared absolute difference, a Euclidean distance, or a cosine distance. γ and δ are coefficients for adjusting the output value of the reference built-in equivalent term.

[0136] λ·R_size(SizeDiff(RefNet, TargetNet)) is a constraint term for the size difference. SizeDiff is a function that represents the size difference. SizeDiff(RefNet, TargetNet) represents the size difference between the reference network and the intermediate network.

[0137] The difference in the number of parameters, the difference in the number of channels, or the difference in the number of layers may be used as the size difference. Alternatively, the difference in the number of parameters weighted per layer, for example by reducing the weight of important layers, may be used as the size difference.

[0138] R_size is a size difference evaluation function used to evaluate the size difference. R_size(SizeDiff(RefNet, TargetNet)) is smaller the larger the size difference between the reference network and the intermediate network, and larger the smaller the size difference between the reference network and the intermediate network.

[0139] The intermediate network is defined as being larger than the reference network. Therefore, R_size(SizeDiff(RefNet, TargetNet)) is smaller the larger the intermediate network is relative to the reference network. Also, λ is a coefficient used to adjust the output value of the size difference constraint term.

[0140] Figure 9 is a graph showing the size difference evaluation function in this embodiment. The larger the input value (x) of the size difference evaluation function, the smaller the output value (R_size(x)) of the size difference evaluation function. Also, ε and θ included in the size difference evaluation function in Figure 9 are coefficients for adjusting the output value of the size difference evaluation function. Figure 9 shows examples where ε=1 and θ=10.

[0141] The evaluation value calculation unit 203 performs the calculation of the loss function described above. The evaluation value calculation unit 203 then outputs the output value of the loss function as the evaluation value. The network search unit 201 obtains the output value of the loss function from the evaluation value calculation unit 203 as the evaluation value and searches for the second network 212 in such a way that the output value of the loss function becomes smaller.

[0142] Alternatively, the learning processing unit 204 may obtain the output value of the loss function from the evaluation value calculation unit 203 as an evaluation value, and train the third network 213 so that the output value of the loss function becomes smaller. Or, the evaluation value calculation unit 203 may calculate an evaluation value used in the learning processing unit 204 separately from the evaluation value used in the network search unit 201. Then, the learning processing unit 204 may train the third network 213 based on an evaluation value different from the output value of the loss function.

[0143] Furthermore, the evaluation value calculation unit 203 may set γ, δ, λ, ε, and θ based on the weight reduction setting and the difficulty of the inference. For example, these coefficients may be adjusted depending on whether the weight reduction setting is proactive or passive.

[0144] Regarding pruning to reduce the number of nodes, aggressive optimization means a high reduction in the number of nodes, while passive optimization means a low reduction in the number of nodes. Regarding quantization, aggressive optimization means a small number of quantization bits and a wide quantization width, while passive optimization means a large number of quantization bits and a narrow quantization width. Furthermore, aggressive optimization may also mean a high reduction in the number of layers, while passive optimization may mean a low reduction in the number of layers.

[0145] For example, if optimization is aggressive, it is assumed that inference accuracy will decrease. Therefore, when optimization is aggressive, the evaluation value calculation unit 203 changes γ, δ, λ, ε, and θ of the loss function to increase the output value of the loss function. Then, the second network 212 is searched so that the output value of the loss function falls below a threshold. This can suppress the decrease in inference accuracy.

[0146] Alternatively, if optimization is a priority, γ and δ are set to be large so that the weight of the difference between the inference results of the reference network and the inference results of the embedded network is large. In this case, λ and θ are set to be large and ε to be small so that the weight of the size difference is large.

[0147] As a result, the smaller the difference between the inference results of the reference network and the inference results of the embedded network, and the larger the size of the intermediate network, the smaller the output value of the loss function becomes.

[0148] The network search unit 201 then searches for an intermediate network in such a way that the output value of the loss function becomes smaller. In other words, the network search unit 201 searches for an intermediate network in such a way that the difference between the inference result of the reference network and the inference result of the embedded network becomes smaller, and the size of the intermediate network becomes larger. This suppresses a decrease in inference accuracy. Furthermore, even if the size of the intermediate network is large, the decrease in inference speed is suppressed by actively optimizing its size.

[0149] If optimization is not prioritized, it is assumed that the inference accuracy will not decrease significantly. Therefore, the opposite settings are applied. For example, if optimization is not prioritized, the evaluation value calculation unit 203 modifies the γ, δ, λ, ε, and θ of the loss function to reduce the output value of the loss function. Alternatively, in this case, γ and δ are set to be small so that the weight of the difference between the inference result of the reference network and the inference result of the embedded network is small. Also in this case, λ and θ are set to be small and ε is set to be large so that the weight of the size difference is small.

[0150] Furthermore, it is assumed that inference accuracy decreases when the difficulty of inference is high. Therefore, when the difficulty of inference is high, the evaluation value calculation unit 203 changes γ, δ, λ, ε, and θ of the loss function to increase the output value of the loss function. Then, the second network 212 is searched so that the output value of the loss function falls below a threshold. This can suppress the decrease in inference accuracy.

[0151] Alternatively, if the inference difficulty is high, γ and δ are set to be large so that the weight of the difference between the inference result of the reference network and the inference result of the embedded network is large. In this case, λ and θ are set to be large and ε is set to be small so that the weight of the size difference is large. This suppresses the decrease in inference accuracy.

[0152] When the inference difficulty is low, it is assumed that the inference accuracy will not decrease significantly. Therefore, the opposite settings are applied. For example, when the inference difficulty is low, the evaluation value calculation unit 203 changes γ, δ, λ, ε, and θ of the loss function to reduce the output value of the loss function. Alternatively, in this case, γ and δ are set small so that the weight of the difference between the inference result of the reference network and the inference result of the embedded network is small. Also in this case, λ and θ are set small and ε is set large so that the weight of the size difference is small.

[0153] The evaluation value calculation unit 203 may adjust one or more coefficients among γ, δ, λ, ε, and θ based on the optimization settings and the difficulty of inference, while maintaining the other coefficients at their initial values. For example, the evaluation value calculation unit 203 may adjust ε and maintain θ among the ε and θ included in the R_size function. Alternatively, for example, the evaluation value calculation unit 203 may adjust only λ, θ, and ε related to the size difference among γ, δ, λ, ε, and θ.

[0154] Figure 10 is a flowchart showing a first example of operation of the information processing system 200 in this embodiment.

[0155] First, the network search unit 201 acquires the first network 211 and sets the first network 211 as the initial value for searching the second network 212 (S401). Next, the network search unit 201 acquires setting information indicating the optimization settings and difficulty information indicating the difficulty of inference (S402). Then, the network search unit 201 determines the initial value for searching the second network 212 based on the optimization settings and the difficulty of inference (S403).

[0156] The processing up to this point is the same as the processing in the first and second operation examples in Embodiment 1.

[0157] Next, the evaluation value calculation unit 203 sets the coefficients of the loss function (S404). Specifically, the evaluation value calculation unit 203 may acquire setting information indicating the optimization settings and difficulty information indicating the difficulty of inference, and set the coefficients of the loss function based on the optimization settings and the difficulty of inference.

[0158] Next, the learning processing unit 206 optionally trains the second network 212 (S405). Then, the lightweighting unit 202 lightweights the second network 212 based on the lightweighting settings to generate the third network 213 (S406). Next, the learning processing unit 204 trains the third network 213 (S407). These processes are the same as those in the first and second operation examples in Embodiment 1.

[0159] Next, the network search unit 201 searches for a second network 212 that will improve the evaluation value obtained based on the training results (S408). Specifically, the network search unit 201 searches for a new second network 212 that will reduce the output value of the loss function described above. That is, the network search unit 201 adjusts the number of layers and nodes of the second network 212 so that the output value of the loss function described above is reduced.

[0160] Furthermore, the processes from training the second network 212 (S405) to exploring the second network 212 (S408) may be repeated before determining whether the performance of the third network 213 meets the requirements (S409).

[0161] Then, if the performance of the third network 213 meets the requirements (Yes in S409), the process ends. For example, if the inference accuracy and inference speed of the third network 213 meet the requirements, the process ends. On the other hand, if the performance of the third network 213 does not meet the requirements (No in S409), the process from training the second network 212 (S405) to exploring the second network 212 (S408) is repeated. This results in a third network 213 whose performance meets the requirements.

[0162] The above requirements may be those shown in Embodiment 1. For example, the above requirements may be that the correct answer match rate is 90% or higher, the reference match rate is 98% or higher, and the processing time is 20ms or less. Alternatively, the output value of the loss function may be used as an indicator of the performance of the third network 213. Furthermore, the above requirement may be that the output value of the loss function is below a threshold.

[0163] Figure 11 is a flowchart showing a second example of operation of the information processing system 200 in this embodiment.

[0164] First, the network search unit 201 acquires the first network 211 and sets the first network 211 as the initial value for searching the second network 212 (S501). Next, the network search unit 201 acquires setting information indicating the optimization settings and difficulty information indicating the difficulty of inference (S502). Then, the network search unit 201 determines the initial value for searching the second network 212 based on the optimization settings and the difficulty of inference (S503).

[0165] Next, the evaluation value calculation unit 203 sets the coefficients of the loss function (S504). Next, the learning processing unit 206 trains the second network 212 as an optional operation (S505). Next, the optimization unit 202 optimizes the second network 212 based on the optimization settings to generate the third network 213 (S506). Next, the learning processing unit 204 trains the third network 213 (S507). Next, the network search unit 201 searches for the second network 212 (S508).

[0166] The processing up to this point is the same as the processing in the first operation example in Embodiment 1. Note that, as in the first operation example, the processing from training the second network 212 (S505) to exploring the second network 212 (S508) may be repeated before determining whether the performance of the third network 213 satisfies the first requirement (S509).

[0167] Next, if the performance of the third network 213 does not meet the first requirement (No in S509), the learning processing unit 204 makes a passive change to the lightweight setting 231 (S510). In other words, the learning processing unit 204 changes the lightweighting for the second network 212 to a more passive lightweighting. Specifically, the learning processing unit 204 may increase the number of quantization bits, or decrease the reduction rate of the number of layers and nodes.

[0168] Then, the process is repeated starting from setting the coefficients of the loss function (S504). Note that in setting the coefficients of the loss function (S504), the coefficients of the loss function are set based on the modified lightweight settings 231.

[0169] Furthermore, if the performance of the third network 213 satisfies the first requirement but does not satisfy the second requirement (Yes in S509 and No in S511), the process is repeated starting from setting the coefficients of the loss function (S504). In this case, the lightweight setting 231 is not changed. Therefore, the process may be repeated starting from training the second network 212 (S505).

[0170] Then, if the performance of the third network 213 satisfies the first requirement, the second requirement, and does not satisfy the third requirement (Yes in S509, Yes in S511, and No in S512), the learning processing unit 204 makes a more aggressive change to the lightweight setting 231 (S513). In other words, the learning processing unit 204 changes the lightweighting for the second network 212 to a more aggressive lightweighting. Specifically, the learning processing unit 204 may reduce the number of quantization bits, or increase the reduction rate of the number of layers and nodes.

[0171] Then, the process is repeated starting from setting the coefficients of the loss function (S504). Note that in setting the coefficients of the loss function (S504), the coefficients of the loss function are set based on the modified lightweight settings 231.

[0172] The process from setting the coefficients of the loss function (S504) to searching for the second network 212 (S508) is repeated until the performance of the third network 213 satisfies the first, second, and third requirements. If the performance of the third network 213 satisfies the first, second, and third requirements (Yes in S509, Yes in S511, and Yes in S512), the process ends. This results in a third network 213 whose performance satisfies the first, second, and third requirements.

[0173] For example, the second requirement is higher than the first requirement. In other words, the second requirement is stricter than the first requirement. After the lightweight setting 231 is modified to satisfy the less stringent first requirement, the second network 212 is searched for to satisfy the stricter second requirement. This is expected to lead to the early discovery of a third network 213 that has the performance to satisfy the strict second requirement.

[0174] Furthermore, the first and second requirements may be requirements for the inference accuracy of the third network 213, and the third requirement may be a requirement for the inference speed of the third network 213.

[0175] In this case, first, the second network 212 is searched for the inference accuracy of the third network 213. Based on the search results for inference accuracy, the second network 212 is searched for the inference speed of the third network 213, along with actively changing the optimization settings 231. This makes it possible to efficiently find a third network 213 that satisfies multiple requirements for inference accuracy and inference speed.

[0176] Specifically, for example, the first requirement may be that the correct answer match rate is 70% or higher and the reference match rate is 90% or higher. The second requirement may be that the correct answer match rate is 80% or higher and the reference match rate is 95% or higher. The third requirement may be that the processing time is 20ms or less.

[0177] Furthermore, in the above operation, the process from setting the coefficients of the loss function (S504) to searching for the second network 212 (S508) is repeated. However, the process from determining the initial values ​​for the search (S503) to searching for the second network 212 (S508) may also be repeated. Alternatively, the process from training the second network 212 (S505) to searching for the second network 212 (S508) may also be repeated.

[0178] As described above, the information processing system 200 in this embodiment searches for a second network 212 using a loss function. This loss function is used to find a second network 212 and a third network 213 such that the loss caused by quantization included in the lightweighting process is suppressed. This allows for the efficient discovery of a network suitable for embedded systems. There is a possibility that a third network 213 may be found.

[0179] The information processing system 200 may include some of the components shown in this embodiment, or it may perform some of the processes shown in this embodiment. Furthermore, at least some of the components and processes shown in this embodiment may be combined with at least some of the components and processes shown in other embodiments.

[0180] (Embodiment 3) In Embodiments 1 and 2, for example, the first network 211 is provided by a third party, so the parameters included in the first network 211 are fixed and not changed.

[0181] In this embodiment, the above is not limiting, and changes to the parameters included in the first network 211 are permitted. Then, operations are performed to bring the inference results of the first network 211 and the inference results of the third network 213 closer together. As a result, it is expected that similar inference results can be obtained whether the device uses floating-point representation or fixed-point representation.

[0182] Specifically, in this embodiment, the information processing system 200 generates a third network 213 that obtains inference results with a high degree of agreement with the inference results of the first network 211 by changing the parameters of the first network 211 without fixing the parameters of the first network 211.

[0183] More specifically, the information processing system 200 distills the second network 212 to train the first network 211. Furthermore, the information processing system 200 generates the third network 213 by streamlining the second network 212.

[0184] Subsequently, the information processing system 200 trains the first network 211 and the third network 213, which share the second network 212 as their parent, so that their inference results approach each other. Distillation learning, adversarial learning, or distance learning may be used to train the first network 211 and the third network 213.

[0185] In this embodiment, since the first network 211 and the third network 213 have the same parent, an improvement in the reference match rate is expected compared to Embodiments 1 and 2.

[0186] Figure 12 is a block diagram showing an example configuration of the information processing system 200 in this embodiment. The information processing system 200 in this embodiment includes the same components as those of the information processing system 200 in Embodiments 1 and 2. However, in this embodiment, the inference results of the second network 212 are referenced in the evaluation value calculation unit 203. In addition, the learning processing unit 204 updates the first network 211.

[0187] The operation of the information processing system 200 in this embodiment can be divided into three phases: the first phase, the second phase, and the third phase.

[0188] In the first phase, the larger second network 212 is trained. In the second phase, the second network 212 is distilled and the first network 211 is trained. In the third phase, the third network 213 is generated by lightening the second network 212, and the first network 211 is distilled and the third network 213 is trained. In the third phase, the second network 212 may also be distilled and the third network 213 trained.

[0189] Specifically, in the first phase, the learning processing unit 206 trains the second network 212. Note that the first phase may be omitted.

[0190] In the second phase, the evaluation value calculation unit 203 acquires the inference results of the first network 211 and the inference results of the second network 212, and calculates an evaluation value that shows the difference between the inference results of the first network 211 and the inference results of the second network 212.

[0191] The learning processing unit 204 then trains the first network 211 based on an evaluation value that shows the difference between the inference result of the first network 211 and the inference result of the second network 212. Specifically, the learning processing unit 204 trains the first network 211 so that the difference between the inference result of the first network 211 and the inference result of the second network 212 becomes smaller.

[0192] In the third phase, the lightweighting unit 202 lightweights the second network 212 to generate the third network 213. Then, the evaluation value calculation unit 203 obtains the inference results of the first network 211 and the inference results of the third network 213, and calculates an evaluation value that shows the difference between the inference results of the first network 211 and the inference results of the third network 213.

[0193] The learning processing unit 204 then trains the third network 213 based on an evaluation value that shows the difference between the inference result of the first network 211 and the inference result of the third network 213. Specifically, the learning processing unit 204 trains the third network 213 so that the difference between the inference result of the first network 211 and the inference result of the third network 213 becomes smaller.

[0194] Furthermore, in the third phase, the evaluation value calculation unit 203 may acquire the inference results of the second network 212 and the inference results of the third network 213, and calculate an evaluation value that shows the difference between the inference results of the second network 212 and the inference results of the third network 213.

[0195] The learning processing unit 204 may then train the third network 213 based on an evaluation value that indicates the difference between the inference result of the second network 212 and the inference result of the third network 213. Specifically, the learning processing unit 204 may train the third network 213 so that the difference between the inference result of the second network 212 and the inference result of the third network 213 becomes smaller.

[0196] The second network 212 has high expressive power. The first network 211 and the third network 213 reflect the information from the highly expressive second network 212. Therefore, the first network 211 and the third network 213 are expected to have similar inference accuracy.

[0197] Furthermore, the information processing system 200 may have three learning processing units corresponding to the three networks instead of the learning processing units 204 and 206. That is, the information processing system 200 may have a first network learning processing unit for training the first network 211, a second network learning processing unit for training the second network 212, and a third network learning processing unit for training the third network 213. In addition, the information processing system 200 may have three evaluation value calculation units corresponding to these.

[0198] Figure 13 is a flowchart showing the first phase of an example of the operation of the information processing system 200 in this embodiment.

[0199] First, the network discovery unit 201 acquires the first network 211 and sets the first network 211 as the initial value for the search of the second network 212 (S601).

[0200] The network discovery unit 201 may acquire the first network 211 by generating the first network 211. Specifically, the network discovery unit 201 may determine the planned size of the first network 211 and the configuration of the first network 211 based on design requirements, etc. Then, the network discovery unit 201 may generate the first network 211 based on the determined configuration.

[0201] Next, the network search unit 201 acquires setting information indicating the optimization settings and difficulty information indicating the difficulty of inference (S602). Then, the network search unit 201 determines the initial values ​​for the search of the second network 212 based on the optimization settings and the difficulty of inference (S603). Next, the learning processing unit 206 trains the second network 212 (S604). These processes are the same as the processes in the first operation example in this embodiment.

[0202] If the performance of the second network 212 does not meet the first requirement (No in S605), the network discovery unit 201 changes the number of layers (S606). Then, the process is repeated starting from training the second network 212 (S604).

[0203] Furthermore, if the performance of the second network 212 satisfies the first requirement but does not satisfy the second requirement (Yes in S605 and No in S607), the network discovery unit 201 changes the number of nodes in each layer or a specific layer (S608). Then, the process is repeated starting from training the second network 212 (S604).

[0204] Then, training (S604), changing the number of layers (S606), and changing the number of nodes (S608) of the second network 212 are repeated until the performance of the second network 212 satisfies the first and second requirements. This results in a second network 212 whose performance satisfies the first and second requirements.

[0205] These processes (S605, S606, S607, and S608) correspond to the processes (S307, S308, S309, and S310) of the second operation example of Embodiment 1. However, while the processes of the second operation example of Embodiment 1 determine the performance of the third network 213, the processes of the first phase of this operation example determine the performance of the second network 212. If the performance of the second network 212 satisfies the first and second requirements (Yes in S605 and Yes in S607), the processes of the first phase are terminated.

[0206] Figure 14 is a flowchart showing the second and third phases of an example of the operation of the information processing system 200 in this embodiment.

[0207] In the second phase, the learning processing unit 204 distills the second network 212 to train the first network 211 (S609).

[0208] Specifically, the evaluation value calculation unit 203 acquires the inference results of the first network 211 and the inference results of the second network 212, and calculates an evaluation value that indicates the difference between the inference results of the first network 211 and the inference results of the second network 212. The learning processing unit 204 refers to the calculated evaluation value and trains the first network 211 so that the difference between the inference results of the first network 211 and the inference results of the second network 212 becomes smaller.

[0209] Subsequently, if the performance of the first network 211 does not meet the third requirement (No in S610), the learning processing unit 204 changes the number of nodes in the first network 211 (S611). Specifically, if the inference accuracy of the first network 211 is not above the standard, the learning processing unit 204 increases the number of nodes in the first network 211. This is expected to improve the inference accuracy of the first network 211. Then, the process is repeated, starting from the acquisition of the optimization settings information and the difficulty level information for inference in the first phase (S602).

[0210] If the performance of the first network 211 satisfies the third requirement (Yes in S610), the lightweight unit 202 lightweights the second network 212 based on the lightweight settings to generate the third network 213 (S612).

[0211] Next, the learning processing unit 204 distills the second network 212 and trains the third network 213 (S613). This step may be omitted.

[0212] Specifically, the evaluation value calculation unit 203 acquires the inference results of the second network 212 and the third network 213, and calculates an evaluation value that shows the difference between the inference results of the second network 212 and the third network 213. The learning processing unit 204 refers to the calculated evaluation value and trains the third network 213 so that the difference between the inference results of the second network 212 and the third network 213 becomes smaller.

[0213] Next, the learning processing unit 204 distills the first network 211 and trains the third network 213 (S614).

[0214] Specifically, the evaluation value calculation unit 203 acquires the inference results of the first network 211 and the third network 213, and calculates an evaluation value that shows the difference between the inference results of the first network 211 and the third network 213. The learning processing unit 204 refers to the calculated evaluation value and trains the third network 213 so that the difference between the inference results of the first network 211 and the third network 213 becomes smaller.

[0215] Subsequently, if the performance of the third network 213 does not meet the fourth requirement (No in S615), the process is repeated from the beginning of the first phase (S601). If the performance of the third network 213 meets the fourth requirement (Yes in S615), the process ends. This results in a third network 213 whose performance meets the fourth requirement.

[0216] Each of the above performance characteristics may be inference accuracy, inference speed, or a combination thereof, as in Embodiments 1 and 2. Furthermore, each requirement is a requirement for these performance characteristics.

[0217] As described above, the first network 211 and the third network 213 in this embodiment are based on the same parent. In particular, the information processing system 200 in this embodiment trains the first network 211 by distilling the second network 212. Therefore, an improvement in the reference match rate is expected compared to Embodiments 1 and 2.

[0218] The information processing system 200 may include some of the components shown in this embodiment, or it may perform some of the processes shown in this embodiment. Furthermore, at least some of the components and processes shown in this embodiment may be combined with at least some of the components and processes shown in other embodiments.

[0219] (Basic implementation example and basic operation example) Basic implementation examples and basic operation examples for Embodiments 1, 2, and 3 are shown below.

[0220] Figure 15 is a block diagram showing basic implementation examples of the information processing system 200 in the above-described embodiments. The information processing system 200 includes, for example, at least one processor 301 and at least one memory 302.

[0221] The processor 301 is an information processing circuit that performs information processing. The processor 301 may also perform the roles of a network search unit 201, a lightweight unit 202, an evaluation value calculation unit 203, a learning processing unit 204, a difficulty calculation unit 205, and a learning processing unit 206. The processor 301 may perform these roles by reading a program from memory 302 and executing the program. The processor 301 may also control the inference processing of the first network 211, the second network 212, and the third network 213.

[0222] Memory 302 is a storage device for storing information, and can also be described as a recording medium. Memory 302 may store information such as the lightweight settings 231, the correct answer data 232, the dataset 233, and the task difficulty 234. Memory 302 may also store a program for the processor 301 to perform information processing. Furthermore, memory 302 may store information for the first network 211, the second network 212, and the third network 213.

[0223] The information processing system 200 is, for example, a computer. The information processing system 200 may consist of one information processing device or may be composed of multiple information processing devices.

[0224] Figure 16 is a flowchart illustrating basic operation examples of the information processing system 200 in multiple embodiments. For example, at least one processor 301 shown in Figure 15 performs the operation shown in Figure 16 using at least one memory 302. In the following description, the first inference model, the second inference model, and the third inference model correspond to the first network 211, the second network 212, and the third network 213, respectively.

[0225] First, processor 301 obtains a reference first inference model (S701). Next, processor 301 computes a second inference model with a larger model size than the first inference model based on the first inference model (S702). Then, processor 301 quantizes the computed second inference model to generate a third inference model (S703).

[0226] Next, processor 301 trains the third inference model using machine learning (S704). Next, processor 301 determines whether the performance of the trained third inference model meets the conditions (S705). Next, if the performance meets the conditions, processor 301 outputs the trained third inference model (S706).

[0227] This results in the quantization of the second inference model, which has a larger model size than the first inference model. It is assumed that the performance of the second inference model, which has a larger model size, will not degrade significantly even after quantization. In other words, it is assumed that the loss caused by quantization will be relatively small in the third inference model generated by the quantization of the second inference model, which has a larger model size than the first inference model. Therefore, it becomes possible to find an inference model in which the loss caused by quantization is suppressed.

[0228] Furthermore, for example, the processor 301 may acquire configuration information indicating the quantization settings for the second inference model. The processor 301 may then set initial values ​​for the calculation of the second inference model based on the configuration information and the first inference model.

[0229] This initiates the calculation of the second inference model based on the quantization settings and the first inference model. Consequently, it becomes possible to quickly find the third inference model corresponding to the quantization and the first inference model.

[0230] Furthermore, for example, the processor 301 may acquire difficulty information indicating the inference difficulty of at least one of the first inference model, the second inference model, and the third inference model. The processor 301 may then set initial values ​​for the calculation of the second inference model based on the difficulty information and the first inference model.

[0231] This initiates the calculation of the second inference model based on the inference difficulty information and the first inference model. Therefore, it becomes possible to quickly find a third inference model that corresponds to the inference difficulty and the first inference model.

[0232] Furthermore, for example, the calculation of the second inference model may be a search for the second inference model using a loss function. The loss function may be a function whose output value decreases as the difference between the inference result of the first inference model and the inference result of the third inference model decreases, and whose output value decreases as the model size of the second inference model becomes relatively larger than that of the first inference model. The search for the second inference model may then be performed in such a way that the output value of the loss function decreases.

[0233] This makes it possible to find an inference model that suppresses the losses caused by quantization, based on the loss function.

[0234] Furthermore, for example, the processor 301 may acquire configuration information indicating the quantization settings for the second inference model. The processor 301 may then modify the loss function based on the configuration information.

[0235] This makes it possible to find an inference model that suppresses the losses caused by quantization, based on a loss function that corresponds to the quantization settings.

[0236] Furthermore, for example, the loss function may be modified such that the output value of the loss function increases as the degree of quantization increases in the settings indicated by the configuration information. The search for the second inference model may then be performed so that the output value of the loss function is below a threshold.

[0237] As a result, the output value of the loss function increases with increasing quantization, but the second inference model is searched for such that the output value of the loss function falls below a threshold. In other words, even when the loss is large due to a high degree of quantization, the second inference model is searched for such that certain conditions for suppressing the loss are met. Therefore, it becomes possible to find an inference model in which the loss is suppressed at a certain level.

[0238] Furthermore, for example, the processor 301 may acquire difficulty information indicating the inference difficulty of at least one of the first inference model, the second inference model, and the third inference model. The processor 301 may then modify the loss function based on the difficulty information.

[0239] This makes it possible to find inference models that suppress the losses caused by quantization, based on a loss function that corresponds to the inference difficulty.

[0240] Furthermore, the loss function may be modified, for example, so that the output value of the loss function increases as the inference difficulty, indicated by the difficulty information, increases. The search for the second inference model may then be performed so that the output value of the loss function is below a threshold.

[0241] As a result, the output value of the loss function increases with higher inference difficulty, but the second inference model is searched for so that the output value of the loss function falls below a threshold. In other words, even when the loss is large due to high inference difficulty, the second inference model is searched for so that certain conditions for suppressing the loss are met. Therefore, it becomes possible to find an inference model in which the loss is suppressed at a certain level.

[0242] Furthermore, for example, if the performance does not meet the requirements, the processor 301 may change the quantization settings for the second inference model.

[0243] This may allow the quantization settings to be modified to satisfy the performance requirements. This then makes it possible to find an inference model that meets those performance requirements.

[0244] Furthermore, the conditions may include, for example, the accuracy or precision of the inference of the third inference model on the inference result of the first inference model or on the reference data. The processor 301 may reduce the degree of quantization if the accuracy or precision of the inference of the third inference model is below a threshold.

[0245] This means that if the accuracy or precision of the third inference model's inference is below a threshold, the quantization applied to the second inference model is narrowed to improve the accuracy or precision of the third inference model's inference. This makes it possible to find an inference model that satisfies the conditions for accuracy or precision of the inference.

[0246] Furthermore, the conditions may include, for example, the inference processing speed of the third inference model. The processor 301 may increase the degree of quantization if the inference processing speed is below a threshold.

[0247] This means that if the inference processing speed of the third inference model is below a threshold, the degree of quantization for the second inference model is increased so that the inference processing speed of the third inference model becomes faster. This makes it possible to find an inference model that satisfies the inference processing speed condition.

[0248] Alternatively, for example, the processor 301 may input data into the first inference model and obtain the inference result of the first inference model. The processor 301 may also input data into the second inference model and obtain the inference result of the second inference model. The processor 301 may then train the first inference model based on the difference between the inference result of the first inference model and the inference result of the second inference model.

[0249] This allows the first and third inference models to be constructed based on the same second inference model. Therefore, it becomes possible to minimize the difference between the inference results of the first inference model and the inference results of the third inference model.

[0250] Furthermore, for example, the processor 301 may also perform the processing shown in any of the above-described embodiments.

[0251] Figure 17 is a block diagram showing another basic implementation example of the information processing system 200 in the above-described embodiments. The information processing system 200 comprises, for example, a calculation unit 401, a generation unit 402, a training unit 403, a determination unit 404, and an output unit 405. Furthermore, the information processing system 200 may also include at least one of an initial value setting unit 406, a loss function changing unit 407, and a quantization setting changing unit 408.

[0252] Each of these components is an information processing circuit that performs information processing. These components may be implemented by the processor 301 shown in Figure 15.

[0253] The calculation unit 401 is a component corresponding to the network search unit 201, etc. The calculation unit 401 performs processing related to the calculation of the second inference model. Specifically, the calculation unit 401 performs the acquisition of the first inference model (S701) and the calculation of the second inference model (S702) as shown in Figure 16.

[0254] The generation unit 402 is a component corresponding to the lightweight unit 202, etc. The generation unit 402 performs processing related to the quantization of the second inference model and the generation of the third inference model. Specifically, the generation unit 402 performs the generation of the third inference model (S703) shown in Figure 16.

[0255] The training unit 403 is a component corresponding to the learning processing unit 204, etc. The training unit 403 performs processing related to the training of the third inference model. Specifically, the training unit 403 performs the training of the third inference model (S704) as shown in Figure 16. The training unit 403 may also perform processing related to the training of the first inference model or the training of the second inference model. The information processing system 200 may have a separate training unit 403 for each of the first inference model, the second inference model, and the third inference model.

[0256] The determination unit 404 is a component corresponding to the evaluation value calculation unit 203, etc. The determination unit 404 performs processing related to determining whether the performance of the third inference model satisfies the conditions. Specifically, the determination unit 404 performs the determination (S705) shown in Figure 16.

[0257] The output unit 405 is a component that corresponds to the evaluation value calculation unit 203, etc. The output unit 405 performs processing related to the output of the third inference model. Specifically, the output unit 405 performs the output of the third inference model (S706) in Figure 16.

[0258] The initial value setting unit 406 is a component corresponding to the network search unit 201, etc. The initial value setting unit 406 performs processing related to setting initial values ​​in the calculation of the second inference model. The loss function modification unit 407 is a component corresponding to the evaluation value calculation unit 203, etc. The loss function modification unit 407 performs processing related to modifying the loss function. The quantization setting modification unit 408 is a component corresponding to the learning processing unit 204, etc. The quantization setting modification unit 408 performs processing related to modifying the quantization settings.

[0259] Conversely, the network search unit 201 corresponds to examples such as the calculation unit 401 and the initial value setting unit 406. The optimization unit 202 corresponds to examples such as the generation unit 402. The evaluation value calculation unit 203 corresponds to examples such as the determination unit 404, the output unit 405, and the loss function modification unit 407. The learning processing unit 204 and the learning processing unit 206 correspond to examples such as the training unit 403 and the quantization setting modification unit 408.

[0260] Note that the configuration shown in Figure 17 is an example, and the configuration of the information processing system 200 is not limited to the example shown in Figure 17. For example, the configuration shown in Figure 17 may be combined with the configuration shown in any of the above-described embodiments.

[0261] Although the embodiments of the information processing system have been described above based on the embodiments, the embodiments of the information processing system are not limited to the embodiments. Modifications that a person skilled in the art can conceive of may be made to the embodiments, and multiple components in the embodiments may be combined arbitrarily. For example, a process performed by a specific component in the embodiments may be performed by another component instead of that specific component. In addition, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0262] Furthermore, the inference model is, for example, a mathematical model for performing inference processing, and may be a machine learning model, a neural network model, or a deep learning model.

[0263] Furthermore, the information processing method, which includes the steps performed by each component of the information processing system, may be executed by any device or system. In other words, this information processing method may be executed by the information processing system or by other devices or systems.

[0264] For example, the above information processing method may be executed by a computer equipped with a processor, memory, and input / output circuits. In this case, the information processing method may be executed by the computer executing a program that causes the computer to execute the information processing method. Alternatively, the program may be recorded on a non-temporary computer-readable recording medium.

[0265] For example, the above program causes the computer to obtain a first inference model as a reference, calculate a second inference model with a larger model size than the first inference model based on the first inference model, generate a third inference model by quantizing the calculated second inference model, train the third inference model using machine learning, determine whether the performance of the trained third inference model satisfies the conditions, and if the performance satisfies the conditions, execute an information processing method that outputs the trained third inference model.

[0266] Furthermore, the multiple components of the information processing system may consist of dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may consist of memory in which the programs are stored, and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0267] Furthermore, dedicated hardware may consist of memory and a dedicated processor, etc. For example, a dedicated processor may refer to memory and execute the above-described information processing method.

[0268] Furthermore, each component of the information processing system may be an electrical circuit. These electrical circuits may form a single electrical circuit as a whole, or they may be separate electrical circuits. Also, these electrical circuits may correspond to dedicated hardware, or they may correspond to general-purpose hardware that executes the above-mentioned programs, etc.

Industrial Applicability

[0269] The present disclosure can be used in an information processing system for finding an inference model in which losses caused by quantization are suppressed, and is applicable to a machine learning model construction system, a neural network model construction system, a deep learning model construction system, and the like.

Explanation of Signs

[0270] 100, 200 Information processing system 101, 201 Network search unit 103, 203 Evaluation value calculation unit 104, 204, 206 Learning processing unit 111, 211 First network 112, 212 Second network 202 Quantization unit 205 Difficulty calculation unit 213 Third network 231 Quantization setting 232 Correct data 233 Dataset 234 Task difficulty 301 Processor 302 Memory 401 Calculation unit 402 Generation unit 403 Training unit 404 Judgment unit 405 Output unit 406 Initial value setting unit 407 Loss function change unit 408 Quantization setting change unit

Claims

1. A method of information processing that is performed by a computer, The first network is trained by distilling a second network that is larger in size than the first network. The third network is generated by reducing the size of the second network. The first network is distilled and the third network is trained, or the second network is distilled and the third network is trained. Information processing methods.

2. It is determined whether the performance of the trained third network meets the conditions. If the performance satisfies the above conditions, the trained third network is output. The information processing method according to claim 1.

3. If the performance does not meet the above conditions, the optimization settings for the second network are changed. The information processing method according to claim 2.

4. The above conditions include the accuracy or precision of the inference results in the first network or the inference in the third network on the reference data, In the aforementioned setting change, if the accuracy or precision of the inference in the third network is below a threshold, the degree of optimization is reduced. The information processing method according to claim 3.

5. The above conditions include the speed of the inference processing in the third network, With the above setting change, if the speed of the inference processing is below a threshold, the degree of optimization is increased. The information processing method according to claim 3.

6. moreover, Data is input to the first network and the inference result from the first network is obtained. The data is input to the second network and the inference result from the second network is obtained. The first network is trained based on the difference between the inference result of the first network and the inference result of the second network. The information processing method according to claim 3.

7. A training unit that distills a second network, which is larger in size than the first network, to train the first network, A generation unit that generates a third network by reducing the weight of the second network, Equipped with, The training unit distills the first network to train the third network, or distills the second network to train the third network. Information processing system.

Images