Neural network derivation method

The neural network derivation method stabilizes weight parameters and improves robustness against variations and adversarial attacks by incorporating a regularization term based on a discriminator's accuracy assessment, ensuring consistent inference accuracy.

JP7593066B2Active Publication Date: 2024-12-03SOCIONEXT INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2020190347
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-02-06
Filing Date
2020-11-16
Publication Date
2024-12-03
Estimated Expiration
2040-11-16

AI Technical Summary

Technical Problem

Existing neural networks are susceptible to variations in parameters and input data, leading to incorrect inference values, particularly when weight parameters are quantized for hardware implementation, and are vulnerable to adversarial attacks.

Method used

A neural network derivation method that includes a first learning step using a loss function and a second learning step with a regularization term, determined by a discriminator that identifies accuracy changes, to stabilize weight parameters and improve robustness against variations.

Benefits of technology

The method generates a neural network that is robust against variations in parameters and input data, maintaining accuracy even after quantization and enhancing resistance to adversarial attacks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593066000006
    Figure 0007593066000006
  • Figure 0007593066000007
    Figure 0007593066000007
  • Figure 0007593066000008
    Figure 0007593066000008
Patent Text Reader

Abstract

To derive a neural network having robustness against variations in a weight parameter.SOLUTION: A neural network derivation method includes: a first learning step S10 for learning a first neural network having a weight parameter w with a first loss function being an index; and after the first learning step S10, a second learning step S20 for learning the first neural network with a second loss function obtained by adding a regularization term to the first loss function being an index. The regularization term is determined on the basis of time-series change in a degree of similarity between a first inference value x of the first neural network and a second inference value G (z) of the second neural network after the second neural network is derived having a weight parameter wq to which variations are added to the weight parameter w on the basis of the first neural network.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a method for deriving a neural network. [Background technology]

[0002] In order to identify or classify input data, an inference model configured with a neural network is used. As an example of a method for generating this inference model, Patent Document 1 describes a method for training a neural network with a training data set to generate an inference model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-525734 Summary of the Invention [Problem to be solved by the invention]

[0004] When this inference model is actually used, fluctuations in the parameters of the neural network or the input data input to the neural network may result in an incorrect inference value being output.

[0005] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide a neural network derivation method for deriving a neural network that is robust against fluctuations in the neural network parameters or input data. [Means for solving the problem]

[0006] In order to achieve the above object, a neural network derivation method according to one embodiment of the present disclosure includes: 1. A computer-implemented method comprising:The method includes a first learning step of learning a first neural network having a first parameter using a first loss function as an index, and a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step, wherein the regularization term is determined based on a correlation between internal features or inference values ​​between the first neural network and the second neural network after a second neural network having second parameters obtained by adding a variation to the first parameters is derived based on the first neural network.

[0007] In order to achieve the above object, a neural network derivation method according to one embodiment of the present disclosure includes: 1. A computer-implemented method comprising: The method includes a first learning step of learning a first neural network having a first weight parameter using a first loss function as an index, and a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step, wherein the regularization term is determined based on a relationship between the first neural network and a second neural network having a second weight parameter obtained by adding a variation to the first weight parameter based on the first neural network.

[0008] In order to achieve the above object, a neural network derivation method according to one embodiment of the present disclosure includes: 1. A computer-implemented method comprising: A first learning step of learning a first neural network having a first parameter using a first loss function as an index, and after the first learning step, learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index. and a second learning step of determining a regularization term based on the first neural network and a second neural network based on the first neural network, the second neural network being further configured such that an input to at least one layer of the second neural network is obtained by adding variation to a feature quantity that is an output of a previous layer.

[0009] In order to achieve the above object, a neural network derivation method according to one embodiment of the present disclosure includes: 1. A computer-implemented method comprising: The method includes a first learning step of learning a first neural network to which first input data is input using a first loss function as an index, and a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step, wherein the regularization term is determined based on a time series change in similarity between a fifth inference value of the first neural network and a sixth inference value of the second neural network after a second neural network to which second input data obtained by adding a variation to the first input data is input based on the first neural network.

[0010] These general or specific aspects may be realized by an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0011] The present disclosure makes it possible to derive neural networks that are robust against variations in the neural network parameters or input data. [Brief description of the drawings]

[0012] [Figure 1]FIG. 1 is a diagram showing a schematic diagram of the change in the accuracy of inference values ​​when the weight parameters of a neural network are changed. [Diagram 2] FIG. 2 is a schematic diagram showing the function of a classifier for identifying the accuracy of an inferred value. [Diagram 3] FIG. 3 is a flow diagram showing an overview of a neural network derivation method. [Figure 4] FIG. 4 is a diagram showing a derivation model for deriving a neural network in the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing a neural network deriving method according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing a neural network derivation method executed subsequent to FIG. [Figure 7] FIG. 7 is a schematic diagram showing a discrimination learning model included in the derivation model of FIG. [Figure 8] FIG. 8 is a diagram illustrating an example of a hardware configuration of a computer that realizes, by software, the functions of the device that realizes the neural network deriving method according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing a derivation model for deriving a neural network in the second embodiment. [Figure 10] FIG. 10 is a flowchart showing a neural network deriving method according to the second embodiment. [Figure 11] FIG. 11 is a flowchart showing a neural network derivation method executed subsequent to FIG. [Figure 12] FIG. 12 is a schematic diagram showing a discriminative learning model included in the derivation model of FIG. [Figure 13] FIG. 13 is a diagram showing a derivation model for deriving a neural network in the third embodiment. [Figure 14] FIG. 14 is a flowchart showing a neural network deriving method according to the third embodiment. [Figure 15]FIG. 15 is a flowchart showing a neural network derivation method executed subsequent to FIG. [Figure 16] FIG. 16 is a schematic diagram showing a discrimination learning model included in the derivation model of FIG. [Figure 17] FIG. 17 is a diagram showing a derivation model for deriving a neural network in the fourth embodiment. [Figure 18] FIG. 18 is a flowchart showing a neural network deriving method according to the fourth embodiment. [Figure 19] FIG. 19 is a diagram illustrating a definition of feature amount similarity in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (Background to this disclosure) Inference models trained through machine learning are increasingly being implemented on LSI (large-scale integrated circuits). Generally, weight parameters expressed as real values ​​are used during training, but during implementation, the weight parameters used during training are quantized (discretized) to fixed-point representation, etc. Using quantized weight parameters can reduce hardware costs during implementation, but on the other hand, quantization errors can degrade the accuracy of the inference values ​​of the inference model.

[0014] In addition, a method has been announced in which intentional noise is added to the input data of a trained inference model to cause incorrect inference. For example, in "DEFENSIVE QUANTIZATION: WHEN EFFICIENCY MEETS ROBUSTNESS, ICLR 2019," it was announced that quantization significantly reduces resistance to adversarial attacks (attacks using adversarial samples).

[0015] Therefore, a robust inference model is required in which the accuracy of the inference value is not easily affected by fluctuations in the weight parameters and input data. In the following, for ease of understanding, the explanation will be given by focusing on the weight parameters out of the weight parameters and the input data.

[0016] Figure 1 is a diagram that shows a schematic diagram of the change in the accuracy of the inference value when the weight parameters of a neural network are changed. The right side of Figure 1 shows an example in which the accuracy of the inference value changes significantly as the weight parameters change, while the left side of Figure 1 shows an example in which the accuracy of the inference value does not change very much even if the weight parameters change. From the viewpoint of the robustness of the inference model, it is desirable that the accuracy of the inference value is less likely to deteriorate even if the weight parameters change, as shown on the left side of Figure 1.

[0017] For example, when a neural network is trained using only the loss function as an index, an inference model with low robustness may be generated as shown on the right side of FIG. 1. Therefore, in the present disclosure, when generating an inference model, training is performed using the loss function plus a regularization term as an index. This regularization term is intended to prevent the weight parameters used in the neural network from becoming weight parameters that tend to change the accuracy of the inference value. For example, in the present disclosure, when training a neural network, the weight parameters are updated to reduce the value of "loss function + regularization term" to generate an inference model. This makes it possible to suppress significant deterioration in the accuracy of the inference value even when the weight parameters are quantized during implementation.

[0018] The regularization term of the above "loss function + regularization term" is determined to be large when the accuracy of the inferred value deteriorates, and to be small when the accuracy of the inferred value improves. Here, we consider how to distinguish whether the accuracy of the inferred value deteriorates or improves.

[0019] Fig. 2 is a schematic diagram showing the function of a classifier that distinguishes the accuracy of an inferred value. Fig. 2 shows how weight parameters that deteriorate the accuracy of an inferred value when quantized (the lower region of Fig. 2) and weight parameters that are less likely to deteriorate the accuracy of an inferred value even when quantized (the upper region of Fig. 2) are classified according to the function of the classifier. If such a classifier can be generated, it will be possible to distinguish whether an unknown weight parameter deteriorates the accuracy of an inferred value, and based on the result of this distinction, it will be possible to determine whether to increase or decrease the regularization term.

[0020] FIG. 3 is a flow showing an overview of a neural network derivation method. In the present disclosure, first, a neural network is trained using a training data set. Next, a classifier is trained using the weight parameters of this neural network to generate a classifier. Next, a regularization term is derived from this classifier, and the neural network is trained again using the "loss function + regularization term" reflecting the regularization term as an index. Then, an inference model is generated by repeating the training of the neural network and the training of the classifier. According to this neural network derivation method, it is possible to derive a neural network that is robust against fluctuations in the weight parameters.

[0021] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, standards, components, the arrangement and connection forms of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept of the present disclosure are described as optional components. In addition, each figure is not necessarily a strict illustration. In each figure, substantially the same configuration is given the same symbol, and duplicated explanations may be omitted or simplified.

[0022] (Embodiment 1) [1-1. Derivation model for deriving neural networks] First, a derivation model for deriving a neural network that is robust against parameter variations will be described.

[0023] 4 is a diagram showing a derived model 10 for deriving a neural network in the embodiment 1. As shown in FIG 4, the derived model 10 is composed of a pre-quantization model 20, a post-quantization model 30, and a discriminative learning model 40.

[0024] For example, the pre-quantization model 20 is a model on which machine learning is performed under certain conditions, and the quantized model 30 is ultimately obtained by quantizing the pre-quantization model 20, and serves as an inference model implemented in an LSI or the like. The discriminative learning model 40 is a model for training a discriminator 41 that discriminates the accuracy of an inference value. Each of the pre-quantization model 20, the quantized model 30, and the discriminator 41 is configured by a neural network. Furthermore, each of the pre-quantization model 20, the quantized model 30, and the discriminative learning model 40 has a learning state and an inference state, and in the inference state, the weight parameters of the model are fixed.

[0025] These models have a multi-layer structure, and are composed of an input layer, an intermediate layer, an output layer, etc. Each layer has a plurality of nodes (not shown) that correspond to neurons. The strength of the connection between neurons is expressed by a weight parameter. Although a neural network has a plurality of weight parameters, in order to facilitate understanding, the following explanation will be given by taking one of the plurality of weight parameters as an example.

[0026] The pre-quantization model 20 is configured by a first neural network having a weight parameter (first parameter) w. The weight parameter w is expressed by a first numerical expression such as a real number consisting of a float (floating-point precision) value. Input data z is input to the pre-quantization model 20. The input data z is learning data and has various input patterns. The pre-quantization model 20 to which the input data z is input outputs an inferred value x as an output value. Machine learning is performed on the pre-quantization model 20 based on a predetermined learning data set including the input data z. In addition, when the classifier 41 is learning, it operates with a weight parameter w+Δw obtained by adding Δw to the weight parameter w.

[0027] The quantized model 30 has a weight parameter (second parameter) w q The second neural network is composed of a weight parameter w q is obtained by converting the weight parameter w of the pre-quantization model 20 into a second numerical representation different from the first numerical representation. The second numerical representation is, for example, a numerical representation with fixed-point precision such as an integer. Specifically, the weight parameter w q is obtained by quantizing the weight parameter w+Δw, which is the weight parameter w plus Δw. Therefore, the weight parameter w q is a value obtained by adding a quantization error to a value obtained by adding a variation to the weight parameter w. Input data z is input to the quantized model 30. The quantized model 30 to which the input data z is input outputs an inferred value G(z) as an output value.

[0028] The discrimination learning model 40 is a model for learning a discriminator 41 that discriminates the accuracy of an inferred value, and includes the discriminator 41 and the like.

[0029] The discriminator 41 is provided with a weighting parameter w+Δw before quantization and a weighting parameter w after quantization. q The classifier 41 outputs an inference value D(w+Δw) corresponding to the weight parameter w+Δw, and q Correspondingly, the inferred value D(wq ) to output.

[0030] The discriminative learning model 40 receives the inferred value (first inferred value) x of the pre-quantization model 20 and the inferred value (second inferred value) G(z) of the post-quantization model 30. The discriminative learning model 40 compares the input inferred value x and inferred value G(z) with the above inferred values ​​D(w+Δw) and D(w q ) and performs backpropagation to train the classifier 41. The discriminative training model 40 then derives the above-mentioned regularization term using the trained classifier 41. The regularization term derived by the discriminative training model 40 is used when retraining the first neural network of the pre-quantization model 20. The discriminative training model 40 will be described in detail later.

[0031] [1-2. Neural network derivation method] Next, a method for deriving a neural network using the above-mentioned derivation model 10 will be described.

[0032] FIG. 5 is a flow diagram showing a neural network deriving method according to the present embodiment.

[0033] The neural network derivation method includes a first learning step S10, a regularization term learning step (first regularization term learning step) S11, and a second learning step S20.

[0034] The first learning step S10 is a step of learning the first neural network of the pre-quantization model 20. The first learning step S10 is executed within the dashed line in FIG. 5(a). In this step, the first neural network is learned using a predetermined learning data set with the first loss function as an index. In the first learning step S10, a weight parameter w in the first neural network is first obtained.

[0035] The regularization term learning step S11 is a step of performing learning to derive the regularization term described above. The regularization term learning step S11 includes a step S12 of generating a quantized model 30, a step S13 of learning a classifier 41, and a step S14 of deriving a regularization term from the classifier 41.

[0036] The step S12 of generating the quantized model 30 includes quantizing the weight parameters w based on the first neural network. q This step S12 is executed within the dashed line in FIG. 5(b). q is a quantized value obtained by further varying the weighting parameter w+Δw, and can be obtained, for example, by quantizing the weighting parameter w+Δw.

[0037] Step S13 for training the classifier 41 is a step executed by the systems within the dashed lines in each of (c) and (d) of Figure 5. Here, the system within the dashed line shown in (c) of Figure 5 is called system 41a on the pre-quantization model side, and the system within the dashed line shown in (d) of Figure 5 is called system 41b on the post-quantization model side.

[0038] As shown in FIG. 5C, in the system 41a on the pre-quantization model side, a weight parameter w+Δw is input to the discriminator 41. Δw also increases the number of samples when learning, and is given as a random number with a maximum value of half the quantization step, for example. The discriminator 41 to which the weight parameter w+Δw is input outputs an inferred value D(w+Δw). The discriminant learning model 40 receives the inferred values ​​x and G(z), which are the outputs of the pre-quantization model 20 and the quantized model 30. The discriminant learning model 40 trains the discriminator 41 so as to improve the accuracy of the inferred value D(w+Δw) based on the change in the similarity between the inferred value x and the inferred value G(z) over time.

[0039] The similarity here indicates the degree of similarity between two inferred values ​​x and G(z). High similarity means high accuracy of the inferred value. For example, the cosine similarity shown in (Equation 1) is used as the similarity. In (Equation 1), the inferred values ​​x and G(z) are vectorized (the Tensor shape is made one-dimensional) and expressed as V x , V Gz It states that:

[0040]

number

[0041] In this way, in the pre-quantization model side system 41a, the discriminator 41 is trained with the pre-quantization model 20 fixed.

[0042] As shown in FIG. 5(d), in the system 41b on the quantized model side, the weight parameter w q (w q =quantizer(w+Δw) is input to the discriminator 41. q The discriminator 41 receives the input of the inference value D(w q ) is output. The discriminative learning model 40 calculates the inferred value D(w) based on the time-series change in the similarity between the input inferred value x and the inferred value G(z). q ) is improved. In this way, in the system 41b on the post-quantization model side, the classifier 41 is trained with the post-quantization model 30 fixed.

[0043] The weights in the system 41a on the pre-quantization model side and the system 41b on the quantization model side are common (the weight parameters in the system 41a on the pre-quantization model side are quantized and become the weight parameters of the system 41b on the quantization model side), and the learning of the classifier 41 by these systems 41a and 41b is performed simultaneously.

[0044] Step S14 of deriving a regularization term is a step of deriving a regularization term using a discriminator 41. The regularization term has a negative correlation with the magnitude of the similarity between the inferred value x and the inferred value G(z), and is determined so as to be small as the similarity increases and large as the similarity decreases, for example, in a time series. The regularization term derived by the discriminator 41 is reflected in the learning of the first neural network of the pre-quantization model 20.

[0045] The second learning step S20 is a step of learning the first neural network using a second loss function (second loss function = first loss function + regularization term) obtained by adding a regularization term to the first loss function as an index. The second learning step S20 is also performed within the dashed line in (a) of FIG. 5, and a predetermined learning data set is used as in the first learning step S10. Learning in the second learning step S20 is performed with the discriminative learning model 40 fixed. The weight parameter w of the first neural network is updated by the second learning step S20.

[0046] Fig. 6 is a flowchart showing a neural network deriving method executed following Fig. 5. In this neural network deriving method, a regularization term learning step S11A similar to the regularization term learning step S11 is executed after the second learning step S20. The regularization term learning step S11A includes a step S12 of generating the above-mentioned quantized model 30, a step S13 of learning the classifier 41, and a step S14 of deriving a regularization term from the classifier 41.

[0047] In the neural network derivation method of the present embodiment, the second learning step S20 and the regularization term learning step S11A are repeated to generate a robust first neural network. In addition, the weight parameter w of the first neural network generated by the above repetition is quantized to obtain the weight parameter w q A second neural network having robustness is generated.

[0048] These learning processes are terminated when the learning level of the classifier 41 reaches or exceeds a predetermined level. Furthermore, these learning processes may be terminated when the similarity between the inferred value x and the inferred value G(z) reaches or exceeds a predetermined threshold value.

[0049] In the above, an example of iteration in which the second learning step S20 and the regularization term learning step S11A are alternately repeated (S20 → S11A → S20 → S11A → ··· → S20) has been shown, but the example of iteration is not limited to this. For example, the second learning step S20 and the regularization term learning step S11A may be repeated in the order of (S20 → S11A → S11A) → (S20 → S11A → S11A) → ··· → S20.

[0050] [1-3. Operation of discriminative learning model] Next, the operation of the discrimination learning model 40 for training the discriminator 41 will be described.

[0051] Fig. 7 is a schematic diagram showing a discriminative learning model 40 included in the derived model 10. Note that Fig. 7 also shows a pre-quantization model 20 and a post-quantization model 30.

[0052] The classifier 41 is configured by a convolutional neural network (CNN) and a fully connected neural network (FC). The classifier 41 is provided with weight parameters w + Δw and w q are input, and the inferred values ​​D(w+Δw) and D(w q ) is output.

[0053] Moreover, the discriminative learning model 40 receives the inferred value x from the pre-quantization model 20 and the inferred value G(z) from the quantized model 30. In the discriminative learning model 40, a classifier 41 is trained using the similarity and expected value calculated from the inferred value x and the inferred value G(z).

[0054] Here, we will explain the expected value (first expected value) used when training the classifier 41. The expected value is a label when training, and is determined based on the inference value x and G(z) input to the discriminative training model 40, as shown in (Equation 2).

[0055] Expected value = {similarity of inferred value (x or G(z)) for x, improvement in similarity of inferred value (x or G(z)) for x compared to the previous evaluation} (Equation 2)

[0056] Table 1 shows the expected values ​​for the system 41a on the pre-quantization model side of the classifier and the system 41b on the quantization model side of the classifier. The expected values ​​are the inferred values ​​D(w+Δw) and D(w q ) is represented by two values, 0 and 1, so that it can be trained as a two-class classifier, which is a relatively easy discriminator to train when rating it good or bad.

[0057] [Table 1]

[0058] As shown in Table 1, the similarity for x in system 41a on the pre-quantization model side is expected to be 1 because the inferred value x is the same as x. The similarity for x in system 41b on the quantization model side is expected to be 0 because the inferred value G(z) is different from x. The improvement in the similarity of the inferred values ​​x and G(z) of system 41a on the pre-quantization model side and system 41b on the quantization model side is expected to be 1 if the similarity calculated this time has improved compared to the similarity calculated last time, and is expected to be 0 if there has been no improvement. Note that system 41a on the pre-quantization model side always compares with the inferred value x itself, so the expected value of the improvement in similarity in learning is essentially 1.

[0059] In the discrimination learning model 40, the discriminator 41 is trained using the expected value determined as described above. Specifically, the inference values ​​D(w+Δw) and D(w q) approaches the expected value 1. When training the classifier 41, training is performed using both the system 41a on the pre-quantization model side and the system 41b on the post-quantization model side, and the weights of the neural network of the classifier 41 are updated. When training of the classifier 41 is completed, a regularization term is derived using this classifier 41. Note that what is reflected in the first neural network of the pre-quantization model 20 is the regularization term derived by the system 41a on the pre-quantization model side of the classifier 41.

[0060] In the above, the expected value (first expected value) is expressed by two values, 0 and 1, but it is not limited to this and may be expressed by two values, 0 and S (S>0). For example, the expected value may always be S (S>0) for an inferred value x, and may be -S for an inferred value G(z) when the similarity of the inferred value G(z) is high compared to the previous inferred value G(z) when viewed in chronological order, and may be 0 when the similarity is not high.

[0061] When training the classifier 41, the first feature amount and the weight parameter w calculated based on the weight parameter w+Δw are q The learning may be performed using a third loss function as an index, the third loss function having as input each of the second feature amounts calculated based on the above.

[0062] Each of the first feature amount and the second feature amount is a value output from the convolutional neural network at the boundary between the convolutional neural network and the fully connected neural network of the classifier 41. The first feature amount is a feature amount in the system 41a on the pre-quantization model side, and the second feature amount is a feature amount in the system 41b on the post-quantization model side.

[0063] In training the classifier 41, a third loss function of the classifier 41 is set based on the first feature amount and the second feature amount, and the classifier 41 is trained so as to reduce the third loss function using the third loss function as an index.

[0064] As the third loss function, for example, a Triplet Loss function is used. The Triplet Loss function has the features of the neural network (reference value, value a and value b derived from the reference value) as factors, and has the property of narrowing the distance between the reference value and value a through learning, while widening the distance between the reference value and value b. This makes it possible to proceed with learning by making it easier to separate (reference value or value a) from value b.

[0065] For example, in order to make it easy to separate the feature amount (positive feature amount) of the system 41a on the pre-quantization model side from the feature amount (negative feature amount) of the system 41b on the post-quantization model side for the number of learning iterations N, Reference value: (N-1)th pre-quantization model side feature quantity of system 41a Value a: Feature value of system 41a on the Nth pre-quantization model side Value b: Feature value of system 41b on the model side after N-th quantization It is desirable to do so.

[0066] These can be expressed by the following equation (Equation 3).

[0067]

number

[0068] That is, when setting the third loss function, it is desirable to set the first feature obtained in the (N-1)th learning (N is an integer greater than 1) as the reference feature, the first feature obtained in the Nth learning as the positive feature, and the second feature obtained in the Nth learning as the negative feature.

[0069] [1-4. Hardware configuration] Next, a hardware configuration of a derivation device constituting a neural network derivation model according to the present embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a hardware configuration of a computer 1000 that realizes the functions of the derivation device by software.

[0070] 8, the computer 1000 is a computer including an input device 1001, an output device 1002, a CPU 1003, an internal storage 1004, a RAM 1005, a reading device 1007, a transmission / reception device 1008, and a bus 1009. The input device 1001, the output device 1002, the CPU 1003, the internal storage 1004, the RAM 1005, the reading device 1007, and the transmission / reception device 1008 are connected by the bus 1009.

[0071] The input device 1001 is a user interface device such as an input button, a touch pad, a touch panel display, etc., and accepts user operations. The input device 1001 may be configured to accept voice operations and remote operations using a remote control or the like in addition to accepting touch operations by the user.

[0072] The output device 1002 is used in combination with the input device 1001, and is configured with a touch pad or a touch panel display, etc., to notify the user of information that should be made known to the user.

[0073] The built-in storage 1004 is a flash memory, etc. The built-in storage 1004 may also store in advance at least one of a program for implementing the functions of the derivation device and an application that utilizes the functional configuration of the derivation device.

[0074] The RAM 1005 is a random access memory, and is used to store data and the like when a program or application is executed.

[0075] The reading device 1007 reads information from a recording medium such as a USB (Universal Serial Bus) memory. The reading device 1007 reads the above-mentioned programs and applications from a recording medium on which the programs and applications are recorded, and stores the programs and applications in the built-in storage 1004.

[0076] The transmitting / receiving device 1008 is a communication circuit for performing wireless or wired communication. The transmitting / receiving device 1008 communicates with, for example, a server device connected to a network, downloads the above-mentioned programs and applications from the server device, and stores them in the built-in storage 1004.

[0077] The CPU 1003 is a central processing unit that copies programs and applications stored in the internal storage 1004 to the RAM 1005, and sequentially reads out instructions included in the programs and applications from the RAM 1005 and executes them.

[0078] [1-5. Effects, etc.] As described above, the neural network derivation method of this embodiment includes a first learning step S10 in which a first neural network having a first parameter (e.g., weight parameter w) is learned using a first loss function as an index, and a second learning step S20 in which, after the first learning step S10, the first neural network is learned using a second loss function obtained by adding a regularization term to the first loss function as an index. The regularization term is a second parameter (e.g., weight parameter w) obtained by adding a variation to the first parameter based on the first neural network. q ) is derived, a similarity between a first inferred value x of the first neural network and a second inferred value G(z) of the second neural network is determined based on a time series change in the similarity between the first inferred value x of the first neural network and the second inferred value G(z) of the second neural network.

[0079] According to this, a regularization term is obtained based on the time-series change in the similarity between the first inferred value x of the first neural network and the second inferred value G(z) of the second neural network, and the first neural network can be trained using the second loss function including this regularization term as an index. This makes it possible to derive a neural network that is robust against fluctuations in the parameters of the neural network.

[0080] The regularization term may also be determined so that it is smaller when the similarity is higher and larger when the similarity is lower.

[0081] This makes it possible to prevent the first parameter used in the neural network from becoming a parameter that is likely to change the accuracy of the inference value, thereby making it possible to derive a neural network that is robust against fluctuations in the neural network parameters.

[0082] The neural network derivation method further includes a first regularization term learning step S11 for learning a classifier 41 for determining a regularization term between the first learning step S10 and the second learning step S20. In the first regularization term learning step S11, the first parameter and the second parameter may be input to the classifier 41, and the classifier 41 may be trained using a first expected value calculated from the first inferred value x and the second inferred value G(z).

[0083] According to this, since the classifier 41 can be trained using the first expected value, the first neural network can be trained using as an index the second loss function including the regularization term determined by the trained classifier 41. This makes it possible to derive a neural network that is robust against fluctuations in the parameters of the neural network.

[0084] In addition, the first expected value may always be S (S>0) for the first inferred value x, and may be -S for the second inferred value G(z) if the similarity is high compared to the previous second inferred value G(z) when viewed in chronological order, or may be 0 if the similarity is not high.

[0085] According to this, the first expected value can be appropriately determined, and the classifier 41 can be appropriately trained. Therefore, the first neural network can be trained using the second loss function including the regularization term determined by the trained classifier 41 as an index. This makes it possible to derive a neural network that is robust against fluctuations in the parameters of the neural network.

[0086] In addition, in the first regularization term learning step S11, the classifier 41 may be trained using a third loss function that receives as input a first feature calculated based on a first parameter and a second feature calculated based on a second parameter.

[0087] According to this, the classifier 41 can be appropriately trained, and the first neural network can be trained using as an index the second loss function including the regularization term determined by the trained classifier 41. This makes it possible to derive a neural network that is robust against fluctuations in the parameters of the neural network.

[0088] In addition, the third loss function may be a Triplet Loss function, in which a first feature obtained in the N-1th learning (N is an integer greater than 1) is set as a reference feature, a first feature obtained in the Nth learning is set as a positive feature, and a second feature obtained in the Nth learning is set as a negative feature.

[0089] According to this, the classifier 41 can be appropriately trained based on the third loss function. Therefore, the first neural network can be trained using the second loss function including the regularization term determined by the trained classifier 41 as an index. This makes it possible to derive a neural network that is robust against fluctuations in the parameters of the neural network.

[0090] Also, the first parameter may be expressed in a first numerical representation, and the second parameter may be obtained by converting the first parameter into the second numerical representation.

[0091] The neural network deriving method of this embodiment can derive a neural network that is robust against conversion even when parameters are converted from a first numerical representation to a second numerical representation.

[0092] Also, the first numerical representation may be a real number consisting of float values, the second numerical representation may be an integer, and the second parameter may be obtained by quantizing the first parameter.

[0093] The neural network deriving method of this embodiment can derive a neural network that is robust against the transformation even when the parameters are quantized and transformed.

[0094] (Embodiment 2) [2-1. Derivation model for deriving neural networks] Next, a derivation model for deriving a neural network will be described in embodiment 2. In embodiment 2, an example of generating a neural network having robustness against input data fluctuations in addition to the robustness against parameter fluctuations shown in embodiment 1 will be described.

[0095] 9 is a diagram showing a derived model 10A for deriving a neural network in the embodiment 2. As shown in FIG 9, the derived model 10A includes a pre-quantization model 20, a post-quantization model 30, and a discriminative learning model 40.

[0096] The pre-quantization model 20 is configured by a first neural network having a weight parameter (first parameter) w. Input data (first input data) z is input to the pre-quantization model 20. The input data z is expressed in a third numerical expression such as a real number consisting of a float value. The pre-quantization model 20 to which the input data z is input outputs an inference value (third inference value) x as an output value. Machine learning is performed on the pre-quantization model 20 based on a predetermined learning data set including the input data z. In addition, when the classifier 41 is trained, it operates with a weight parameter w+Δw obtained by adding Δw to the weight parameter w.

[0097] The quantized model 30 has a weight parameter (second parameter) w q The quantized model 30 is configured by a second neural network having the following. Input data (second input data) z+Δz is input to the quantized model 30. Δz is obtained, for example, by fixing the weight parameters of the trained pre-quantization model 20 and training the input data z so that it becomes an incorrect inferred value. In this case, Δz appears as a difference from the original input data z. The input data z+Δz is expressed by a fourth numerical expression different from the above-mentioned third numerical expression. The fourth numerical expression is, for example, a numerical expression with fixed-point precision such as an integer. The input data z+Δz is a value obtained by adding a fluctuation to the input data z, and is slightly different in value from the input data z. The quantized model 30 to which the input data z+z is input outputs an inferred value (fourth inferred value) G(z+Δz) as an output value.

[0098] The discrimination learning model 40 is a model for learning a discriminator 41 that discriminates the accuracy of an inferred value, and includes the discriminator 41 and the like.

[0099] The discriminator 41 is provided with a weighting parameter w+Δw before quantization and a weighting parameter w after quantization. q are input, and also input data z and z+Δz are input. The discriminator 41 outputs an inference value D(z, w+Δw) corresponding to the input data z and the weight parameter w+Δw. The discriminator 41 also outputs an inference value D(z, w+Δw) corresponding to the input data z+Δz and the weight parameter wq Correspondingly, the inferred value D(z+Δz,w q Note that the notation D(A,B) here means an inferred value that depends on both tensor A and tensor B.

[0100] The discriminative learning model 40 receives the inferred value x of the pre-quantization model 20 and the inferred value G(z+Δz) of the quantized model 30. The discriminative learning model 40 compares the input inferred value x and inferred value G(z+Δz) with the above inferred values ​​D(z,w+Δw) and D(z+Δz,w q ) and performs backpropagation to train the classifier 41. The discriminative training model 40 then derives the above-mentioned regularization term using the trained classifier 41. The regularization term derived by the discriminative training model 40 is used when retraining the first neural network of the pre-quantization model 20.

[0101] [2-2. Neural network derivation method] Next, a method for deriving a neural network using the above-mentioned derivation model 10A will be described.

[0102] FIG. 10 is a flow diagram showing a neural network deriving method according to this embodiment.

[0103] The neural network derivation method includes a first learning step S10, a regularization term learning step S17 (a second regularization term learning step), and a second learning step S20.

[0104] The first learning step S10 is a step of learning the first neural network of the pre-quantization model 20. The first learning step S10 is executed within the dashed line in (a) of FIG. 10. In this step, the first neural network is learned using a predetermined learning data set with the first loss function as an index. In the first learning step S10, a weight parameter w in the first neural network is first obtained.

[0105] The regularization term learning step S17 is a step of learning to derive the regularization term described above. The regularization term learning step S17 includes a step S12 of generating the quantized model 30, a step S13 of learning the classifier 41, and a step S14 of deriving the regularization term from the classifier 41.

[0106] The step S12 of generating the quantized model 30 includes quantizing the weight parameters w based on the first neural network. q This step S12 is executed within the dashed line in FIG. 10(b). q is a quantized value obtained by further varying the weighting parameter w+Δw, and can be obtained, for example, by quantizing the weighting parameter w+Δw.

[0107] Step S13 for training the classifier 41 is a step executed by the systems within the dashed lines in each of (c) and (d) of Fig. 10. Here, the system within the dashed line shown in (c) of Fig. 10 is called system 41a on the pre-quantization model side, and the system within the dashed line shown in (d) of Fig. 10 is called system 41b on the post-quantization model side.

[0108] As shown in (c) of FIG. 10, in a system 41a on the pre-quantization model side, a weight parameter w+Δw and input data z are input to a discriminator 41. The discriminator 41 to which the weight parameter w+Δw and the input data z are input outputs an inferred value D(z, w+Δw). In addition, the discriminative learning model 40 receives the inferred value x and the inferred value G(z+Δz), which are outputs of the pre-quantization model 20 and the quantized model 30. The discriminative learning model 40 trains the discriminator 41 so as to improve the accuracy of the inferred value D(z, w+Δw) based on a time-series change in the similarity between the inferred value x and the inferred value G(z+Δz).

[0109] The similarity here indicates the degree of similarity between the two inferred values ​​x and G(z+Δz). The cosine similarity mentioned above is used as the similarity.

[0110] As shown in FIG. 10(d), in the system 41b on the quantized model side, the weight parameter w q and the input data z+Δz are input to the discriminator 41. q The discriminator 41 to which the input data z+Δz is input outputs an inference value D(z+Δz, w q The discriminative learning model 40 outputs an inferred value D(z+Δz, w) based on the time-series change in the similarity between the input inferred value x and the inferred value G(z+Δz). q The classifier 41 is trained so as to improve the accuracy of the classification.

[0111] The weights in the system 41a on the pre-quantization model side and the system 41b on the quantization model side are common (the weight parameters in the system 41a on the pre-quantization model side are quantized and become the weight parameters of the system 41b on the quantization model side), and the learning of the classifier 41 by these systems 41a and 41b is performed simultaneously.

[0112] Step S14 of deriving a regularization term is a step of deriving a regularization term using the discriminator 41. The regularization term has a negative correlation with the magnitude of the similarity between the inferred value x and the inferred value G(z+Δz), and is determined so as to be small as the similarity increases and large as the similarity decreases, for example, in a time series. The regularization term derived by the discriminator 41 is reflected in the learning of the first neural network of the pre-quantization model 20.

[0113] The second learning step S20 is a step of learning the first neural network using a second loss function (second loss function = first loss function + regularization term) obtained by adding a regularization term to the first loss function as an index. The second learning step S20 is also executed within the dashed line in (a) of FIG. 10, and a predetermined learning data set is used as in the first learning step S10. The weight parameter w of the first neural network is updated by the second learning step S20.

[0114] Fig. 11 is a flowchart showing a neural network deriving method executed following Fig. 10. In this neural network deriving method, a regularization term learning step S17A similar to the regularization term learning step S17 is executed after the second learning step S20. The regularization term learning step S17A includes a step S12 of generating the above-mentioned quantized model 30, a step S13 of learning the classifier 41, and a step S14 of deriving a regularization term from the classifier 41.

[0115] In the neural network derivation method of the present embodiment, a robust first neural network is generated by repeating the second learning step S20 and the regularization term learning step S17A. In addition, a weight parameter w of the first neural network generated by the above repetition is quantized to generate a second neural network that is robust against fluctuations in parameters and input data.

[0116] [2-3. Operation of discriminative learning model] Next, the operation of the discrimination learning model 40 for training the discriminator 41 will be described.

[0117] Fig. 12 is a schematic diagram showing a discriminative learning model 40 included in the derived model 10A. Note that Fig. 12 also shows a pre-quantization model 20 and a post-quantization model 30.

[0118] A weight parameter w+Δw and input data z are input to a system 41a on the pre-quantization model side of the classifier 41, and an inferred value D(z, w+Δw) is output. A weight parameter w q , input data z+Δz is input, and the inferred value D(z+Δz, w q ) is output.

[0119] Moreover, the discriminative learning model 40 receives the inferred value x from the pre-quantization model 20 and the inferred value G(z+Δz) from the quantized model 30. In the discriminative learning model 40, a classifier 41 is trained using the similarity and expected value calculated from the inferred value x and the inferred value G(z+Δz).

[0120] Here, the expected value (second expected value) used when training the classifier 41 will be described.

[0121] The expected value is a label used when learning, and is determined based on the inference value x and G(z+Δz) input to the discriminative learning model 40, as shown in (Equation 4).

[0122] Expected value = {similarity of inferred value (x or G(z+Δ)) for x, improvement in similarity of inferred value (x or G(z+Δ)) for x compared to previous evaluation} (Equation 4)

[0123] Table 2 shows the expected values ​​for the system 41a on the pre-quantization model side of the classifier and the system 41b on the post-quantization model side of the classifier. The expected values ​​are the inferred values ​​D(z,w+Δw) and D(z+Δz,w q ) is represented by two values, 0 and 1, so that it can be trained as a two-class classifier, which is a relatively easy discriminator to train when rating it good or bad.

[0124] [Table 2]

[0125] As shown in Table 2, the similarity for x of system 41a on the pre-quantization model side is expected to be 1 because the inferred value x is the same as x. The similarity for x of system 41b on the quantization model side is expected to be 0 because the inferred value G(z+Δz) is different from x. The improvement in the similarity of the inferred values ​​x and G(z+Δz) of system 41a on the pre-quantization model side and system 41b on the quantization model side is expected to be 1 if the similarity calculated this time has improved compared to the similarity calculated last time, and is expected to be 0 if there has been no improvement. Note that system 41a on the pre-quantization model side always compares with the inferred value x itself, so the expected value of the improvement in similarity in learning is essentially 1.

[0126] In the discrimination learning model 40, the discriminator 41 is trained using the expected value determined as described above. Specifically, the inference values ​​D(z, w+Δw) and D(z+Δz, w q ) approaches the expected value 1. When training the classifier 41, training is performed using both the system 41a on the pre-quantization model side and the system 41b on the post-quantization model side, and the weights of the neural network of the classifier 41 are updated. When training of the classifier 41 is completed, a regularization term is derived using the system 41a on the pre-quantization model side of the classifier 41.

[0127] In the above, the expected value (second expected value) is expressed by two values, 0 and 1, but it is not limited to this and may be expressed by two values, 0 and S (S>0). For example, the expected value may always be S (S>0) for an inferred value x, and may be -S for an inferred value G(z+Δz) when the inferred value G(z+Δz) is viewed in chronological order and is 0 when the similarity is not high compared to the previous inferred value G(z+Δz).

[0128] When training the classifier 41, the third feature calculated based on the weight parameter w+Δw and the input data z, and the weight parameter w q and the fourth feature amount calculated based on the input data z+Δz may be used as an index for learning.

[0129] Each of the third feature amount and the fourth feature amount is a value output from the convolutional neural network at the boundary between the convolutional neural network and the fully connected neural network of the classifier 41. The third feature amount is a feature amount in the system 41a on the pre-quantization model side, and the fourth feature amount is a feature amount in the system 41b on the post-quantization model side.

[0130] In training the classifier 41, a fourth loss function of the classifier 41 is set based on the third feature amount and the fourth feature amount, and the classifier 41 is trained so as to reduce the fourth loss function using the fourth loss function as an index.

[0131] As the fourth loss function, for example, a Triplet Loss function is used. The Triplet Loss function has features of a neural network (a reference value, and values ​​a and b derived from the reference value) as factors, and has the property of narrowing the distance between the reference value and value a through learning while widening the distance between the reference value and value b.

[0132] For example, in order to make it easy to separate the feature amount (positive feature amount) of the system 41a on the pre-quantization model side from the feature amount (negative feature amount) of the system 41b on the post-quantization model side for the number of learning iterations N, Reference value: (N-1)th pre-quantization model side feature quantity of system 41a Value a: Feature value of system 41a on the Nth pre-quantization model side Value b: Feature value of system 41b on the model side after N-th quantization It is desirable to do so.

[0133] That is, when setting the fourth loss function, it is desirable to set the third feature obtained in the (N-1)th learning (N is an integer greater than 1) as the reference feature, set the fourth feature obtained in the Nth learning as the positive feature, and set the fourth feature obtained in the Nth learning as the negative feature.

[0134] [2-4. Effects, etc.] As described above, the neural network derivation method of the second embodiment includes, in addition to the first embodiment, a second regularization term learning step S17 for learning the classifier 41 to determine the regularization term between the first learning step S10 and the second learning step S20. In the second regularization term learning step S17, the first input data (e.g., input data z) and the second input data (e.g., input data z+Δz) are input to the classifier 41, and the classifier 41 is learned using a second expected value obtained from a third inference value x of the first neural network when the first input data is input to the first neural network and a fourth inference value G(z+Δz) of the second neural network when the second input data obtained by adding a variation to the first input data is input to the second neural network.

[0135] According to this, since the classifier 41 can be trained using the second expected value, the first neural network can be trained using the second loss function including the regularization term determined by the trained classifier 41 as an index. This makes it possible to derive a neural network that is robust against fluctuations in parameters and input data. This also significantly improves resistance to adversarial attacks (attacks using adversarial samples).

[0136] In addition, the second expected value is always S (S>0) for the third inferred value x, and for the fourth inferred value G(z+Δz), it may be -S if the similarity of the fourth inferred value G(z+Δz) is high compared to the previous fourth inferred value G(z+Δz) when viewed in time series, and may be 0 if the similarity is not high.

[0137] According to this, the second expected value can be appropriately determined, and the classifier 41 can be appropriately trained. Therefore, the first neural network can be trained using the second loss function including the regularization term determined by the trained classifier 41 as an index. This makes it possible to derive a neural network that is robust against fluctuations in parameters and input data.

[0138] In addition, in the second regularization term learning step S17, the classifier 41 may be trained using a fourth loss function having as inputs a third feature calculated based on the first parameter and the first input data, and a fourth feature calculated based on the second parameter and the second input data.

[0139] According to this, the classifier 41 can be appropriately trained, and the first neural network can be trained using as an index the second loss function including the regularization term determined by the trained classifier 41. This makes it possible to derive a neural network that is robust against fluctuations in parameters and input data.

[0140] In addition, the fourth loss function may be a Triplet Loss function, in which a third feature obtained in the N-1th learning (N is an integer greater than 1) is set as a reference feature, a third feature obtained in the Nth learning is set as a positive feature, and a fourth feature obtained in the Nth learning is set as a negative feature.

[0141] According to this, the classifier 41 can be appropriately trained based on the fourth loss function. Therefore, the first neural network can be trained using the second loss function including the regularization term determined by the trained classifier 41 as an index. This makes it possible to derive a neural network that is robust against fluctuations in parameters and input data.

[0142] Also, the first input data may be represented in a third numerical representation, and the second input data may be represented in a fourth numerical representation different from the third numerical representation.

[0143] The neural network deriving method of the present embodiment can derive a neural network having robustness even when the input data changes from the third numerical representation to the fourth numerical representation.

[0144] Furthermore, the third numerical representation may be a real number consisting of a float value, and the fourth numerical representation may be an integer.

[0145] The neural network deriving method of the present embodiment can derive a neural network having robustness even when the third numerical representation is a real number consisting of a float value and the fourth numerical representation is an integer.

[0146] (Embodiment 3) [3-1. Derivation model for deriving neural networks] Next, a derivation model for deriving a neural network will be described in embodiment 3. In embodiment 3, an example of generating a neural network having robustness against fluctuations in input data will be described.

[0147] 13 is a diagram showing a derived model 10B for deriving a neural network in the embodiment 3. As shown in FIG 13, the derived model 10B is composed of a reference model 20B, a clone model 30B, and a discriminative learning model 40.

[0148] The reference model 20B is configured by a first neural network having a weight parameter (first parameter) w. Input data (first input data) z is input to the reference model 20B. The input data z is expressed by a third numerical expression such as a real number consisting of a float value. The reference model 20B to which the input data z is input outputs an inferred value (fifth inferred value) x as an output value. Machine learning is performed on the reference model 20B based on a predetermined learning data set including the input data z. In addition, when the classifier 41 is trained, it operates with a weight parameter w+Δw obtained by adding Δw to the weight parameter w.

[0149] The clone model 30B is configured by a second neural network having the same weight parameter w as the reference model 20B. The clone model 30B is input with input data (second input data) z+Δz. Δz is obtained, for example, by fixing the weight parameter of the trained reference model 20B and training the input data z so that it becomes an incorrect inference value. In this case, Δz appears as a difference from the original input data z. The input data z+Δz is expressed by a fourth numerical expression different from the above-mentioned third numerical expression. The fourth numerical expression is, for example, a numerical expression with fixed-point precision such as an integer. The input data z+Δz is a value obtained by adding a fluctuation to the input data z, and is slightly different in value from the input data z. The clone model 30B to which the input data z+z is input outputs an inference value (sixth inference value) G(z+Δz) as an output value. In addition, when the classifier 41 is trained, it operates with a weight parameter w+Δw obtained by adding Δw to the weight parameter w.

[0150] The discrimination learning model 40 is a model for learning a discriminator 41 that discriminates the accuracy of an inferred value, and includes the discriminator 41 and the like.

[0151] The two weighting parameters w+Δw and the input data z and z+Δz are input to the classifier 41. The classifier 41 outputs an inferred value D(z,w+Δw) corresponding to the input data z and the weighting parameter w+Δw. The classifier 41 also outputs an inferred value D(z+Δz,w+Δw) corresponding to the input data z+Δz and the weighting parameter w+Δw. Note that the notation of the inferred value D(A,B) here means an inferred value that depends on both tensor A and tensor B.

[0152] The discriminative learning model 40 receives the inferred value x of the reference model 20B and the inferred value G(z+Δz) of the clone model 30B. The discriminative learning model 40 compares the input inferred value x and inferred value G(z+Δz) with the above inferred values ​​D(z,w+Δw) and D(z+Δz,w+Δw) and trains the discriminator 41 by performing backpropagation. The discriminative learning model 40 then derives the above-mentioned regularization term using the trained discriminator 41. The regularization term derived by the discriminative learning model 40 is used when retraining the first neural network of the reference model 20B.

[0153] [3-2. Neural network derivation method] Next, a method for deriving a neural network using the above-mentioned derivation model 10B will be described.

[0154] FIG. 14 is a flow diagram showing a neural network deriving method according to this embodiment.

[0155] The neural network derivation method includes a first learning step S10, a regularization term learning step S18 (a third regularization term learning step), and a second learning step S20.

[0156] The first learning step S10 is a step of learning the first neural network of the reference model 20B. The first learning step S10 is performed within the dashed line in (a) of FIG. 14. In this step, the first neural network is learned using a predetermined learning data set with the first loss function as an index. In the first learning step S10, a weight parameter w in the first neural network is first obtained.

[0157] The regularization term learning step S18 is a step of learning to derive the regularization term described above. The regularization term learning step S18 includes a step S12A of generating a clone model 30B, a step S13 of learning a classifier 41, and a step S14 of deriving a regularization term from the classifier 41.

[0158] Step S12A for generating the clone model 30B is a step for deriving a second neural network having the same weight parameter w based on the first neural network. This step S12A is executed within the dashed line in (b) of FIG.

[0159] Step S13 for training the classifier 41 is a step executed in the systems within the dashed lines in each of (c) and (d) of Figure 14. Here, the system within the dashed line shown in (c) of Figure 14 is called the reference model side system 41c, and the system within the dashed line shown in (d) of Figure 14 is called the clone model side system 41d.

[0160] As shown in (c) of FIG. 14, in system 41c on the reference model side, weight parameter w+Δw and input data z are input to discriminator 41. Discriminator 41 to which weight parameter w+Δw and input data z are input outputs an inferred value D(z, w+Δw). In addition, inferred values ​​x and G(z+Δz), which are outputs of reference model 20B and clone model 30B, are input to discriminative learning model 40. Discriminative learning model 40 trains discriminator 41 so as to improve the accuracy of inferred value D(z, w+Δw) based on the time-series change in the similarity between inferred value x and inferred value G(z+Δz).

[0161] The similarity here indicates the degree of similarity between the two inferred values ​​x and G(z+Δz). The cosine similarity mentioned above is used as the similarity.

[0162] 14(d), in system 41d on the clone model side, weight parameter w+Δw and input data z+Δz of clone model 30B are input to discriminator 41. Discriminator 41 to which weight parameter w+Δw and input data z+Δz are input outputs an inferred value D(z+Δz, w+Δw). Discrimination learning model 40 trains discriminator 41 so as to improve the accuracy of inferred value D(z+Δz, w+Δw) based on a time-series change in the similarity between the input inferred value x and inferred value G(z+Δz).

[0163] The weights in the reference model side system 41c and the clone model side system 41d are common to each other, and the learning of the classifier 41 by these systems 41c and 41d is performed simultaneously.

[0164] Step S14 of deriving a regularization term is a step of deriving a regularization term using the discriminator 41. The regularization term has a negative correlation with the magnitude of the similarity between the inferred value x and the inferred value G(z+Δz), and is determined so as to be small as the similarity increases and to be large as the similarity decreases, for example, in a time series. The regularization term derived by the discriminator 41 is reflected in the learning of the first neural network of the reference model 20B.

[0165] The second learning step S20 is a step of learning the first neural network using a second loss function (second loss function = first loss function + regularization term) obtained by adding a regularization term to the first loss function as an index. The second learning step S20 is also executed within the dashed line in (a) of FIG. 14, and a predetermined learning data set is used as in the first learning step S10. The weight parameter w of the first neural network is updated by the second learning step S20.

[0166] Fig. 15 is a flowchart showing a neural network deriving method executed following Fig. 14. In this neural network deriving method, a regularization term learning step S18A similar to the regularization term learning step S18 is executed after the second learning step S20. The regularization term learning step S18A includes a step S12A of generating the clone model 30B described above, a step S13 of learning the discriminator 41, and a step S14 of deriving a regularization term from the discriminator 41.

[0167] In the neural network derivation method of the present embodiment, a first neural network having robustness is generated by repeating the second learning step S20 and the regularization term learning step S18A. Also, a second neural network having robustness against fluctuations in input data is generated by giving the same weight parameter w as the first neural network generated by the above repetition.

[0168] [3-3. Operation of discriminative learning model] Next, the operation of the discrimination learning model 40 for training the discriminator 41 will be described.

[0169] Fig. 16 is a schematic diagram showing a discriminative learning model 40 included in the derived model 10B. Note that Fig. 16 also shows a reference model 20B and a clone model 30B.

[0170] A weight parameter w+Δw and input data z are input to system 41c on the reference model side of classifier 41, and an inferred value D(z, w+Δw) is output. A weight parameter w+Δw and input data z+Δz are input to system 41d on the clone model side of classifier 41, and an inferred value D(z+Δz, w+Δw) is output.

[0171] The discriminative learning model 40 also receives an inferred value x from the reference model 20B and an inferred value G(z+Δz) from the clone model 30B. In the discriminative learning model 40, a classifier 41 is trained using the similarity and expected value calculated from the inferred value x and the inferred value G(z+Δz).

[0172] Here, the expected value (third expected value) used when training the classifier 41 will be described.

[0173] The expected value is a label used when learning, and is determined based on the inference value x and G(z+Δz) input to the discriminative learning model 40, as shown in (Equation 5).

[0174] Expected value = {similarity of inferred value (x or G(z+Δz)) for x, improvement of similarity of inferred value (x or G(z+Δz)) for x compared to previous evaluation} (Equation 5)

[0175] Table 3 shows the expected values ​​of the classifier for the reference model system 41c and the clone model system 41d. The expected values ​​are expressed as binary values, 0 and 1, in order to train the classifier as a two-class classifier, which is a classifier that is relatively easy to train, when determining whether the inference values ​​D(z,w+Δw) and D(z+Δz,w+Δw) that are the outputs of the classifier 41 are good or bad.

[0176] [Table 3]

[0177] As shown in Table 3, the similarity of system 41c on the reference model side to x is expected to be 1 because the inferred value x is the same as x. The similarity of system 41d on the clone model side to x is expected to be 0 because the inferred value G(z+Δz) is different from x. The improvement in the similarity of the inferred values ​​x and G(z+Δz) of system 41c on the reference model side and system 41d on the clone model side is expected to be 1 if the similarity calculated this time has improved compared to the similarity calculated last time, and is expected to be 0 if there has been no improvement. Note that system 41c on the reference model side is always compared to the inferred value x itself, so the expected value of the improvement in similarity in learning is essentially 1.

[0178] In the discrimination learning model 40, the classifier 41 is trained using the expected value determined as described above. Specifically, the classifier 41 is trained so that the inference values ​​D(z, w + Δw) and D(z + Δz, w + Δw) output from the classifier 41 approach the expected value 1. When training the classifier 41, training is performed using both the system 41c on the reference model side and the system 41d on the clone model side, and the weights of the neural network of the classifier 41 are updated. When training of the classifier 41 is completed, a regularization term is derived using the system 41c on the reference model side of the classifier 41.

[0179] In the above, the expected value (third expected value) is expressed by two values, 0 and 1, but it is not limited to this and may be expressed by two values, 0 and S (S>0). For example, the expected value may always be S (S>0) for an inferred value x, and may be -S for an inferred value G(z+Δz) when the inferred value G(z+Δz) is viewed in chronological order and is 0 when the similarity is not high compared to the previous inferred value G(z+Δz).

[0180] In addition, when training the classifier 41, training may be performed using a fifth loss function as an index, the fifth loss function having as inputs the fifth feature calculated based on the input data z and the sixth feature calculated based on the input data z+Δz.

[0181] Each of the fifth feature amount and the sixth feature amount is a value output from the convolutional neural network at the boundary between the convolutional neural network and the fully connected neural network of the classifier 41. The fifth feature amount is a feature amount in the system 41c on the reference model side, and the sixth feature amount is a feature amount in the system 41d on the clone model side.

[0182] In training the classifier 41, a fifth loss function of the classifier 41 is set based on the fifth feature amount and the sixth feature amount, and the classifier 41 is trained so as to reduce the fifth loss function using the fifth loss function as an index.

[0183] As the fifth loss function, for example, a Triplet Loss function is used. The Triplet Loss function has features of a neural network (a reference value, and values ​​a and b derived from the reference value) as factors, and has the property of narrowing the distance between the reference value and value a through learning while widening the distance between the reference value and value b.

[0184] For example, in order to make it easy to separate the feature amount (positive feature amount) of the system 41c on the reference model side from the feature amount (negative feature amount) of the system 41d on the clone model side for the number of times N of learning repetitions, Reference value: (N-1)th time feature value of system 41c on the reference model side Value a: Feature value of the system 41c on the reference model side for the Nth time Value b: Feature value of system 41d on the Nth clone model side It is desirable to do so.

[0185] That is, when setting the fifth loss function, it is desirable to set the fifth feature obtained in the (N-1)th learning (N is an integer greater than 1) as the reference feature, set the sixth feature obtained in the Nth learning as the positive feature, and set the sixth feature obtained in the Nth learning as the negative feature.

[0186] [3-4. Effects, etc.] As described above, the neural network derivation method of the third embodiment includes a first learning step S10 for learning a first neural network to which first input data (e.g., input data z) is input using a first loss function as an index, and a second learning step S20 for learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step S10. The regularization term is determined based on a time-series change in the similarity between the fifth inferred value x of the first neural network and the sixth inferred value G(z+Δz) of the second neural network after a second neural network to which second input data (e.g., input data z+Δz) obtained by adding a variation to the first input data is input based on the first neural network is derived.

[0187] According to this, a regularization term is obtained based on the time series change in the similarity between the fifth inference value x of the first neural network and the sixth inference value G(z+Δz) of the second neural network, and the first neural network can be trained using the second loss function including this regularization term as an index. This makes it possible to derive a neural network that is robust against fluctuations in the input data of the neural network. This also significantly improves resistance to adversarial attacks (attacks using adversarial samples).

[0188] The regularization term may also be determined so that it is smaller when the similarity is higher and larger when the similarity is lower.

[0189] This makes it possible to prevent the first parameter used in the neural network from becoming a parameter that is likely to change the accuracy of the inference value, thereby enabling the deriving of a neural network that is robust against fluctuations in the input data of the neural network.

[0190] The neural network derivation method further includes a third regularization term learning step S18 for learning a classifier 41 for determining a regularization term between the first learning step S10 and the second learning step S20. In the third regularization term learning step S18, the first input data and the second input data may be input to the classifier 41, and the classifier 41 may be trained using a third expected value calculated from the fifth inferred value x and the sixth inferred value G(z+Δz).

[0191] According to this, since the classifier 41 can be trained using the third expected value, the first neural network can be trained using as an index the second loss function including the regularization term determined by the trained classifier 41. This makes it possible to derive a neural network that is robust against fluctuations in input data.

[0192] In addition, the third expected value is always S (S>0) for the fifth inferred value, and for the sixth inferred value, it may be -S if the sixth inferred value is more similar to the previous sixth inferred value when viewed in chronological order, or it may be 0 if the similarity is not more similar.

[0193] According to this, the third expected value can be appropriately determined, and the classifier 41 can be appropriately trained. Therefore, the first neural network can be trained using the second loss function including the regularization term determined by the trained classifier 41 as an index. This makes it possible to derive a neural network that is robust against fluctuations in input data.

[0194] In addition, in the third regularization term learning step, the classifier 41 may be trained using a fifth loss function that has as inputs the fifth feature calculated based on the first input data and the sixth feature calculated based on the second input data.

[0195] According to this, the classifier 41 can be appropriately trained, and the first neural network can be trained using as an index the second loss function including the regularization term determined by the trained classifier 41. This makes it possible to derive a neural network that is robust against fluctuations in input data.

[0196] In addition, the fifth loss function may be a Triplet Loss function, in which the fifth feature obtained in the N-1th learning (N is an integer greater than 1) is set as the reference feature, the fifth feature obtained in the Nth learning is set as the positive feature, and the sixth feature obtained in the Nth learning is set as the negative feature.

[0197] According to this, the classifier 41 can be appropriately trained based on the fifth loss function. Therefore, the first neural network can be trained using the second loss function including the regularization term determined by the trained classifier 41 as an index. This makes it possible to derive a neural network that is robust against fluctuations in input data.

[0198] Also, the first input data may be represented in a third numerical representation, and the second input data may be represented in a fourth numerical representation different from the third numerical representation.

[0199] The neural network deriving method of the present embodiment can derive a neural network having robustness even when the input data changes from the third numerical representation to the fourth numerical representation.

[0200] Furthermore, the third numerical representation may be a real number consisting of a float value, and the fourth numerical representation may be an integer.

[0201] The neural network deriving method of the present embodiment can derive a neural network having robustness even when the third numerical representation is a real number consisting of a float value and the fourth numerical representation is an integer.

[0202] (Embodiment 4) [4-1. Derivation model for deriving neural networks] Next, a derivation model for deriving a neural network will be described in embodiment 4. In embodiment 4, an example of generating a neural network having robustness against fluctuations in weight parameters will be described.

[0203] Fig. 17 is a diagram showing a derivation model 10C for deriving a neural network in the embodiment 4. As shown in Fig. 17, the derivation model 10C is composed of a reference model 20C and a minute fluctuation model 30C.

[0204] The reference model 20C is configured by a first neural network having a weight parameter (first parameter) w. Input data (first input data) z is input to the reference model 20C. The input data z is expressed by a third numerical expression such as a real number consisting of float values. The reference model 20C to which the input data z is input outputs an inference value (seventh inference value) x as an output value. Machine learning is performed on the reference model 20C based on a predetermined learning data set including the input data z. The reference model is also configured by multiple layers, and the output of each layer is represented as a feature value Rfeat[N] (N is the layer index). The input feature value Rfeatin[N] of each layer is equal to Rfeat[N-1].

[0205] The minute fluctuation model 30C is configured by a second neural network having a weight parameter (second parameter) that is the same as the weight parameter w of the reference model 20C plus a minute fluctuation Δw. Input data (first input data) z is input to the minute fluctuation model 30C, and an inferred value (eighth inferred value) G(z) is output as an output value. The minute fluctuation model 30C and the reference model 20C share the weight parameter w, and the weight parameter w of the reference model 20C is updated by learning of the minute fluctuation model 30C. The minute fluctuation model is configured by multiple layers, and the output of each layer is represented as a feature Gfeat[N] (N is the layer index). The input feature Gfeatin[N] of each layer is Gfeat[N-1] plus the minute fluctuation ΔGfeat[N].

[0206] In addition, the feature quantity output from each layer is originally the feature quantity inside the network. Therefore, the feature quantity output from each layer is included in the internal feature quantity of the network, and is substantially the same as the internal feature quantity. The minute variation model 30C has the same configuration as the reference model 20C, except for the weight parameters.

[0207] [4-2. Neural network derivation method] Next, a method for deriving a neural network using the above-mentioned derivation model 10C will be described.

[0208] FIG. 18 is a flowchart showing a neural network deriving method according to the fourth embodiment.

[0209] The neural network derivation method includes a first learning step S10, a regularization term construction step S19, and a second learning step S21.

[0210] The first learning step S10 is a step of learning the first neural network of the reference model 20C. The first learning step S10 is executed within the dashed line in (a) of FIG. 18. In this step, the first neural network is learned using a predetermined learning data set with the first loss function as an index. In the first learning step S10, a weight parameter w in the first neural network is first obtained.

[0211] The regularization term construction step S19 is a step of deriving a regularization term different from the above-mentioned regularization term. The regularization term construction step S19 includes a step S12B of generating a small fluctuation model 30C, and a step S15 of deriving a regularization term from the reference model 20C and the small fluctuation model 30C.

[0212] Step S12B of generating the minute variation model 30C is a step of deriving a second neural network based on the first neural network, which has the same weight parameter w plus the minute variation Δw. This step S12B is performed within the dashed line in FIG. 18(b). The weight is (w+Δw). The original weight w is shared with the reference model.

[0213] Step S15 of deriving the regularization term is a step executed in the system of FIG. 18(c). As shown in FIG. 18(c), the feature similarity between the reference model and the small variation model is obtained. FIG. 19 is a diagram showing the definition of feature similarity in the fourth embodiment. In this figure, in the reference model and the small variation model that are composed of multiple layers, the output of each layer is shown as Rfeat[N] and Gfeat[N] (N is the layer index) as a feature. The similarity here indicates the degree of similarity between the two Rfeat[N] and Gfeat[N]. As the similarity, the cos similarity described above is used.

[0214] The regularization term is defined as the inverse sign of the feature similarity of layer N corresponding to each layer weight w[N].

[0215] The second learning step S21 is a step of learning the first neural network using a second loss function (second loss function = first loss function + regularization term) obtained by adding a regularization term to the first loss function as an index. The second learning step S21 is executed within the dashed line in (c) of FIG. 18, and a predetermined learning data set is used as in the first learning step S10. As described above, the original weight w is shared between the reference model and the small variation model, so the weight parameter w of the first neural network is updated by the second learning step S21.

[0216] [4-3. Effects, etc.] As described above, the neural network derivation method of the fourth embodiment includes a first learning step S10 in which a first neural network to which first input data (e.g., input data z) is input is trained using a first loss function as an index, and a second learning step S21 in which the first neural network is trained using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step S10. The regularization term is determined based on the similarity between the feature value of the output of each layer of the first neural network and the feature value of the output of the corresponding layer of the second neural network after a second neural network to which a second parameter (e.g., weight parameter (w+Δw), input data (z+Δz)) obtained by adding a variation to the first parameter is input based on the first neural network is derived, or after a second neural network having a feature value (Gfeat[N-1]+ΔGfeat[N]) obtained by adding a variation to the feature value of the output of each layer of the first neural network is derived.

[0217] According to this, a regularization term is obtained based on the similarity between the feature amount (Rfeat) of the first neural network and the feature amount (Gfeat) of the second neural network, and the first neural network can be trained using the second loss function including this regularization term as an index. This makes it possible to derive a neural network that is robust against variations in the parameters of the neural network without using the classifier used in the first, second, and third embodiments.

[0218] Furthermore, the first parameter may be expressed in a third numerical representation, and the second parameter may be expressed in a fourth numerical representation different from the third numerical representation.

[0219] The neural network deriving method of the present embodiment can derive a neural network having robustness even when the parameters change from the third numerical expression to the fourth numerical expression.

[0220] Furthermore, the third numerical representation may be a real number consisting of a float value, and the fourth numerical representation may be an integer.

[0221] The neural network deriving method of the present embodiment can derive a neural network having robustness even when the third numerical representation is a real number consisting of a float value and the fourth numerical representation is an integer.

[0222] Furthermore, the variation in the second neural network may be limited to a weight parameter (eg, w+Δw).

[0223] The neural network deriving method of the present embodiment can derive a neural network that has robustness against weight parameters even if the fluctuations in the second neural network are only the weight parameters.

[0224] Moreover, the neural network derivation method of this embodiment may be configured as follows.

[0225] For example, the neural network derivation method includes a first learning step of learning a first neural network having a first parameter using a first loss function as an index, and a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step, and the regularization term may be determined based on the correlation between the internal features or inference values ​​of the first neural network and the second neural network after a second neural network having a second parameter obtained by adding a variation to the first parameter based on the first neural network is derived. Also, the regularization term may be determined based on the similarity between the features of the output of at least one layer other than the final layer of the first neural network and the features of the output of the corresponding layer of the second neural network.

[0226] For example, the neural network derivation method may include a first learning step of learning a first neural network having a first weight parameter using a first loss function as an index, and a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step, where the regularization term may be determined based on a relationship between the first neural network and a second neural network having a second weight parameter obtained by adding a variation to the first weight parameter based on the first neural network.

[0227] For example, the neural network derivation method includes a first learning step of learning a first neural network having a first parameter using a first loss function as an index, and a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function after the first learning step, the regularization term being determined based on a relationship between the first neural network and a second neural network based on the first neural network, and the second neural network may further have a configuration in which the input of at least one layer of the first neural network is a feature quantity that is an output of a previous layer plus a variation. Note that the "at least one layer" in the above does not necessarily have to be all layers. In other words, the second neural network may further have a configuration in which the input of some layers of the first neural network is a feature quantity that is an output of a previous layer plus a variation.

[0228] (Other embodiments) Although the neural network derivation method according to the present disclosure has been described based on each embodiment, the present disclosure is not limited to these embodiments. As long as it does not deviate from the gist of the present disclosure, various modifications conceivable by a person skilled in the art to each embodiment and other forms constructed by combining some of the components of each embodiment are also included within the scope of the present disclosure.

[0229] The following forms may also be included within the scope of one or more aspects of the present disclosure.

[0230] (1) Some of the components constituting the above-mentioned derivation device may be a computer system composed of a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse, etc. A computer program is stored in the RAM or the hard disk unit. The microprocessor operates according to the computer program to achieve its function. Here, the computer program is composed of a combination of multiple instruction codes that indicate commands for a computer to achieve a predetermined function.

[0231] (2) Some of the components constituting the above-mentioned derivation device may be composed of one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating multiple components on one chip, and specifically, is a computer system including a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating in accordance with the computer program.

[0232] (3) Some of the components constituting the above derivation device may be composed of an IC card or a standalone module that can be attached to each device. The IC card or the module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or the module may include the above-mentioned ultra-multifunction LSI. The microprocessor operates according to a computer program, causing the IC card or the module to achieve its functions. The IC card or the module may be tamper-resistant.

[0233] (4) Furthermore, some of the components constituting the above-mentioned derivation device may be the computer program or the digital signal recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. Also, some of the components constituting the above-mentioned derivation device may be the digital signal recorded on such a recording medium.

[0234] In addition, some of the components constituting the above-mentioned derivation device may transmit the computer program or the digital signal via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc.

[0235] (5) The present disclosure may be embodied as any of the methods described above, a computer program for implementing these methods by a computer, or a digital signal comprising the computer program.

[0236] (6) The present disclosure may also provide a computer system having a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating in accordance with the computer program.

[0237] (7) The program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network, etc.

[0238] (8) The above-described embodiments and modifications may be combined with each other. [Industrial Applicability]

[0239] The present disclosure can be used as a method for implementing a neural network in a computer or the like, for example, in an image processing method, a voice recognition method, or an object control method. [Explanation of symbols]

[0240] 10, 10A, 10B, 10C Derived Models 20 Pre-quantization model 20B, 20C Standard Model 30 Quantized model 30B Clone Model 30C Microfluctuation Model 40 Discriminative Learning Model 41 Classifier 41a Pre-quantization model system 41b Quantized model system 41c Reference model system 41d Clone model system 1000 Computers 1001 Input Device 1002 Output device 1003 CPU 1004 Internal Storage 1005 RAM 1007 Reading device 1008 Transmitting and receiving device 1009 Bus W, w q Weight Parameter z, z+Δz Input data x, G(z), D(w+Δw), D(w q ), D(z,w+Δw), D(z+Δz,w q ), D(z+Δz,w+Δw) inferred value

Claims

1. A computer-implemented method comprising: a first learning step of learning a first neural network having a first parameter using a first loss function as an index; a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step; Including, The regularization term is a second neural network having second parameters obtained by varying the first parameters based on the first neural network is derived, and then the second neural network is determined based on a correlation between internal features or inference values ​​of the first neural network and the second neural network; Neural network derivation method.

2. A computer-implemented method comprising: a first learning step of learning a first neural network having a first weight parameter using a first loss function as an index; a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step; Including, The regularization term is is determined based on a relationship between the first neural network and a second neural network having a second weight parameter obtained by adding a variation to the first weight parameter based on the first neural network. Neural network derivation method.

3. A computer-implemented method comprising: a first learning step of learning a first neural network having a first parameter using a first loss function as an index; a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step; Including, the regularization term is determined based on a relationship between the first neural network and a second neural network based on the first neural network; The second neural network further has a configuration in which an input of at least one layer of the first neural network is a feature quantity that is an output of a previous layer and a variation is added thereto. Neural network derivation method.

4. the regularization term is determined based on a time-series change in a similarity between a first inference value of the first neural network and a second inference value of the second neural network; 2. The method of claim 1, wherein the neural network is derived from the neural network.

5. the regularization term is determined based on a similarity between a feature amount of an output of at least one layer excluding a final layer of the first neural network and a feature amount of an output of a corresponding layer of the second neural network; 2. The method of claim 1, wherein the neural network is derived from the neural network.

6. The regularization term is determined so as to be small when the similarity is high and to be large when the similarity is low.

6. A method for deriving a neural network according to claim 4 or 5.

7. further comprising a first regularization term learning step of learning a classifier for determining the regularization term between the first learning step and the second learning step; In the first regularization term learning step, the first parameter and the second parameter are input to the classifier; the classifier is trained using a first expectation value calculated from the first inferred value and the second inferred value.

5. The method of claim 4, wherein the neural network is derived.

8. The first expectation value is For the first inferred value, always S (S>0); With respect to the second inferred value, when the second inferred value is viewed in time series, if the similarity is high with respect to the immediately preceding second inferred value, the second inferred value is set to -S, and if the similarity is not high, the second inferred value is set to 0.

8. The method of claim 7, wherein the neural network is derived.

9. In the first regularization term learning step, the classifier is learned using a third loss function having as inputs a first feature amount calculated based on the first parameter and a second feature amount calculated based on the second parameter.

9. A method for deriving a neural network according to claim 7 or 8.

10. The third loss function is a Triplet Loss function, The first feature amount obtained in the (N-1)th learning (N is an integer greater than 1) is set as a reference feature amount, The first feature amount obtained in the N-th learning is set as a positive feature amount, The second feature amount obtained in the N-th learning is set as a negative feature amount.

10. The method of claim 9, wherein the neural network is derived.

11. the first parameter is expressed in a first numerical representation; the second parameter is obtained by converting the first parameter into a second numerical representation. The method for deriving a neural network according to any one of claims 4 to 10.

12. the first numerical representation is a real number consisting of a float value, and the second numerical representation is an integer; The second parameter is obtained by quantizing the first parameter. The method for deriving a neural network according to claim 11.

13. further comprising a second regularization term learning step of learning a classifier for determining the regularization term between the first learning step and the second learning step; In the second regularization term learning step, First input data and second input data are input to the discriminator; the classifier is trained using a second expected value calculated from a third inference value of the first neural network when the first input data is input to the first neural network, and a fourth inference value of the second neural network when second input data obtained by adding a variation to the first input data is input to the second neural network. The method for deriving a neural network according to any one of claims 4 to 12.

14. The second expectation is For the third inferred value, always S (S>0); With respect to the fourth inferred value, when the fourth inferred value is viewed in time series, if the similarity is high with respect to the fourth inferred value immediately preceding it, the fourth inferred value is set to -S, and if the similarity is not high, the fourth inferred value is set to 0. The method of claim 13 .

15. In the second regularization term learning step, the classifier is learned using a fourth loss function having as input a third feature amount calculated based on the first parameter and the first input data, and a fourth feature amount calculated based on the second parameter and the second input data. A method for deriving a neural network according to claim 13 or 14.

16. The fourth loss function is a Triplet Loss function, The third feature amount obtained in the (N-1)th learning (N is an integer greater than 1) is set as a reference feature amount, The third feature amount obtained in the N-th learning is set as a positive feature amount, The fourth feature amount obtained in the N-th learning is set as a negative feature amount.

16. The method of claim 15 for deriving a neural network.

17. the first input data is expressed in a third numerical representation; the second input data is represented in a fourth numerical representation different from the third numerical representation; The method for deriving a neural network according to any one of claims 13 to 16.

18. The third numerical representation is a real number consisting of a float value, and the fourth numerical representation is an integer.

18. A method for deriving a neural network according to claim 17.

19. A computer-implemented method comprising: a first learning step of learning a first neural network to which first input data is input using a first loss function as an index; a second learning step of learning the first neural network using a second loss function obtained by adding a regularization term to the first loss function as an index after the first learning step; Including, the regularization term is determined based on a time-series change in a similarity between a fifth inferred value of the first neural network and a sixth inferred value of the second neural network after a second neural network is derived to which second input data obtained by adding a variation to the first input data based on the first neural network is input. Neural network derivation method.

20. The regularization term is determined so as to be small when the similarity is high and to be large when the similarity is low.

20. The method of claim 19, wherein the neural network is derived.

21. further comprising a third regularization term learning step of learning a classifier for determining the regularization term between the first learning step and the second learning step; In the third regularization term learning step, The first input data and the second input data are input to the classifier; the classifier is trained using a third expected value calculated from the fifth inferred value and the sixth inferred value.

21. A method for deriving a neural network according to claim 19 or 20.

22. The third expectation is For the fifth inferred value, always S (S>0); With respect to the sixth inferred value, when the sixth inferred value is viewed in time series, if the similarity is high with respect to the sixth inferred value immediately preceding the sixth inferred value, the sixth inferred value is set to -S, and if the similarity is not high, the sixth inferred value is set to 0.

22. The method of claim 21 for deriving a neural network.

23. In the third regularization term learning step, the classifier is learned using a fifth loss function having as input a fifth feature calculated based on the first input data and a sixth feature calculated based on the second input data.

23. A method for deriving a neural network according to claim 21 or 22.

24. The fifth loss function is a Triplet Loss function, The fifth feature amount obtained in the (N-1)th learning (N is an integer greater than 1) is set as a reference feature amount, The fifth feature obtained in the N-th learning is set as a positive feature, The sixth feature amount obtained in the N-th learning is set as a negative feature amount.

24. A method for deriving a neural network according to claim 23.

25. the first input data is expressed in a third numerical representation; the second input data is represented in a fourth numerical representation different from the third numerical representation; A method for deriving a neural network according to any one of claims 19 to 24.

26. The third numerical representation is a real number consisting of a float value, and the fourth numerical representation is an integer.

26. A method for deriving a neural network according to claim 25.

Citation Information

Patent Citations

  • Method and device for pattern recognition of image

    JP1996305855A

  • Transfer learning in neural networks

    JP2018525734A