Neural network learning device and neural network learning method

Abstract

This neural network learning device comprises: an initialization unit that sets an initial value of a model parameter θ and an initial value of a score S corresponding to a weight W included in the model parameter θ, and generates an initial value of a binary mask M on the basis of the initial value of the score S; a a model calculation unit that generates a model value f(x;θ(init)*M ](t)) using learning data; a loss function calculation unit that calculates the value of a loss function using the model value f(x;θ(init)*M(t); a score update unit that calculates a score S(t+1) when the number of instances of learning is t+1 from a score S(t) when the number of instances of learning is t by using the value of the loss function; a binary mask generation unit that generates a binary mask M(t+1) using the score S(t+1); and an end condition determination unit that outputs the model parameter θ(init)*M(t+1) as a model parameter obtained through learning when an end condition is satisfied.

Description

Neural network learning device and neural network learning method

[0001] The present invention relates to deep learning technology.

[0002] Models obtained through deep learning store training data. Therefore, deep learning poses the risk of training data being leaked from model information (model parameters). Attacks that reconstruct training data from model parameters are called data reconstruction attacks, and one example is a data reconstruction attack method based on the stationary point condition of continuous optimization. This method poses a widespread threat to neural networks that continuously optimize model parameters.

[0003] Techniques to prevent data reconstruction against such threats have been researched, such as DP-SGD (Differentially Private Stochastic Gradient Descent) described in Non-Patent Document 1. In DP-SGD, model parameters are optimized by adding noise to the gradient of the loss function to guarantee differential privacy.

[0004] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang, “Deep Learning with Differential Privacy,” In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS '16), 2016.

[0005] However, DP-SGD has the problem that the accuracy of the model obtained by training decreases because noise is added during the continuous optimization process in model training.

[0006] Therefore, the present invention aims to provide a deep learning technology that can learn a model in a manner that reduces the risk of data leakage due to data reconstruction attacks while preventing a decrease in accuracy.

[0007] One aspect of the present invention is a neural network learning device that learns a model f(x;θ) (where x is the input of the neural network and θ is a model parameter) of a neural network to be learned, and includes an initialization unit that sets an initial value of the model parameter θ and an initial value of a score S corresponding to a weight W included in the model parameter θ and generates an initial value of a binary mask M based on the initial value of the score S; and an initialization unit that uses learning data to calculate the value of the model f(x;θ) (init) *M (t) ) (where θ (init) is the initial value of the model parameter θ, M (t) is the binary mask when the number of learning times is t, * is the product of elements), and the model calculation part calculates the value of the model f(x;θ (init) *M (t) ) and a loss function calculation unit that calculates the value of the loss function using the value of the loss function when the number of learning times is t. (t) The score S when the number of learning times is t+1 (t+1) a score updater that calculates the score S (t+1) Using the binary mask M (t+1) and a binary mask generator that generates the model parameters θ (init) *M (t+1) The size of the model parameter θ is large enough to be considered as a strong lottery hypothesis.

[0008] According to the present invention, it is possible to train a model in a manner that reduces the risk of data leakage due to data reconstruction attacks while preventing a decrease in accuracy.

[0009] It is a block diagram showing the configuration of the neural network learning device 100. It is a flowchart showing the operation of the neural network learning device 100. It is a diagram showing an example of the functional configuration of a computer that realizes each device in the embodiment of the present invention.

[0010] Hereinafter, an embodiment of the present invention will be described in detail. Note that components having the same functions are given the same numbers and redundant explanations will be omitted.

[0011] Before describing each embodiment, the notation used in this specification will be explained.

[0012] ^ (caret) represents a superscript, e.g., x y^z Yes z is a superscript to x, and x y^z Yes z is a subscript to x. Also, _ (underscore) represents a subscript. For example, x y_z Yes z is a superscript to x, and x y_z Yes z is a subscript to x.

[0013] The superscripts "^" and "~" such as ^x and ~x for a certain letter x should be written directly above the "x", but due to restrictions on the notation in the specification, they are written as ^x and ~x.

[0014] <Technical Background> In an embodiment of the present invention, instead of continuously changing model parameters for learning (i.e., continuous optimization), the model parameters are fixed and the connection structure between neurons in the neural network is changed for learning. In this case, the Strong Lottery Ticket Hypothesis is assumed. Here, the Strong Lottery Ticket Hypothesis is a hypothesis that a neural network in which model parameters are randomly initialized has a subnetwork with accuracy equivalent to that obtained when the model parameters are continuously optimized. Therefore, model learning in an embodiment of the present invention involves fixing the model parameters to randomly selected initial values ​​and then pruning the connections between neurons (called pruning optimization) to obtain a subnetwork with accuracy equivalent to that obtained when the model parameters are continuously optimized.

[0015] Below, we explain pruning optimization. f(x;θ) (where x is the input of the neural network, θ=(W i , b i ) i=1 L (However, W i , b i (i=1, …, L, L is the number of layers that make up the neural network) are the weights of the i-th layer, and the biases of the i-th layer are the model parameters) are the model of the neural network to be trained. Note that the input x, weights W i , bias b i are real vectors, real matrices, and real vectors, respectively. The neural network to be trained can be a fully connected layer in which all neurons in the i-th layer are connected to all neurons in the i+1-th layer.

[0016] In continuous optimization, the model value f(x;θ), which is the output of the neural network, is calculated using, for example, the activation function σ, as follows: f(x;θ)=σ(W L (W L-1 (… (W2 (W1x+b1) +b2) …) +b L-1 ) +b L ) is calculated. On the other hand, in pruning optimization, θ (init) =(W i (init) , b i (init) ) i=1 L (However, W i (init) , b i (init) (i=1, …, L) are the initial values ​​of the weights of the i-th layer, and the initial values ​​of the biases of the i-th layer) are the initial values ​​of the model parameters θ, and M=(M i ) i=1 L (However, M i (i=1, …, L) is a matrix where all elements are either 0 or 1) is a binary mask, and the output of the neural network, f(x;θ (init) *M) (where θ (init) *M= (W i (init) *M i , bi (init) ) i=1 L , * is the element product (Hadamard product), for example, f(x;θ (init) *M)=σ(W L (init) *M L (W L-1 (init) *M L-1 (… (W2 (init) *M2 (W1 (init) *M1x+b1 (init) ) +b2 (init) ) …) +b L-1 (init) ) +b L (init) ) where M i is W i is a matrix of the same type as the binary mask M i The elements of 0 mean that the connections between neurons are pruned.

[0017] In continuous optimization, for example, the weights W = (W i ) i=1 L The model parameter θ is optimized by sequentially updating the weights W (init) *M=(W i (init) *M i ) i=1 L The binary mask M contained in (init) *M, where the binary mask M i Since the elements of must be either 0 or 1, their values ​​cannot be changed continuously. i The score S corresponding to i Introduce S i is W i In pruning optimization, a binary mask M=(M i ) i=1 L Instead of score S = (Si ) i=1 L is continuously changed and updated. Then, the binary mask M is updated by determining the elements of the binary mask M whose value will be set to 1 based on the value of the score S. As a method for determining the elements of the binary mask M whose value will be set to 1 based on the value of the score S, for example, top-k or k-heavy hitters can be used. Here, top-k is a method in which, among the elements of the binary mask M, elements whose corresponding score values ​​are in the top k percent are set to 1, and the rest are set to 0. Note that the absolute value of the score may be used instead of the score value. Furthermore, k-heavy hitters is a method in which, among the elements of the binary mask M, elements whose corresponding score values ​​are greater than 1 / k times the average score are set to 1, and the rest are set to 0.

[0018] Pruning optimization does not add noise during learning, so the model can be trained without a loss of accuracy. Furthermore, pruning optimization uses a binary mask to discretely optimize the neuron connection structure. Therefore, it is robust against data reconstruction attacks and can reduce the risk of data leakage.

[0019] First Embodiment A neural network training device 100 trains a neural network model f(x;θ) (where x is the input of the neural network and θ is a model parameter) of the neural network to be trained. Here, the size of the model parameter θ (i.e., the number of neurons included in the neural network to be trained) is assumed to be sufficiently large so that a strong lottery hypothesis can be considered to hold.

[0020] The neural network training device 100 will be described below with reference to FIGS. 1 and 2. FIG. 1 is a block diagram showing the configuration of the neural network training device 100. FIG. 2 is a flowchart showing the operation of the neural network training device 100. As shown in FIG. 1, the neural network training device 100 includes an initialization unit 110, a model calculation unit 120, a loss function calculation unit 130, a score update unit 140, a binary mask generation unit 150, a termination condition determination unit 160, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for the processing of the neural network training device 100. The recording unit 190 records, for example, training data in advance.

[0021] The operation of the neural network learning device 100 will be described with reference to FIG.

[0022] In S110, the initialization unit 110 initializes the model parameter θ=(W i , b i ) i=1 L (However, W i , b i (i=1, …, L, L is the number of layers in the neural network) are the weight of the i-th layer, the bias of the i-th layer, and the weight W=(W i ) i=1 L The score S=(S i ) i=1 L The initialization unit 110 initializes the initial values ​​θ of the model parameters using, for example, random numbers. (init) =(W i (init) , b i (init) ) i=1 L , the initial score S (init) =(S i (init) ) i=1 L The initialization unit 110 sets the initial value S of the score. (init) =(S i (init) ) i=1 L Based on the binary mask M = (M i )i=1 L The initialization unit 110 generates an initial value M of the binary mask using, for example, the top-k or k-heavy hitters described in the <Technical Background> section. (init) =(M i (init) ) i=1 L Furthermore, the initialization unit 110 initializes a counter t that indicates the number of times of learning. That is, the initialization unit 110 sets the initial value of the counter t to 0. Hereinafter, the score when the number of times of learning is t and the binary mask when the number of times of learning is t are respectively referred to as S (t) , M (t) Therefore, S (0) =S (init) , M (0) =M (init) is.

[0023] In S120, the model calculation unit 120 calculates the model value f(x;θ) by forward propagation using the learning data previously recorded in the recording unit 190. (init) *M (t) ) is calculated.

[0024] In S130, the loss function calculation unit 130 calculates the value f(x;θ) of the model calculated in S120. (init) *M (t) ) to calculate the value of the loss function.

[0025] In S140, the score update unit 140 updates the score S by backpropagation using the value of the loss function calculated in S130. That is, the score S (t) Score S from (t+1) The score update unit 140 can use, for example, SGD as the backpropagation.

[0026] In S150, the binary mask generation unit 150 calculates the score S (t+1) Using the binary mask M (t+1) Similarly to the initialization unit 110, the binary mask generation unit 150 generates a binary mask M (t+1) Generate.

[0027] Similarly to the initialization unit 110, the binary mask generation unit 150 generates the binary mask M (t+1) The value of the score element corresponding to the binary mask element having a value of 0 may be initialized.

[0028] In S160, the termination condition determination unit 160 determines whether a predetermined termination condition is satisfied, and if the condition is satisfied, the model parameter θ (init) *M (t+1) and terminates the process, otherwise the value of the counter t is incremented by 1 and the process returns to step S120. Here, the termination condition can be, for example, that the counter t is equal to or greater than T, where T is the upper limit of the number of learning times.

[0029] According to an embodiment of the present invention, it is possible to train a model in a manner that reduces the risk of data leakage due to data reconstruction attacks while preventing a decrease in accuracy. Since the initial values ​​of the model parameters discretized by the binary mask are obtained as model parameters obtained through training, it becomes difficult to restore the training data from the model parameters. Therefore, it becomes possible to operate a service in which the model parameters obtained through training are made public.

[0030] <Additional Notes> The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0031] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0032] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0033] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 3, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0034] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0035] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0036] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its storage device and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of the server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0037] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0038] The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed.

Claims

1. A neural network learning device for learning a neural network model f(x;θ) (where x is the input of the neural network and θ is a model parameter) to be learned, comprising: an initialization unit that sets an initial value of the model parameter θ and an initial value of the score S corresponding to the weight W included in the model parameter θ, and generates an initial value of the binary mask M based on the initial value of the score S; and an initialization unit that uses learning data to set the model value f(x;θ) (init) *M (t) ) (where θ (init) is the initial value of the model parameter θ, M (t) is the binary mask when the number of learning times is t, * is the product of elements), and the model calculation part calculates the value of the model f(x;θ (init) *M (t) ) and a loss function calculation unit that calculates the value of a loss function using the value of the loss function, and a score S when the number of learning times is t using the value of the loss function. (t) The score S when the number of learning times is t+1 (t+1) a score updater that calculates the score S (t+1) Using the binary mask M (t+1) and a binary mask generator that generates the model parameters θ when the termination condition is met. (init) *M (t+1) as a model parameter obtained by learning, and a termination condition determination unit that outputs the model parameter θ, wherein the size of the model parameter θ is large enough that a strong lottery hypothesis can be considered to hold.

2. A neural network training method in which a neural network training device trains a model f(x;θ) (where x is the input of the neural network and θ is a model parameter) of a neural network to be trained, comprising: an initialization step in which the neural network training device sets an initial value of the model parameter θ and an initial value of the score S corresponding to the weight W included in the model parameter θ, and generates an initial value of the binary mask M based on the initial value of the score S; and the neural network training device uses training data to calculate the value of the model f(x;θ) (init) *M (t) ) (where θ (init) is the initial value of the model parameter θ, M (t) is a binary mask when the number of learning times is t, and * is an element product), and (init) *M (t) a loss function calculation step of calculating a value of a loss function using the loss function; and a score S when the number of learning times is t, using the value of the loss function. (t) The score S when the number of learning times is t+1 (t+1) a score update step of calculating a score S (t+1) Using the binary mask M (t+1) a binary mask generation step of generating a binary mask; and when a termination condition is satisfied, the neural network learning device generates a model parameter θ (init) *M (t+1) and a termination condition determination step of outputting the model parameters θ obtained by learning as the model parameters θ, wherein the size of the model parameters θ is large enough to be considered as a strong lottery hypothesis.

Images