Data processing device, deep learning method, and program

Abstract

To provide an information processing apparatus, an information processing method and a program that detect abnormality by excluding ignorable known abnormality in inspection of a product on a production line and the like.SOLUTION: A data processing device 1 comprises: a calculation processing unit 11 that has a multi-layer neural network; and a learning processing unit 12 that updates a parameter to make an error between output data y of training data and prediction data y* from the neural network smaller. The learning processing unit 12 creates a training data set by mixing between first noise data, in which output data having a predetermined label given by adding first noise to input data x of the training data is associated with the training data, and second noise data, in which any of output data having a predetermined label given by adding second noise to input data of the training data is randomly associated with, and repeats updating of the parameter while changing a mixture ratio of the first noise data and a mixture ratio of the second noise data in the training data set.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a data processing apparatus, a method for deep learning, and a program.

Background Art

[0002] In the field of machine learning, techniques for detecting anomalies are known (see, for example, Patent Document 1). Anomaly detection is used for inspecting products on a production line and the like. Generally, a large amount of sample data without anomalies is used for training artificial intelligence (AI). Then, for input data including anomalies, a response indicating that it is an anomaly can be expected from the AI.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Among the detected anomalies, there are not only the anomalies to be detected but also anomalies that can be ignored. For example, fine granular patterns may appear on the surface of a product. If it is already known that such patterns are not a problem in terms of standards, they can be ignored even if detected as anomalies. On the other hand, anomalies that are still unknown to the user should be detected as anomalies because it is unclear whether they meet the product standards.

[0005] To distinguish between these anomalies, it is necessary to let the AI learn various types of anomalies. However, it is difficult for the user to grasp all types of anomalies in advance.

[0006] An object of the present invention is to detect anomalies excluding known anomalies.

Means for Solving the Problems

[0007] One aspect of the present invention is a data processing device (1) that performs deep learning. The data processing device (1) includes a calculation processing unit (11) that has a multilayer neural network (30) that calculates prediction data from input data using parameters and outputs the prediction data, and a learning processing unit (12) that acquires a plurality of training data sets in which input data and output data to which predetermined labels are associated, and updates the parameters so that the error between the prediction data calculated by the neural network (30) from the input data of the training data and the output data to which the input data is associated is small. The learning processing unit (12) generates first noise data by adding a first noise, which is a known anomaly, to the input data of the training data, and associating the output data of the training data with the input data containing the first noise. The learning processing unit (12) generates second noise data by adding a second noise, which is an unknown anomaly, to the input data of the training data, and randomly associating any of the output data to which predetermined labels are assigned to the input data containing the second noise. The learning processing unit (12) generates a training dataset by mixing the training data, the first noise data, and the second noise data. The learning processing unit (12) repeatedly updates the parameters while changing the mixing ratio (T1) of the first noise data and the mixing ratio (T2) of the second noise data in the training dataset.

[0008] Another aspect of the present invention is a deep learning method for a multilayer neural network (30) that computes predictive data from input data using parameters. The method includes the steps of obtaining a plurality of training data in which input data are associated with output data to which predetermined labels are assigned, and updating the parameters such that the error between the predictive data computed by the neural network (30) from the input data of the training data and the output data associated with the input data is reduced. The updating step includes: generating first noise data by adding a first noise, which is a known anomaly, to the input data of the training data and associating the output data of the training data with the input data containing the first noise; generating second noise data by adding a second noise, which is an unknown anomaly, to the input data of the training data and randomly associating any of the output data to which a predetermined label has been assigned with the input data containing the second noise; generating a training dataset by mixing the training data, the first noise data, and the second noise data; and repeatedly updating the parameters while changing the mixing ratio (T1) of the first noise data and the mixing ratio (T2) of the second noise data in the training dataset.

[0009] Another aspect of the present invention is a program executed by a computer for performing deep learning of a multilayer neural network (30) that calculates prediction data from input data using parameters. The program causes the computer to acquire a plurality of training data sets in which input data are associated with output data to which predetermined labels are assigned, and to perform the steps of updating the parameters so that the error between the prediction data calculated by the neural network (30) from the input data of the training data and the output data to which the input data is associated is reduced. The updating steps include generating first noise data by adding a first noise, which is a known anomaly, to the input data of the training data and associating the output data of the training data with the input data containing the first noise, and generating second noise data by adding a second noise, which is an unknown anomaly, to the input data of the training data and randomly associating any of the output data to which the predetermined labels are assigned with the input data containing the second noise, The method includes the steps of generating a training dataset by mixing the training data, the first noise data, and the second noise data, and repeatedly updating the parameters while changing the mixing ratio (T1) of the first noise data and the mixing ratio (T2) of the second noise data in the training dataset. [Effects of the Invention]

[0010] According to the present invention, abnormalities can be detected while excluding known abnormalities. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram shows the configuration of a data processing device. [Figure 2] This figure shows an example of a multi-layered neural network. [Figure 3] This diagram shows the configuration of a system, which is an example of a data processing device implementation. [Figure 4] This is a flowchart of the learning process. [Figure 5] This figure shows an example of the first noise data and the second noise data. [Figure 6] This graph shows the change in the mixing ratio between the first noise data and the second noise data. [Figure 7] This diagram shows the configuration of the data processing device in a modified example. [Figure 8] This diagram shows a neural network in which dropout has been performed. [Modes for carrying out the invention]

[0012] Embodiments of the data processing device, deep learning method, and program of the present invention will be described below with reference to the drawings. The configuration described below is an example (representative example) of the present invention, and the present invention is not limited to this configuration.

[0013] Figure 1 shows the configuration of the data processing device 1 of this embodiment. The data processing unit 1 comprises a calculation processing unit 11 and a learning processing unit 12. The data processing unit 1 performs deep learning, that is, machine learning using a multi-layer neural network.

[0014] (Computation processing unit) The calculation processing unit 11 generates predicted data y from the input data x. * Calculate the predicted data y. * This is data (e.g., multiple values ​​from 0 to 1) that represents the probability of belonging to one of several pre-assigned labels. The calculation processing unit 11 has a multilayer neural network with parameters θ set, and uses this neural network to predict data y using parameters θ. * Perform the calculation.

[0015] Figure 2 shows an example of a multi-layered neural network 30. The neural network 30 includes an input layer 4a, two hidden layers 4b, and an output layer 4c. The input layer 4a, hidden layers 4b, and output layer 4c are arranged in a line in this order and connected to subsequent layers.

[0016] The neural network 30 comprises multiple nodes 31 within each layer and multiple edges 32 connecting the nodes 31 in each layer. Each node 31 is associated with a bias parameter b and an activation function fn. Each edge 32 is associated with a weight parameter wi.

[0017] In neural network 30, the following forward propagation calculation is performed. First, the input data xi is input to each node 31 of the input layer 4a. xi represents a set of i variables x1, x2, ..., xi. This input data xi is output from the input layer 4a to each node 31 of the hidden layer 4b via edges 32.

[0018] At each node 31 of the hidden layer 4b, the output data yi is calculated from the input data xi. Equations (1) and (2) below show the calculation formulas.

number

[0019] The calculated output data yi is output to each node 31 of the subsequent layer via the edge 32. At the node 31 of the subsequent layer, the same calculation as above is performed using i output data yi from each node 31 of the previous layer as input data xi. Finally, the output data yi calculated at node 31 of output layer 4c becomes the predicted data y * It will be output as follows.

[0020] The number of layers in the neural network 30 and the number of nodes 31 in each layer can be set as appropriate.

[0021] (Learning Processing Unit) The learning processing unit 12 trains the neural network 30 of the computation processing unit 11 with training data. Through this training, the parameters θ set in the neural network 30 are updated. In this embodiment, the learning processing unit 12 can perform not only online learning, which updates the parameters θ once using one training data set, but also mini-batch learning, which updates the parameters θ once using a training dataset consisting of multiple training data sets.

[0022] The learning processing unit 12 includes a set generation unit 121, an update unit 125, and a mixed control unit 126. The set generation unit 121 includes a data storage unit 122, a data generation unit 123, and a grouping unit 124.

[0023] The grouping unit 124 can acquire training data D(x,y) to be used for learning from the data storage unit 122. The grouping unit 124 can also acquire first noise data N1(x) from the data generation unit 123. N1 ,y) and second noise data N2(x N2 ,y) can be obtained. In the case of mini-batch learning, the grouping unit 124 groups these to obtain training data D(x,y) and first noise data N1(x N1 ,y) and second noise data N2(x N2 A training dataset consisting of ,y) can be generated.

[0024] The data storage unit 122 stores the training data D(x,y). The training data D(x,y) is a set of input data x and output data y to which a predetermined label has been assigned. This input data x and output data y are related to each other, in the relationship between an example problem and its solution. For example, when predicting whether someone is female or male, the features extracted from the image of a woman are used as input data x, and the data with the label "female" and a value of 1.0 is used as output data y, thus relating them to each other. Similarly, the features extracted from the image of a man are used as input data x, and the data with the label "male" and a value of 0.0 is used as output data y, thus relating them to each other.

[0025] The data generation unit 123 generates the first noise data N1(x N1 , y) and the second noise data N2(x N2 , y) from the training data D(x, y). Details thereof will be described later.

[0026] In this embodiment, the training data D(x, y) is stored, and the first noise data N1(x N1 , y) and the second noise data N2(x N2 , y) are generated, but it is not limited thereto. The training data D(x, y) may be generated, and the first noise data N1(x N1 , y) and the second noise data N2(x N2 , y) may be stored in advance.

[0027] The update unit 125 updates the parameter θ in the calculation processing unit 11 using the training data set. The update is performed by the backpropagation method. The parameter θ is, for example, the weight wi, the bias b, etc. As long as it affects the calculation result in the calculation processing unit 11, the number of layers, the number of nodes 31 in each layer, etc. can also be the parameter θ.

[0028] The mixing control unit 126 controls the mixing ratio T1(0≦T1≦1) of the first noise data N1(x N1 , y) and the mixing ratio T2(0≦T2≦1) of the second noise data N2(x N2 , y) in the training data set. If the mixing ratio of the training data D(x, y) is represented as T3, then T1 + T2 + T3 = 1 is satisfied.

[0029] In the learning processing unit 12, the mini-batch learning using the training data set is performed by the backpropagation method as follows. First, the grouping unit 124 acquires N pieces of training data D(x, y). Among this training data set, the input data x of one piece of training data D(x, y) is output to the calculation processing unit 11, and the output data y is output to the update unit 125. In the calculation processing unit 11, the predicted data y * is calculated from the input data x.

[0030] The update unit 125 processes the output data y and the predicted data y. * The objective function is determined from the error between the two. The objective function consists of the output data y, which is the target value, and the predicted data y, which is the measured value. * This function takes a value as input, calculates the difference, and outputs the evaluation value. In other words, the objective function represents the magnitude of the error between the target value and the measured value, and can serve as an indicator of whether the parameter θ is optimal.

[0031] The update unit 125 calculates the objective function for each training data point D(x,y) in the training dataset and finds its average. For example, in the case of mean squared error, the averaged objective function E is calculated using the following equation (3).

number

[0032] The update unit 125 updates the output data y and predicted data y of each training data D(x,y). * The parameter θ is updated so that the average error is minimized. Specifically, the update unit 125 updates the parameter θ so that the gradient of the objective function E expressed by equation (3) above is minimized. * The data is processed in the order from the output layer 4c to the input layer 4a, i.e., the predicted data y * The calculations are performed in the reverse order of when they were performed. For example, the updated weight wi * and bias b * This can be calculated from the weights wi and bias b before the update using the following equations (4) and (5).

number

[0033] The update unit 125 changes the current parameter θ in the calculation processing unit 11 to the θ calculated as described above. * Update to [date / time].

[0034] Thus, in mini-batch learning, the predicted data y is calculated by the neural network 30 from the input data x of each training data D(x,y). * Then, the parameter θ is updated once using backpropagation so that the average error between each input data x and its corresponding output data y is minimized.

[0035] The calculation processing unit 11 and learning processing unit 12 described above can be realized by a computer, such as a processor, reading and executing a program from a storage medium that causes the computer to perform the processing procedures described later. Semiconductor memory, magnetic disks, optical disks, etc., can be used as the recording medium for the program.

[0036] Figure 3 shows an example of a system in which the data processing device 1 can be implemented. In Figure 3, the system 20 includes a processor 21 such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), RAM (Random Access Memory) 22, a storage device 23 such as a hard disk, a communication interface 24, an input interface 25 such as a keyboard, and an output interface 26 such as a display.

[0037] The processor 21 can function as a calculation processing unit 11 and a learning processing unit 12 by reading a program from the storage device 23, loading it into the RAM 22, and executing it. The storage device 23 can also function as a data storage unit 122 for storing training data D(x,y).

[0038] The calculation processing unit 11 and the learning processing unit 12 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0039] (Learning process) In this embodiment, the learning processing unit 12 adds the first noise data N1(x) to the training data D(x,y). N1 ,y) and second noise data N2(x N2 Mini-batch learning is performed using a training dataset that mixes ,y).

[0040] Figure 4 is a flowchart of the learning process performed by the learning processing unit 12. First, in step S11, the data generation unit 123 acquires training data D(x,y) from the data storage unit 122. The data generation unit 123 uses the first noise N1 and the second noise N2 to generate the first noise data N1(x N1 ,y) and second noise data N2(x N2 Generates ,y).

[0041] Figure 5 shows the first noise data N1(x N1 ,y) and the second noise data N2(x N2 An example of generating ,y) is shown. For example, when the data processing device 1 identifies the gender of a person in an image, training data D(x1,y1) is provided, where the features of a female image are input data x1 and data labeled "female" with a value of 1.0 is output data y1. Similarly, training data D(x2,y2) is provided, where the features of a male image are input data x2 and data labeled "male" with a value of 0.0 is output data y2. The labels "female" and "male" are the correct labels to be classified.

[0042] In the case of training data D(x1,y1), the data generation unit 123 adds a first noise N1 to the input data x1, and then generates the input data x1 N1The first noise N1 is a known anomaly that has been determined by the user to be abnormal but is deemed to be an anomaly that can be ignored. For example, as the first noise N1, an image that has been previously determined by the user to be an anomaly that can be ignored can be used. The data generation unit 123 then generates this input data x1 N1 The set associated with the original output data y1 is the first noise data N1(x1 N1 Generated as y1).

[0043] Furthermore, the data generation unit 123 adds a second noise N2 to the input data x1 to create the input data x1 N2 This generates the second noise N2, which is an unknown anomaly that has not yet been determined by the user as to whether it is an anomaly or not. As the second noise N2, a perturbation that causes the neural network 30 to make an incorrect prediction can be used.

[0044] For example, since input data x1 is associated with the label "female", if the neural network 30's prediction for input data x1 is correct, then the predicted data y should be 1.0 or close to it, which would have originally been labeled "female". * The output is as follows. However, the second noise N2 is the predicted data y, which is 0.0 or close to it and labeled as "male". * The output is used to predict "male". Alternatively, the second noise N2 is a neutral prediction data y such as 0.35 to 0.65. * , in other words, predictive data y that makes it difficult to determine whether it is "female" or "male" * Change it to this.

[0045] In this way, a perturbation that causes erroneous predictions is intentionally created and stored in advance by the data generation unit 123 as a second noise N2, or the data generation unit 123 acquires the second noise N2 stored externally. As a perturbation, a uniform noise such as Gaussian noise or impulse noise can be used. Also, by image processing the input data x1 with a median filter or the like, the input data x1 including the perturbation is obtained. N2 It may be generated.

[0046] The data generation unit 123 generates input data x1 N2 Then, either output data y1 with a value of 1.0, labeled "female," or output data y2 with a value of 0.0, labeled "male," is randomly associated with the second noise data N2(x1). N2 Let ,y). In other words, the second noise data N2(x1 N2 The output data y of (y) is either y1 or y2, which is randomly selected. For example, if the input data x1 is 0.0 or 1.0 output from a random number generator or a predetermined probability function, N2 By associating them, random associations are possible.

[0047] The data generation unit 123 similarly generates the first noise data N1(x2, y2) for the training data D(x2, y2) N1 ,y2) and second noise data N2(x2 N2 It is possible to generate ,y).

[0048] Next, in step S12, the mixing control unit 126 sets the mixing ratio T2 to T2=1.0. When T2=1.0, T1=0.0 and T3=0.0.

[0049] In step S13, the grouping unit 124 adds the first noise data N1(x,y) with a mixing ratio of T1 to the training data D(x,y) with a mixing ratio of T3. N1 ,y) and the second noise data N2(x) with mixing ratio T2. N2 By mixing ,y) with the second noise data N, a training dataset of N data points is generated. When T2=1.0, the training dataset consists entirely of the second noise data N2(x N2 It consists of ,y).

[0050] In step S14, learning is performed using the training dataset. During learning, the grouping unit 124 outputs the input data x of the training data D(x,y) to the calculation processing unit 11 and the output data y to the update unit 125. The calculation processing unit 11 predicts the data y from the input data x. * Calculate the first noise data N1(x N1 ,y) and second noise data N2(xN2 The same calculation is performed for y).

[0051] The update unit 125 combines the output data y output from the grouping unit 124 with the predicted data y calculated by the calculation processing unit 11. * From there, the training data D(x,y), the first noise data N1(x N1 ,y) and second noise data N2(x N2 The objective function E for each of ,y) is calculated and averaged. The update unit 125 optimizes the parameters θ so that the averaged objective function E is minimized. * The update unit 125 calculates the parameter θ set in the calculation processing unit 11. * Update to [date / time].

[0052] Once learning is complete, in step S15, the mixing control unit 126 determines whether the mixing ratio T2 is T2 = 0.0 or not. If T2 is not 0.0 (step S15: NO), in step S16, the mixing control unit 126 decreases the mixing ratio T2 and increases the mixing ratio T1. Then, the process returns to step S13, and the first noise data N1(x) of the increased mixing ratio T1 is processed. N1 ,y) and the second noise data N2(x) of the reduced mixing ratio T2. N2 A new training dataset is generated consisting of the training data D(x,y) with a mixing ratio T3 (T3=1-(T1+T2)) and the training data D(x,y), and learning is repeated using this training dataset. If T2=0.0 (step S15:YES), that is, the second noise data N2(x N2 When y) is no longer available, the learning process is terminated.

[0053] Figure 6 shows the changes in mixing ratios T1 and T2. Second noise data N2(x N2 The mixing ratio T2 of ,y) decreases as the learning frequency, i.e., the number of parameter updates k, increases. Second noise data N2(x N2 The mixing ratio T2 of ,y) is preferably T2=0.0 at the end of training. By setting T2=0.0 at the end, the second noise data N2(x N2It is possible to converge to the same conditions as during training when y) is not used.

[0054] On the other hand, the first noise data N1(x N1 The mixing ratio T1 of ,y) increases as the number of parameter updates k increases. First noise data N1(x N1 From the viewpoint of converging to the same conditions as when y is not used, it is preferable that the mixing ratio T3 of the training data D(x,y) at the end of training is T3 ≠ 0.0, and T3 > T1. For example, the mixing ratio T1 at the end of training is about 0.3 to 0.4.

[0055] Figure 6 shows an example of mixing ratios T1 and T2 changing in a curve-like manner, but mixing ratios T1 and T2 may also change linearly. Furthermore, if mixing ratio T1 increases and mixing ratio T2 decreases by the end of the final learning cycle, mixing ratio T1 can be temporarily decreased and mixing ratio T2 can be temporarily increased during that time. The mixing control unit 126 can perform such control of mixing ratios T1 and T2 based on a table in which the increase or decrease in mixing ratios T1 and T2 is associated with the number of parameter updates.

[0056] The mixing control unit 126 can also adjust the rate at which the mixing ratio T1 is increased or the mixing ratio T2 is decreased in relation to the number of updates of the parameter θ. Such control can also be performed based on a table that pre-associates the rate of increase or decrease of the mixing ratio T1 or mixing ratio T2 with the number of updates, indicating how much to increase or decrease the mixing ratio T1 or mixing ratio T2 after a certain number of updates. For example, the rate at which the mixing ratio T2 is decreased may be scheduled to decrease with each learning session, such as to 0.9 after the first learning session and 0.7 after the second learning session. Alternatively, the mixing ratio T2 may be scheduled to remain at 1.0 after the first and second learning sessions, decrease to 0.9 after the third learning session, remain at 0.9 after the fourth learning session, and decrease to 0.5 after the fifth learning session, for example, after a certain number of learning sessions.

[0057] Rapidly changing the mixing ratio T1 or T2 with a small number of updates shortens the solution search time, but tends to narrow the search region. On the other hand, gradually changing the mixing ratio T1 or T2 over many updates lengthens the solution search time, but tends to increase the probability of converging to the global optimal solution. The mixing control unit 126 can adjust the rate at which the mixing ratio T1 or T2 is changed relative to the number of parameter θ updates so that convergence to the global optimal solution occurs in a short time.

[0058] As described above, according to this embodiment, when learning is performed by the backpropagation method, the training data D(x,y) is used to obtain the first noise data N1(x N1 ,y) and second noise data N2(x N2 The parameter θ is updated using a training dataset in which ,y) is mixed.

[0059] First noise data N1(x N1 In this model, for input data x containing a known anomaly, namely the first noise N1, output data y is associated with the same "female" or "male" label as the training data D(x,y). Therefore, even if the input data x contains an anomaly, if the anomaly is known, the neural network 30 can be trained to ignore it and predict either the "female" or "male" label.

[0060] On the other hand, the second noise data N2(x N2 In y), the input data x contains a second noise N2, which is an unknown anomaly. N2 Output data y, labeled either "female" or "male," is randomly associated with the input data x. Therefore, if the input data x contains an unknown anomaly, the neural network 30 can be trained to predict a neutral state that is neither "female" nor "male."

[0061] As a result, for inputs that do not contain unknown anomalies, a value of 1.0 or 0.0 or close to it, which are predetermined labels, is output. For example, for an image of a woman that does not contain anomalies or contains known anomalies, an output such as (female, male)=(0.9, 0.1) is obtained. Based on this output, the user can determine that the person in the image is female. On the other hand, for inputs that contain unknown anomalies, a neutral value around 0.5 is more likely to be output. For example, for an image of a woman that contains an unknown anomaly, an output such as (female, male)=(0.4, 0.6) is obtained. Since the classification of whether it is female or male is not clear, the user can determine that the image contains an unknown anomaly.

[0062] Therefore, the first noise data N1(x N1 ,y) and the second noise data N2(x N2 By (y), a neural network 30 can be provided that can detect anomalies while excluding known anomalies.

[0063] Furthermore, in this embodiment, the update of the above parameter θ is performed by the first noise data N1(x) in the training dataset. N1 Mixing ratio T1 and second noise data N2(x) N2 This process is repeated while changing the mixing ratio T2 of y).

[0064] In backpropagation, optimizing the parameter θ according to the gradient of the objective function E can sometimes lead to a local optimum rather than a global optimum. This is because, in order to find the state (solution) with the minimum error, the state is constantly updated in the direction that minimizes the gradient of the objective function E, making it impossible to escape the local optimum. To avoid this, the simulated annealing method is known.

[0065] Pseudo-simulated annealing allows for updates to states where the error increases. The parameter that controls the degree to which updates to states where the error increases are allowed is called "temperature." By gradually cooling from a "high temperature" state where the error increases to a "low temperature" state where the error increases are not allowed, it is possible to reach a global optimum without falling into a local optimum.

[0066] Unlike the training data D(x,y), the second noise data N2(x) contains unknown anomalies. N2 y) allows updating to a state where the error increases. Also, the larger the mixing ratio T2, the more likely the error is to increase. Therefore, the mixing ratio T2 corresponds to the temperature parameter in pseudo-annealing, and the learning process of this embodiment, which repeats learning while changing the mixing ratio T2, realizes pseudo-annealing in the backpropagation method.

[0067] In conventional pseudo-annealing methods, the objective function E is calculated using training data at an initial temperature T(0), then the parameter θ is changed, and the objective function E is calculated again using the training data. If the error ΔE of the objective function E decreases before and after the change in parameter θ, the change is accepted; if the error ΔE increases, the change is accepted with a certain probability p; otherwise, the change is canceled.

[0068] For example, the probability p can be defined using the sigmoid function as follows, where n is the current number of repetitions, T(n) is the temperature, and ΔE is the increase in error.

number

[0069] Thus, implementing conventional pseudo-simulated annealing methods requires the addition of a configuration to calculate the error ΔE of the objective function E and determine whether or not to accept the change in parameter θ based on its change. This increases the computational complexity, making it difficult to implement in systems when used in conjunction with backpropagation.

[0070] However, according to this embodiment, since only the mixing ratio T2 needs to be adjusted as a temperature parameter, no special calculations or judgments are required. Pseudo-simulated annealing can be easily performed in the backpropagation method, making it easy to implement in a system. Therefore, a data processing device 1 that can easily reach a global optimal solution and is easy to implement can be provided.

[0071] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. It can be modified as appropriate within the scope of the present invention.

[0072] [Variation] In the above embodiment, dropout may be further incorporated.

[0073] Figure 7 shows the configuration of the data processing device 1A in a modified example. The data processing device 1A has the same configuration as the data processing device 1 described above, except that it further includes a dropout control unit 127.

[0074] The dropout control unit 127 performs dropout on the neural network 30. Dropout is a process that disables some of the nodes 31 in the hidden layer 4b. The disabled nodes 31 do not perform calculations on the input data x, and therefore produce no output.

[0075] Figure 8 shows the neural network 30 when dropout is incorporated. In Figure 8, the black node 31 is a node where dropout has been performed. In the example in Figure 8, the dropout rates for the two hidden layer 4b nodes 31 are 1 / 2 and 1 / 3, respectively.

[0076] The dropout rate refers to the percentage of nodes 31 that have been dropped in a single layer. The dropout rate can be set appropriately depending on the objective. A higher dropout rate tends to improve the generalization performance of the neural network 30. Therefore, it becomes easier to obtain optimal solutions not only for the learned training data but also for unknown data. On the other hand, a lower dropout rate tends to yield solutions that are as learned.

[0077] In the modified example, the learning processing unit 12 performs the same learning process as in the above embodiment after the dropout control unit 127 performs dropout of the computation processing unit 11. As a result, with some nodes 31 dropped out, the first noise data N1(x) in the training dataset is processed. N1 Increase the mixing ratio T1 of ,y) and the second noise data N2(x N2 Mini-batch learning is repeated while decreasing the mixing ratio T2 of ,y).

[0078] By using pseudo-simulated annealing with the mixing ratio T as a temperature parameter, the first noise data N1(x) is added to the training data D(x,y). N1 The first noise data N1(x) can also be used for training, increasing the number of data points N that the neural network 30 learns from. However, the first noise data N1(x) N1 Since y) differs from the original training data D(x,y) by a small amount of variation, depending on the degree of pseudo-annealing, it may output neutral prediction data for input data other than the training data D(x,y), i.e., a prediction result that is not any solution. In this case, it is the same as creating a dictionary of the training data D(x,y), and the generality (also called generalization ability) of the neural network 30, which outputs the optimal solution even for unknown data, cannot be guaranteed.

[0079] In contrast, dropout disables the computation of some nodes 31, thus preventing some features of the input data from being overvalued. Therefore, by further combining dropout with the training process that performs pseudo-simulated annealing, the predicted data y of the optimal solution can be obtained even if some of the input data is missing. * The parameter θ is updated so that the first noise data N1(x N1 Using y) for training makes it easier to obtain the optimal solution for classifying into the given label. This reduces the number of neutral outputs like those described above, thereby increasing the generality of the neural network 30.

[0080] The present invention can also be applied to neural networks used in Generative Adversarial Networks (GANs). Generally, GANs use a neural network called a Generator and a neural network called a Discriminator. The Generator generates false data from given data, and the Discriminator uses real data to determine whether the generated false data is true or false. Based on the determination result, the Generator generates false data that is closer to the real data. The present invention can also be applied to Generators and Discriminators that learn in a competitive manner in this way. [Explanation of Symbols]

[0081] 1...Data processing unit, 11...Calculation processing unit, 12...Learning processing unit, 121...Set generation unit, 122...Data storage unit, 123...Data generation unit, 124...Grouping unit, 125...Update unit, 126...Mixing control unit, 127...Dropout control unit

Claims

1. In a data processing device (1) that performs deep learning, It has a multilayer neural network (30) that calculates prediction data from input data using parameters, and a calculation processing unit (11) that outputs the prediction data, The system includes: acquiring a plurality of training data sets in which input data and output data to which predetermined labels are assigned; and a learning processing unit (12) that updates the parameters so as to reduce the error between the predicted data calculated by the neural network (30) from the input data of the training data and the output data associated with the input data. The learning processing unit (12) is: By adding a first noise, which is an anomaly determined in advance by the user to be negligible, to the input data of the training data, and associating the output data of the training data with the input data containing the first noise, a first noise data is generated. Second noise data is generated by adding a second noise, which is an anomaly not yet determined by the user, to the input data of the training data, and randomly associating one of the output data to which a predetermined label has been assigned to the input data containing the second noise. A training dataset is generated by mixing the training data, the first noise data, and the second noise data. The parameters are updated repeatedly while changing the mixing ratio (T1) of the first noise data and the mixing ratio (T2) of the second noise data in the training dataset. Data processing device (1).

2. The learning processing unit (12) increases the mixing ratio (T1) of the first noise data as the number of parameter updates increases. The data processing device (1) according to claim 1.

3. The learning processing unit (12) reduces the mixing ratio (T2) of the second noise data as the number of parameter updates increases. The data processing device (1) according to claim 1 or 2.

4. The learning processing unit (12) performs dropout of the neural network (30). A data processing device (1) according to any one of claims 1 to 3.

5. A method executed by a data processing device (1) that performs deep learning of a multilayer neural network (30) that calculates predicted data from input data using parameters, The steps include: obtaining a plurality of training data sets in which input data and output data to which predetermined labels are assigned; updating the parameters so that the error between the predicted data calculated by the neural network (30) from the input data of the training data and the output data associated with the input data is reduced; The aforementioned update step is: The steps include: adding a first noise, which is an anomaly determined in advance by the user to be negligible, to the input data of the training data, and associating the output data of the training data with the input data containing the first noise to generate first noise data; The steps include: adding a second noise, which is an anomaly not yet determined by the user, to the input data of the training data, and randomly associating one of the output data to which a predetermined label has been assigned with the input data containing the second noise, thereby generating second noise data; The steps include generating a training dataset by mixing the training data, the first noise data, and the second noise data, The step includes repeatedly updating the parameters while changing the mixing ratio (T1) of the first noise data and the mixing ratio (T2) of the second noise data in the training dataset. method.

6. A program executed by a computer to perform deep learning on a multilayer neural network (30) that calculates prediction data from input data using parameters, The program is sent to the computer, The process involves obtaining a plurality of training data sets, each associated with an input data and an output data set to which a predetermined label has been assigned; and updating the parameters so that the error between the prediction data calculated by the neural network (30) from the input data of the training data and the output data associated with the input data is reduced. The aforementioned update step is: The steps include: adding a first noise, which is an anomaly determined in advance by the user to be negligible, to the input data of the training data, and associating the output data of the training data with the input data containing the first noise to generate first noise data; The steps include: adding a second noise, which is an anomaly not yet determined by the user, to the input data of the training data, and randomly associating one of the output data to which a predetermined label has been assigned with the input data containing the second noise, thereby generating second noise data; The steps include generating a training dataset by mixing the training data, the first noise data, and the second noise data, The step includes repeatedly updating the parameters while changing the mixing ratio (T1) of the first noise data and the mixing ratio (T2) of the second noise data in the training dataset. program.

Images