DATA PROCESSING DEVICE, METHOD AND PROGRAM FOR DEEP LEARNING OF NEURAL NET

Abstract

To provide an intended characteristic to a neural network in advance.SOLUTION: The present invention relates to a data processor (1D) for performing deep learning of a neural network (30). The data processor (1D) includes: a calculation processing unit (11) having a neural network (30) for calculating prediction data from input data by using a parameter and outputting the prediction data; and a learning processing unit (12D) for executing learning processing of updating the parameter by using training data as a pair of input data and output data related to each other in advance. The learning processing unit (12D) executes pre-processing of providing a characteristic to the prediction model before the learning processing.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to a data processing device, a method and a program for deep learning of a neural network. [Background technology]

[0002] Conventionally, neural networks are known as one type of predictive model. Neural networks learn combinations of input data and output data, i.e., training data. In order to output optimal solutions not only for training data but also for unknown data, the parameters used to calculate output data in the predictive model are optimized through learning.

[0003] The backpropagation method is known as a parameter optimization technique. In the backpropagation method, parameters may fall into a local solution rather than a global optimum. To avoid this, the backpropagation method may be combined with dropout scheduling to perform simulated annealing (see, for example, Patent Document 1). Also, the Monte Carlo method may be used instead of the backpropagation method (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Pat. No. 1,038,0484 [Patent Document 2] Patent No. 3178884 Summary of the Invention [Problem to be solved by the invention]

[0005] The parameters are updated based on the given training data to extract and classify the features of the input data and derive the optimal solution. Usually, the parameters are initialized to an arbitrary value such as 0 or a random number, and then the parameter update starts.

[0006] In this way, since a prediction model is constructed by learning from an initialization state in which arbitrary parameters are set, the characteristics of the constructed prediction model are imparted by the training data. In general, training data sampled in advance is used for learning, but the data is not necessarily sampled with the same weight, and it may not be possible to impart the desired characteristics to the prediction model.

[0007] In addition, it is difficult to clarify the principles of how predictive models such as neural networks classify input data while calculating output data from the input data.It is also difficult to design the learning content because it is unclear what kind of learning can give the predictive model the desired characteristics.

[0008] An object of the present invention is to provide a neural network with desired characteristics in advance. [Means for solving the problem]

[0009] One aspect of the present invention is a data processing device (1D) that performs deep learning of a neural network. The data processing device (1D) includes a neural network (30) that calculates predicted data from input data using parameters, a calculation processing unit (11) that outputs the predicted data, and a learning processing unit (12D) that executes a learning process that updates the parameters using training data that is a pair of input data and output data that are previously associated. The learning processing unit (12D) performs pre-processing to give characteristics to the neural network (30) before the learning process.

[0010] Another aspect of the present invention is a method for deep learning of a neural network (30) that calculates prediction data from input data using parameters, the method including the steps of: performing a learning process to update the parameters using training data that is a pair of input data and output data that are previously associated with each other; and performing a pre-processing step, prior to the learning process, to give characteristics to the neural network (30).

[0011] Another aspect of the present invention is a program executed by a computer to perform deep learning of a neural network (30) that calculates prediction data from input data using parameters. The program causes the computer to execute a step of performing a learning process to update the parameters using training data, which is a pair of input data and output data that are previously associated with each other, and a step of performing a pre-processing to give characteristics to the neural network (30) before the learning process. Effect of the Invention

[0012] According to the present invention, it is possible to give a neural network a desired characteristic in advance. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a data processing device according to a first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a multi-layer neural network. [Diagram 3] FIG. 1 is a diagram illustrating a system configuration that is an implementation example of a data processing device. [Figure 4] 4 is a flowchart of a learning process in the first embodiment. [Diagram 5] FIG. 11 is a diagram illustrating a configuration of a data processing device according to a second embodiment. [Figure 6] FIG. 1 illustrates a neural network in which dropout has been implemented. [Figure 7] FIG. 13 is a diagram illustrating a configuration of a data processing device according to a third embodiment. [Figure 8] 13 is a flowchart of a learning process in the third embodiment. [Figure 9] FIG. 13 is a diagram conceptually illustrating the complexity of the output boundary. [Figure 10] 13 is a flowchart of a learning process in the fourth embodiment. [Figure 11] FIG. 13 is a diagram illustrating a configuration of a data processing device according to a fifth embodiment. [Figure 12] 13 is a flowchart of a pre-processing according to the fifth embodiment. [Figure 13] 23 is a flowchart of a pre-processing according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the data processing device, the neural network deep learning method, and the program according to the present invention will be described with reference to the drawings. The configuration described below is an example (representative example) of the present invention, and the present invention is not limited to this configuration.

[0015] [First embodiment] FIG. 1 shows the configuration of a data processing device 1A according to a first embodiment of the present invention. The data processing device 1A includes a calculation processing unit 11 and a learning processing unit 12A. The data processing device 1A performs deep learning of a neural network. Deep learning refers to machine learning of a multi-layer neural network.

[0016] (Computation processing unit) The calculation processing unit 11 calculates predicted data y from input data x. * Calculate the predicted data y * is data (e.g., multiple values ​​from 0 to 1) that represent the probability that the solution belongs to one of a number of solutions. The calculation processing unit 11 has a multi-layered neural network in which a parameter θ is set, and calculates predicted data y * The following calculation is performed.

[0017] FIG. 2 shows an example of a multi-layer neural network 30. The neural network 30 includes an input layer 4a, two hidden layers 4b, and an output layer 4c. The input layer 4a, the hidden layer 4b, and the output layer 4c are arranged in a line in this order and connected to the subsequent layers.

[0018] The neural network 30 includes a plurality of nodes 31 in each layer and a plurality of edges 32 connecting the nodes 31 in each layer. Each node 31 is associated with a bias b parameter and an activation function fn. Each edge 32 is associated with a weight wi parameter.

[0019] In the neural network 30, the following forward propagation calculation is performed. First, input data x i is input to each node 31 of the input layer 4a. x i means a set of i variables x1, x2, ..., x i. This input data x i is output from the input layer 4a to each node 31 of the hidden layer 4b via edges 32.

[0020] In each node 31 of the hidden layer 4b, output data yi is calculated from input data xi. The following formulas (1) and (2) show the calculation formulas.

number

[0021] The calculated output data yi is output to each node 31 in the subsequent layer via an edge 32. In the subsequent layer node 31, the same calculation as above is performed using the i output data yi from each node 31 in the previous layer as input data xi. Finally, the output data yi calculated in the node 31 of the output layer 4c is converted into the predicted data y * will be output as:

[0022] The number of layers in the neural network 30 and the number of nodes 31 in each layer can be set appropriately.

[0023] (Learning processing unit) The learning processing unit 12A causes the neural network 30 of the calculation processing unit 11 to learn the training data. The parameter θ set in the neural network 30 is updated by the learning. The learning processing unit 12A of this embodiment can perform not only online learning in which the parameter θ is updated once using one piece of training data, but also mini-batch learning in which the parameter θ is updated once using a training data group consisting of a plurality of training data.

[0024] The learning processing unit 12A includes an acquisition unit 121, a data storage unit 122, a data generation unit 123, an update unit 124, and a mixing control unit 125.

[0025] The acquiring unit 121 acquires training data D(x, y) to be used for learning from the data storage unit 122. In the case of mini-batch learning, the acquiring unit 121 can generate a training data group consisting of a plurality of training data D(x, y).

[0026] The acquiring unit 121 can also acquire noise data Dn(x, y) from the data generating unit 123 and generate a training data group by mixing the noise data Dn(x, y) with the training data D(x, y).

[0027] The data storage unit 122 stores training data D(x,y). The training data D(x,y) is a pair of input data x and output data y that are associated in advance. In the training data D(x,y), the input data x and the output data y have a relationship of example problem and solution.

[0028] The data generating unit 123 generates noise data Dn(x,y). The noise data Dn(x,y) is a pair of input data x and output data y that is different from the training data D(x,y). The input data x and output data y of the noise data Dn(x,y) are in a mutually independent relationship, regardless of whether they are in a relationship of example problem and solution like the training data D(x,y).

[0029] Specifically, the input data x of the noise data Dn(x,y) is arbitrarily extracted from a data group existing in the same data space as the input data x of the training data D(x,y). In addition to the extraction of this input data x, the output data y of the noise data Dn(x,y) is arbitrarily extracted from a data group existing in the same data space as the output data y of the training data D(x,y). The noise data Dn(x,y) is obtained by excluding the combination of the input data x and output data y of the training data D(x,y) from the combinations of the input data x and output data y that are independently extracted.

[0030] In the training data D(x,y), the input data x and the output data y are associated with each other, with the relationship between the example and the solution being recognized in advance. For example, when predicting whether it is a cat or a dog, the features extracted from a photo of a cat are associated as the input data x, and the data representing a cat (for example, data in which cat is 0 and dog is 1) are associated as the output data y. On the other hand, in the noise data Dn(x,y), the input data x and the output data y extracted independently are combined without the relationship between cat and dog being recognized. Therefore, the noise data Dn(x,y) may include a combination of the input data x having the features of a cat and the output data y representing a cat, which is not recognized in advance. In other words, in the noise data Dn(x,y), there are combinations of the input data x and the output data y that are unintentionally in the relationship between the example and the solution, and there are also combinations that are not in the relationship.

[0031] Examples of noise data Dn(x,y) include a combination of any input data and output data, such as a random bit string output by a random bit encoder, or a combination of input data x and output data y that are any distance away in any direction from the training data D(x,y).

[0032] In this embodiment, the training data D(x,y) is stored and the noise data Dn(x,y) is generated, but this is not limited to this. The training data D(x,y) may be generated and the noise data Dn(x,y) may be stored in advance.

[0033] The update unit 124 updates the parameter θ in the calculation processing unit 11 by using the training data group. The update is performed by the backpropagation method. The parameter θ is, for example, a weight wi and a bias b. The number of layers, the number of nodes 31 in each layer, etc. can also be the parameter θ if they affect the calculation result in the calculation processing unit 11.

[0034] The mixing control unit 125 controls the mixing ratio T (T=0 to 1) of the noise data Dn(x, y) in the training data group.

[0035] In the learning processing unit 12A, mini-batch learning using a group of training data is performed by the error backpropagation method as follows. First, the acquisition unit 121 acquires N pieces of training data D(x, y). Among the training data group, input data x of one piece of training data D(x, y) is output to the calculation processing unit 11, and output data y is output to the update unit 124. The calculation processing unit 11 calculates predicted data y from the input data x. * is calculated.

[0036] The update unit 124 updates the output data y and the predicted data y * The objective function is calculated from the error between the output data y, which is the target value, and the predicted data y, which is the actual measured value. * This is a function that inputs, calculates and outputs an evaluation value of the difference. In other words, the objective function represents the magnitude of the error between the target value and the actual measurement value, and can be an index of whether the parameter θ is optimal.

[0037] The update unit 124 calculates an objective function for each training data D(x, y) in the training data group, and calculates the average of the objective functions. For example, in the case of the mean square error, the average objective function E is calculated by the following formula (3).

number

[0038] The update unit 124 updates the output data y and the predicted data y * Specifically, the update unit 124 updates the updated parameter θ so that the gradient of the objective function E expressed by the above formula (3) is minimized. * In the order from the output layer 4c to the input layer 4a, that is, the predicted data y * For example, the updated weights wi * and bias b * can be calculated from the weights wi and bias b before the update using the following equations (4) and (5).

number

[0039] The update unit 124 updates the current parameter θ in the calculation processing unit 11 with the θ calculated as above. * to update.

[0040] In this way, in mini-batch learning, predicted data y calculated by the neural network 30 from input data x of each training data D(x,y) is * Then, the parameter θ is updated once by the backpropagation method so that the average error between each input data x and the paired output data y becomes small.

[0041] The calculation processing unit 11 and the learning processing unit 12A can be realized by a computer such as a processor reading a program for causing the computer to execute the processing procedure described below from a storage medium and executing the program. As the storage medium for the program, a semiconductor memory, a magnetic disk, an optical disk, etc. can be used.

[0042] FIG. 3 shows an example of a system in which data processing device 1A can be implemented. In FIG. 3, a system 20 includes a processor 21 such as a CPU (Central Processing Unit), a RAM 22, a storage device 23 such as a hard disk, a communication interface 24, an input interface 25 such as a keyboard, and an output interface 26 such as a display.

[0043] The processor 21 can function as the calculation processing unit 11 and the learning processing unit 12A by reading a program from the storage device 23, expanding it in the RAM 22, and executing it. The storage device 23 can also function as a data storage unit 122 that stores the training data D(x, y).

[0044] Without being limited to the above, the calculation processing unit 11 and the learning processing unit 12A may be realized by hardware such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0045] (Learning process) In this embodiment, the learning processing unit 12A performs mini-batch learning using a training data group in which noise data Dn(x, y) is mixed with training data D(x, y).

[0046] FIG. 4 is a flowchart of the learning process by the learning processing unit 12A. First, in step S11, the mixing control unit 125 sets the mixing ratio T to T=1.

[0047] In step S12, the acquisition unit 121 generates a training data group with the number of data items N by mixing the training data D(x, y) with noise data Dn(x, y) having a mixing ratio T. That is, the training data group is a mixture of the training data D(x, y) having a mixing ratio (1-T) acquired from the data storage unit 122 and the noise data Dn(x, y) having a mixing ratio T acquired from the data generation unit 123. When T=1, the training data group is entirely made up of noise data Dn(x, y).

[0048] In step S13, learning is performed using a group of training data. During learning, the acquisition unit 121 outputs input data x of the training data D(x, y) to the calculation processing unit 11, and outputs output data y to the update unit 124. The calculation processing unit 11 derives predicted data y from the input data x. * Calculate.

[0049] The update unit 124 updates the output data y output from the acquisition unit 121 and the predicted data y calculated by the calculation processing unit 11. * From the above, the objective function of each training data D(x, y) or noise data Dn(x, y) is calculated and averaged. The update unit 124 calculates the parameters θ * The update unit 124 updates the parameter θ set in the calculation processing unit 11 with the calculated parameter θ * to update.

[0050] When learning is completed, in step S14, the mixing control unit 125 determines whether the mixing ratio T is T=0. If T=0 is not (S14:N), in step S15, the mixing control unit 125 reduces the mixing ratio T. Thereafter, the process returns to step S12, and noise data Dn(x,y) for the reduced mixing ratio T and training data (x,y) for the mixing ratio (1-T) are newly acquired, and learning is repeated using the mixed training data group. If T=0 (S14:Y), that is, when there is no noise data Dn(x,y) left in the training data group, the learning process ends.

[0051] In the learning process, the mixing control unit 125 can adjust the rate at which the mixing ratio T is reduced with respect to the number of learning times. The mixing control unit 125 may reduce the mixing ratio T according to a schedule in which the rate of reduction with respect to the number of learning times is determined in advance, i.e., how much the mixing ratio T is reduced for each learning time. For example, the rate at which the mixing ratio T is reduced may be scheduled for each learning time, such as -10% after the first learning time, and -20% after the second learning time. In addition, the mixing ratio T may be scheduled to be reduced every time a certain number of learning times are performed, such as 0% after the first and second learning times, -10% after the third learning time, 0% after the fourth learning time, and -50% after the fifth learning time.

[0052] If the mixture ratio T is rapidly reduced with a small number of learning iterations, the time required to search for a solution is shortened, but the search area tends to become narrower. On the other hand, if the mixture ratio T is gradually reduced after many learning iterations, the time required to search for a solution is lengthened, but the probability of converging to a global optimum solution tends to increase. The mixture control unit 125 can adjust the ratio of reduction of the mixture ratio T relative to the number of learning iterations so as to converge to a global optimum solution in a short time.

[0053] Although an example in which the mixing ratio T is gradually decreased has been described, the process of changing the mixing ratio T may be scheduled so as to temporarily increase the mixing ratio T.

[0054] As described above, according to the first embodiment, when learning is performed by the backpropagation method, mini-batch learning is repeated to update the parameter θ using a training data set mixed with noise data Dn(x, y), and the mixing ratio T of the noise data Dn(x, y) in the training data set is reduced during the mini-batch learning.

[0055] In the backpropagation method, when the parameter θ is optimized according to the gradient of the objective function E, it may fall into a local optimum rather than a global optimum. This is because, in order to find the state (solution) that minimizes the error, if the state is always updated in the direction that minimizes the gradient of the objective function, it is impossible to escape from the local optimum. The simulated annealing method is known as a method to avoid this.

[0056] Simulated Annealing allows updates to a state where the error increases. The parameter that controls the degree to which updates to a state where the error increases are allowed is called "temperature." By gradually cooling from a "high temperature" state, which allows an increase in error, to a "low temperature" state, which does not allow an increase in error, it is possible to reach a global optimum without falling into a local optimum.

[0057] Noise data Dn(x,y) different from the training data D(x,y) allows updating to a state in which the error increases. Also, the larger the mixture ratio T of the noise data Dn(x,y), the more likely the error is to increase. Therefore, the mixture ratio T corresponds to the temperature parameter in simulated annealing, and the learning process of this embodiment, which repeats learning while changing the mixture ratio T, realizes simulated annealing in the backpropagation method.

[0058] In the conventional general simulated annealing method, the objective function E is calculated using training data at the initial temperature T(0), and then the parameter θ is changed and the objective function E is calculated again using the training data. If the error ΔE of the objective function E decreases before and after the change in the parameter θ, the change is accepted, and if the error ΔE increases, the change is accepted with a certain probability p, and if it is not accepted, the change is canceled.

[0059] For example, the probability p is determined by the following equation using a sigmoid function, where n is the current number of iterations, T(n) is the temperature, and ΔE is the amount of increase in error.

number

[0060] Thus, to implement the conventional simulated annealing method, it is necessary to add a configuration to calculate the error ΔE of the objective function E and determine whether or not to accept a change in the parameter θ based on the change. Since the amount of calculation also increases, it was difficult to implement it in a system when used in conjunction with the backpropagation method.

[0061] However, according to the present embodiment, since it is only necessary to adjust the mixing ratio T of the noise data Dn(x,y) as a temperature parameter, no special calculation or judgment is required. Simulated annealing can be easily performed in the backpropagation method, and implementation in the system is easy. Therefore, it is possible to provide a data processing device 1A that is easy to implement and can easily reach a global optimum solution.

[0062] Second Embodiment In the first embodiment in which simulated annealing is performed using the mixture ratio T of the noise data Dn(x, y) as a temperature parameter, dropout may be further combined.

[0063] FIG. 5 shows the configuration of a data processing device 1B according to the second embodiment. The data processing device 1B includes a calculation processing unit 11 and a learning processing unit 12B. The learning processing unit 12B includes an acquisition unit 121, a data storage unit 122, a data generation unit 123, an update unit 124, a mixing control unit 125, and a dropout control unit 126.

[0064] The acquiring unit 121, the data saving unit 122, the data generating unit 123, the updating unit 124, and the mixing control unit 125 have the same configurations as those in the first embodiment. In Fig. 5, the same components as those in the data processing device 1A in the first embodiment are denoted by the same reference numerals.

[0065] The dropout control unit 126 performs dropout on the neural network 30. Dropout is a process of disabling some of the nodes 31 in the hidden layer 4b. In the disabled nodes 31, no calculation is performed on the input data x, and no output is produced.

[0066] FIG. 6 shows a neural network 30 in which dropout is combined. In Fig. 6, the black nodes 31 are the nodes where dropout has been implemented. According to the example of Fig. 6, the dropout rates of the nodes 31 in the two hidden layers 4b are 1 / 2 and 1 / 3, respectively.

[0067] The dropout rate refers to the proportion of nodes 31 in one layer where dropout has been performed. The dropout rate can be set appropriately depending on the purpose. The higher the dropout rate, the easier it is to improve the generalization performance of the neural network 30. Therefore, it becomes easier to obtain an optimal solution not only for learned training data, but also for unknown data. On the other hand, the lower the dropout rate, the easier it is to obtain a solution as learned.

[0068] The learning processing unit 12B of the second embodiment performs the same learning process as in the first embodiment after the dropout control unit 126 performs dropout of the calculation processing unit 11. As a result, in a state in which some nodes 31 are dropped out, mini-batch learning is repeated while reducing the mixture ratio T of the noise data Dn(x, y) in the training data group.

[0069] By using simulated annealing with the mixture ratio T as a temperature parameter, noise data Dn(x,y) can be learned in addition to the training data D(x,y), and the number of data N learned by the neural network 30 can be increased. However, the noise data Dn(x,y) is different from the original training data D(x,y) and does not necessarily have a relationship between an example problem and a solution. Therefore, depending on the degree of simulated annealing, neutral predicted data, that is, a prediction result that is not any solution, may be output for input data other than the training data D(x,y). In this case, it is the same as creating a dictionary of the training data D(x,y), and the generality (also called generalization ability) of the neural network 30, which outputs an optimal solution even for unknown data, cannot be guaranteed.

[0070] In contrast, dropout invalidates the calculations of some nodes 31, and therefore can prevent some features of the input data from being over-evaluated. Therefore, in the second embodiment, in which dropout is further combined with the learning process that performs simulated annealing in the first embodiment, the predicted data y of the optimal solution can be obtained even if some of the input data is missing. * The parameter θ is updated so that the following is calculated. As a result, even if the noise data Dn(x,y) is used for learning, it becomes easier to obtain an optimal solution for unknown data. The above-mentioned neutral output can be reduced, and the generality of the neural network 30 can be improved.

[0071] According to the following third and fourth embodiments, a neural network 30 is provided which is highly robust and has a low vulnerability to output changes caused by a slight change in input.

[0072] Third Embodiment FIG. 7 shows the configuration of a data processing device 1C according to the third embodiment. The data processing device 1C includes a calculation processing unit 11 and a learning processing unit 12C. The learning processing unit 12C includes an acquisition unit 121, a data storage unit 122, a data generation unit 123, an update unit 124, and a conversion unit 127.

[0073] The acquiring unit 121, the data saving unit 122, the data generating unit 123 and the updating unit 124 have the same configurations as those in the first embodiment. In Fig. 7, the same components as those in the data processing device 1A in the first embodiment are denoted by the same reference numerals.

[0074] The conversion unit 127 converts the predicted data y output from the calculation processing unit 11 into the input data x of the training data D(x, y). * is converted to solution data Y using a threshold. The predicted data y * is data (eg, multiple values ​​from 0 to 1) representing the probability of belonging to one of a number of solutions, while solution data Y is data (eg, binary, 0 or 1) representing one of the multiple solutions.

[0075] FIG. 8 is a flowchart of the learning process in the learning processing unit 12C. First, in step S31, the acquisition unit 121 acquires training data D(x, y) and noise data Dn(x, y) to generate a predetermined number N of training data groups.

[0076] In step S32, mini-batch learning is performed using the generated training data group. During learning, the acquisition unit 121 outputs input data x of training data D(x, y) or noise data Dn(x, y) in the training data group to the calculation processing unit 11, and outputs output data y to the update unit 124. The calculation processing unit 11 derives predicted data y from the input x. * Calculate.

[0077] The update unit 124 updates the output data y output from the acquisition unit 121 and the predicted data y calculated by the calculation processing unit 11. * From the above, the objective function of each training data D(x, y) or noise data Dn(x, y) is calculated and averaged. The update unit 124 calculates the parameters θ * The parameter θ set in the calculation processing unit 11 is calculated as the calculated parameter θ * to update.

[0078] When the mini-batch learning is completed, in step S33, the conversion unit 127 converts the output data y of the noise data Dn(x,y) used in the learning into one of a plurality of solution data Y. The conversion is performed based on a certain probability. The certain probability is a probability given by a threshold value.

[0079] The threshold value may be a boundary value for classifying the solution data Y. For example, when the solution data Y is a binary value of 0 or 1, the predicted data y * represents the probability that the solution belongs to either the solution represented by 0 or the solution represented by 1. If 0.5 is given as the boundary value between these two solutions, 0.5 will cause the predicted data y * is probabilistically converted into solution data Y. Specifically, the predicted data y * is stochastically converted to solution data Y of 1 if it is 0.5 or more, and to solution data Y of 0 if it is less than 0.5. The solution data Y in this example is expressed as Y=F(x). F(x) represents a function that converts the input data x into solution data Y through calculations in each layer of the neural network 30 from the input layer 4a to the output layer 4c.

[0080] The threshold may be determined by a sigmoid function. Alternatively, the threshold may be a random number. In this case, the conversion unit 127 acquires a random number and converts it into predicted data y * If x is greater than the random number, it may be probabilistically converted to solution data Y of 1, and if x is less than the random number, it may be probabilistically converted to solution data Y of 0.

[0081] Next, in step S34, the acquisition unit 121 acquires training data D(x, y). The acquisition unit 121 mixes the acquired training data D(x, y) with the converted noise data Dn(x, y) to generate a predetermined number N of training data groups.

[0082] In step S35, mini-batch learning is performed again using the generated training data set to update the parameter θ in the same manner as in step S32.

[0083] If the number of learning times has not reached the predetermined number in step S36 (S36: N), the process returns to step S33. This repeats the conversion of the output data y of the noise data Dn(x, y) to solution data Y and the learning using the training data group including the converted noise data Dn(x, y). If the number of learning times has reached the predetermined number in step S36 (S36: Y), the learning process ends.

[0084] In the above learning process, the conversion unit 127 can adjust the ratio of the number of times the output data y is converted to the number of times learning is performed. The conversion unit 127 may perform the conversion according to a schedule in which the timing of performing the conversion for the number of times learning is determined in advance, that is, how many times learning is performed before the conversion is performed. For example, the conversion may be performed for each learning as described above, or the conversion may be performed once every two learnings.

[0085] The ratio of the number of times the conversion unit 127 converts the output data y to the number of times the conversion unit 127 learns is relatively large. * This is preferable because it is easy to suppress bias towards one solution. Furthermore, the ratio may be adjusted according to the number of data N in the training data group used for learning.

[0086] As described above, according to the third embodiment, mini-batch learning is repeated while the output data y of the noise data Dn(x,y) used for learning is converted into solution data Y. As a result, the output data y of the noise data Dn(x,y) approaches one of the solution data Y.

[0087] While the training data D(x,y) is originally a combination of output data that is an optimal solution to input data that is an example problem, the noise data Dn(x,y) does not necessarily have such a relationship, and the combination of input data and output data is arbitrary. However, in the third embodiment, while repeating learning, it is possible to learn noise data Dn(x,y) that is probabilistically close to solution data Y, which is the optimal solution for the input data, by combining certain output data with input data different from the training data D(x,y). Therefore, even for unknown input data that has changed slightly from the learned input data, the predicted data y * becomes less likely to change, improving the robustness of the prediction.

[0088] The vulnerability of the output changing when the input changes slightly is easily caused by the complexity of the output boundary, which is the range of the output around the input. The complexity of the output boundary can be evaluated, for example, by how many outputs for the input's neighboring values ​​are different from the output for the input.

[0089] FIG. 9 conceptually illustrates the complexity of the output boundary. If the output boundary 63 around the input is not around the output point 61 for the input and has high complexity, the prediction may be fragile. In contrast, according to the third embodiment, as the output data y of the noise data Dn(x,y) is transformed and learning is repeated, the output boundary 63 changes to the output boundary 62 with lower complexity. As a result, the output is less likely to change even if the input changes slightly.

[0090] [Fourth embodiment] In the third embodiment, the output data y is converted into the solution data Y based on a certain probability. However, the robustness of the prediction can also be improved by converting the output data y as follows.

[0091] 10 is a flowchart of the learning process in the fourth embodiment. This learning process can be executed by the same configuration as the data processing device 1C in the third embodiment. First, in step S41, the acquisition unit 121 acquires noise data Dn(x, y).

[0092] In step S42, the acquisition unit 121 outputs the input data x of the noise data Dn(x,y) to the calculation processing unit 11. The calculation processing unit 11 outputs the predicted data y * Calculate and output.

[0093] In step S43, the acquisition unit 121 outputs one or more pieces of input data x^ that exist in the vicinity of the input data x to the calculation processing unit 11. The surrounding input data x^ refers to data that is adjacent to the input data x in the data space of the input data x or that is within a certain distance from the input data x. The data space is a high-dimensional space in which the multiple variables xi that the input data x has are vector elements. The calculation processing unit 11 outputs predicted data y for each input data x^. * Calculate and output ^.

[0094] In step S44, the conversion unit 127 converts each prediction data y * and y * Among the multiple solution data Y into which ^ is classified, the conversion unit 127 determines the majority solution data Y. Specifically, the conversion unit 127 converts each prediction data y * and y * The number of identical solution data Y among the multiple solution data Y into which ^ is classified is counted, and the solution data Y with the largest count is determined as the majority solution data Y. Then, the conversion unit 127 converts the output data y of the noise data Dn(x, y) used for learning into the determined solution data Y.

[0095] For example, if the input data x of the noise data Dn(x,y) is a randomly generated 4-bit string of “1010”, the 4-bit string of “1010” which is the input data x and the 4-bit strings of “1000”, “1001”, “1011”, and “1100” which are the surrounding input data x^ are output to the calculation processing unit 11.

[0096] For each input data x and x^, the following predicted data y * and y * When ^ is output, the threshold value of 0.5 is used to estimate each predicted data y * and y * ^ is classified into solution data Y of three “0”s and two “1”s. [Table 1]

[0097] Of the five solution data Y found from each input data x and x^, the number of solution data Y of "0" is three and the number of solution data Y of "1" is two. Therefore, the majority solution data Y is determined to be "0". If the output data y paired with the input data x of "1010" is "1", the noise data Dn(1010,1) is converted to noise data Dn(1010,0). Note that if the majority solution data Y is "0", the same as the original output data y, the conversion can be skipped.

[0098] In step S45, the acquisition unit 121 acquires training data D(x, y). The acquisition unit 121 mixes the training data D(x, y) with the converted noise data Dn(x, y) to generate a training data group.

[0099] In step S46, mini-batch learning is performed using the generated training data group. During learning, the acquisition unit 121 outputs input data x of each training data D(x, y) or noise data Dn(x, y) in the training data group to the calculation processing unit 11, and outputs output data y to the update unit 124. The calculation processing unit 11 derives predicted data y from the input data x. * Calculate.

[0100] The update unit 124 updates the output data y output from the acquisition unit 121 and the predicted data y calculated by the calculation processing unit 11. * From the above, the objective function of each training data D(x, y) or noise data Dn(x, y) is calculated and averaged. The update unit 124 calculates the parameters θ *The parameter θ set in the calculation processing unit 11 is calculated as the calculated parameter θ * to update.

[0101] If the number of learning times has not reached the predetermined number in step S47 (S47: N), the process returns to step S42. As a result, the output data y of the noise data Dn(x,y) used in learning in step S46 is converted into majority solution data Y, and mini-batch learning is repeated using a training data group including the converted noise data Dn(x,y). If the number of learning times has reached the predetermined number in step S47 (S47: Y), the learning process ends.

[0102] As in the third embodiment, the conversion unit 127 can adjust the ratio of the number of times the output data y is converted to the number of times the learning is performed. * This is preferable because it is easy to prevent bias toward one solution. Also, the ratio of the number of conversions to the number of learnings may be adjusted according to the number of data N in the training data group used for learning.

[0103] As described above, according to the fourth embodiment, the output data y of the noise data Dn(x,y) is calculated by subtracting the predicted data y of the output data y and the surrounding input data x^. * Among the multiple solution data Y classified as ^ and ^, the majority solution data Y is converted into the majority solution data Y, and mini-batch learning is repeated. As a result, if the input data x^ is around the input data x of the noise data Dn(x,y), the same prediction data y * is approaching.

[0104] The training data D(x,y) is originally a combination of output data that is an optimal solution to input data, which is an example problem, but the noise data Dn(x,y) does not necessarily have such a relationship, and the combination of input data and output data is arbitrary. However, in the fourth embodiment, as learning is repeated, it is possible to learn noise data Dn(x,y) whose output data is in tune with solution data Y surrounding the input data. Therefore, even for unknown input data that has changed slightly from the learned input data x, the predicted data y * becomes less likely to change, improving the robustness of the prediction.

[0105] As described above, the vulnerability of the output changing due to a slight change in the input is due to the complexity of the output boundary for the periphery of the input. According to the fourth embodiment, when the output for the input is different from the majority of the outputs for the periphery of the input, local tuning can be realized by correcting the output data y of the noise data Dn(x,y) so as to tune to the periphery. Therefore, as in the third embodiment, the complexity of the output boundary can be reduced, and the above vulnerability can be reduced.

[0106] According to the following fifth to seventh embodiments, it is possible to impart desired characteristics to the neural network 30 by executing preprocessing.

[0107] Fifth embodiment In the calculation processing unit 11, the parameter θ is usually initialized to an arbitrary value, for example, 0 or a random number, and is updated by repeating learning of the training data D(x, y). The initial value of this parameter θ affects the learning process and results. In the fifth embodiment, the initial value of this parameter θ is adjusted by preprocessing so as to impart the neural network 30 with the desired learning characteristics.

[0108] FIG. 11 shows the configuration of a data processing device 1D according to the fifth embodiment. The data processing device 1D includes a calculation processing unit 11 and a learning processing unit 12D. The learning processing unit 12D includes an acquisition unit 121, a data storage unit 122, a data generation unit 123, an update unit 124, and a dropout control unit 126. These have the same configuration as in the second embodiment. In Fig. 11, the same reference numerals are used to denote the same components as in the data processing device 1B of the second embodiment.

[0109] FIG. 12 shows the procedure of the pre-processing. First, in step S51, the update unit 124 sets a parameter θ in the calculation processing unit 11. The initial value of this parameter θ is an arbitrary value such as 0 or a random number.

[0110] In step S52, the dropout control unit 26 performs dropout of the calculation processing unit 11. The dropout rate can be determined appropriately. The higher the dropout rate, the smaller the influence of the input data on the calculation of the optimal solution. The gradients of the objective functions calculated from any training data D(x,y) are more likely to be similar, and the sensitivity to each training data D(x,y) is more likely to be reduced, making it easier to give high generality to the neural network 30. On the other hand, the lower the dropout rate, the larger the influence of the input data on the calculation of the optimal solution. Therefore, the gradients of the objective functions calculated from any training data D(x,y) are more likely to be independent of each other, improving learning efficiency and allowing more data to be learned.

[0111] In step S53, the acquisition unit 121 acquires a predetermined number N of noise data Dn(x, y) and generates a training data group made up of the noise data Dn(x, y).

[0112] In step S54, mini-batch learning is performed using the generated training data group. During learning, the acquisition unit 121 outputs input data x of each noise data D(x, y) to the calculation processing unit 11, and outputs output data y to the update unit 124. The calculation processing unit 11 derives predicted data y from the input data x. * Calculate.

[0113] The update unit 124 updates the output data y output from the acquisition unit 121 and the predicted data y calculated by the calculation processing unit 11. * The objective function of each noise data Dn(x, y) is calculated from the error between the noise data Dn(x, y) and the average is calculated. The update unit 124 updates the parameters θ * The parameter θ set in the calculation processing unit 11 is calculated as the calculated parameter θ * to update.

[0114] Updated parameters θ * is set as the initial value of the parameter θ. After that, a learning process is performed using the training data D(x, y), and the initial value of the parameter θ is set to the parameter θ optimized by the training data D(x, y). * It will be updated to.

[0115] As described above, according to the fifth embodiment, the parameter θ is initialized by the noise data Dn(x, y) before the learning process using the training data D(x, y).

[0116] Since the initial value of the parameter θ is not an arbitrary value with no meaning, but is the result of learning the noise data Dn(x,y), the learning sensitivity to the training data D(x,y) used in the subsequent learning process is increased. Also, since the noise data Dn(x,y) is learned in preprocessing in addition to the training data D(x,y) to be learned in the subsequent learning process, the overall amount of data to be learned can be increased. Therefore, the time until the parameter θ is optimized by learning can be shortened, and the neural network 30 can be endowed with learning characteristics that enable it to effectively learn the training data D(x,y).

[0117] In addition, since the input data and output data of the noise data Dn(x,y) are not necessarily in a relationship between an example and a solution like the training data D(x,y), the use of the noise data D(x,y) can impart generality to the neural network 30. Furthermore, by combining dropout, it is possible to adjust the sensitivity to any training data D(x,y), that is, the dependency of the output on the training data D(x,y). By reducing the dependency of the output on the training data D(x,y), it is possible to prevent the output from changing even if the input changes slightly from the input data of the training data D(x,y), and the generality of the neural network 30 can be increased. Therefore, it is possible to impart the desired generality to the neural network 30 in advance by preprocessing.

[0118] Sixth Embodiment In the fifth embodiment, preprocessing was performed to initialize the parameter θ, but generality can also be imparted to the neural network 30 by preprocessing to selectively obtain a group of training data to be used for learning.

[0119] FIG. 13 is a flowchart of the pre-processing according to the sixth embodiment. The pre-processing of the sixth embodiment can be executed by the data processing device 1D of the fifth embodiment. This pre-processing is a process for preparing a training data group in advance, and this training data group is used in the learning process executed after the pre-processing.

[0120] In the pre-processing of the sixth embodiment, in step S61, the acquisition unit 121 acquires training data D(x, y). In step S62, the data generator 123 generates noise data Dn(x, y) based on the density of the acquired training data D(x, y).

[0121] The density of the training data D(x,y) refers to the density of each input data x in the data space in which the input data x of the multiple training data D(x,y) are distributed. As described above, the data space is a high-dimensional space in which the multiple variables xi of the input data x are vector elements.

[0122] The density of input data x can be expressed by the density function of input data x when a group of N input data x is given and each input data x has a variable xi. When xi is defined as δ(x,h) by the delta function, the density function p(x) is expressed as p(x)=Σδ(xi,h) / N. δ(x,h) is a Gaussian with a peak only at x and a skirt of radius h.

[0123] For example, the data generator 123 may generate more noise data Dn(x,y) in a region where the density of the training data D(x,y) is high than in a region where the density of the training data D(x,y) is low. Specifically, noise data Dn(x,y) having the same density as the training data D(x,y) may be generated around some training data (x,y) in the data space of the training data D(x,y). Alternatively, data located at the center of gravity of some training data (x,y) may be generated as noise data Dn(x,y), and the density of the training data D(x,y) may be increased by adding the noise data Dn(x,y).

[0124] On the other hand, the data generator 123 can generate noise data Dn(x,y) having a similarly low density in the vicinity of a low density region of the training data D(x,y). For example, noise data Dn(x,y) having the same density as the training data D(x,y) may be generated around some training data (x,y) in the data space of the training data D(x,y).

[0125] According to the training data set to which the noise data Dn(x, y) has been added in this way, it is possible to impart different generality to the neural network 30 depending on the training data D(x, y).

[0126] Specifically, in an area where the density of training data D(x,y) is high, input data x associated with the optimal solution is present in the vicinity of input data x associated with output data y, which is the optimal solution, so that the output changes sensitively even to slight differences in input, and the optimal solution is likely to be output. In other words, in an area where the density of training data D(x,y) is high, the generality of the neural network 30 can be reduced, and the discrimination ability can be improved.

[0127] On the other hand, in an area where the density of the training data D(x, y) is low, noise data Dn(x, y) having the same or lower density as the training data D(x, y) is added to the periphery, and the area where the density of the training data D(x, y) is low expands. This reduces the area of ​​unknown data, and even if the input changes slightly, the output changes little, so that the generality of the neural network 30 can be increased in the area where the density of the training data D(x, y) is low.

[0128] Next, in step S63, the acquisition unit 121 mixes the noise data Dn(x, y) generated by the data generation unit 123 with the training data D(x, y) acquired from the data storage unit 122 to generate a training data group.

[0129] Alternatively, the data generating unit 123 may generate random noise data Dn(x, y), and the acquiring unit 121 may acquire noise data Dn(x, y) from the random noise data Dn(x, y) according to the density of the training data D(x, y).

[0130] In general, training data D(x,y) is not necessarily sampled from the target with uniform weights, and there are areas where training data D(x,y) is sampled a lot and areas where it is little. In this way, when the distribution of training data D(x,y) differs depending on the area, the same output is not necessarily expected in every area. For example, in an area where a lot of training data D(x,y) is sampled, it is expected that the output will change sensitively due to slight differences in the input data. Conversely, in an area with a small number of samples, the same output is expected even if the input data changes slightly.

[0131] In contrast, according to the sixth embodiment, noise data Dn(x,y) is acquired according to the density of the training data D(x,y). Since the noise data Dn(x,y) according to the density of the training data D(x,y) can be supplemented to the data space in which the training data D(x,y) is distributed, it is possible to adjust the generality expected for each region as in the above example. Therefore, the noise data Dn(x,y) can be given different generality for each region in which the density of the training data D(x,y) is different, by the pre-processing that generates the training data group.

[0132] Seventh embodiment In the seventh embodiment, a pre-processing step for controlling the dropout rate is performed to impart to the neural network 30 a property of classifying the input data x hierarchically from major classifications to minor classifications.

[0133] The pre-processing in the seventh embodiment can be executed by the data processing device 1D in the fifth embodiment. After this pre-processing, the learning process is carried out.

[0134] In the preprocessing of the seventh embodiment, dropout is performed by the dropout control unit 26 in the calculation processing unit 11. At this time, the dropout control unit 26 sets the dropout rate of each layer to be gradually lower from the input layer 4a to the output layer 4c.

[0135] For example, when dropping out three hidden layers 4b, the dropout rates of the hidden layers 4b from the input layer 4a to the output layer 4c may be set to 50%, 25%, and 10%, in that order.

[0136] As described above, according to the seventh embodiment, the dropout rate of the hidden layer 4b decreases from the input layer 4a to the output layer 4c. When the input data x has i variables xi, the calculation in each node 31 in the hidden layer 4b has a function of extracting features from each variable xi of the input data x and classifying them. Dropout stops the classification function of such a node 31.

[0137] Therefore, by controlling the dropout rate gradient as described above, there are many nodes 31 on the input layer 4a side where the classification function stops, and there are fewer such nodes 31 on the output layer 4b side. In other words, the classification function of the hidden layer 4b works more easily from the input layer 4a to the output layer 4c, enabling finer classification. As a result, each variable xi of the input data x is classified and characterized hierarchically from large classification to small classification.

[0138] In this way, according to the seventh embodiment, the neural network 30 can be provided with the property of classifying the input data x hierarchically from major classifications to minor classifications through preprocessing.

[0139] Generally, neural networks are prediction models in which it is difficult to clarify the principles of prediction, such as how they make predictions. However, even for neural networks, prediction models with the above-mentioned hierarchical classification function make it easy for users to understand the principles of prediction. This makes it easier for users to design learning content and use prediction results.

[0140] The first to seventh embodiments described above can be appropriately combined. For example, after the parameter θ is initialized in the pre-processing of the fifth embodiment, the learning process of the first embodiment can be performed using the initialized parameter θ. Furthermore, when generating a training data group to be used in mini-batch learning in the second embodiment, the sixth embodiment can be combined to perform pre-processing to generate a training data group.

[0141] An example of the application of the data processing device of the present invention is vehicle driving support. For example, the data processing device may predict a target steering amount, a driving speed, etc., using data from various sensors such as a camera, a vehicle speed sensor, a steering angle sensor, or a GPS mounted on the vehicle as input data. The data processing device may also predict the possibility of a collision with a nearby vehicle or pedestrian using data from a camera, a millimeter wave radar, etc. as input data.

[0142] The present invention can be used for various purposes other than vehicles, such as analyzing big data and detecting lesions in X-ray image data.

[0143] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and can be modified as appropriate within the scope of the present invention.

[0144] For example, the present invention can be applied to prediction models other than neural networks as long as the prediction model has a plurality of calculation modules connected in a line, such as an input layer 4a, a hidden layer 4b, and an output layer 4c of a neural network, and calculates predicted data by using parameters associated with the calculation modules, such as weights wi and biases bi, to perform calculations on the output from the previous calculation module, and outputting the results of this calculation to the subsequent calculation module.

[0145] The present invention can also be applied to neural networks used in generative adversarial networks (GANs). Generally, a neural network called a generator and a neural network called a discriminator are used in a GAN. The generator generates fake data from given data, and the discriminator uses real data to determine whether the generated fake data is true or false. Depending on the result of the determination, the generator generates fake data that is closer to the real data. The present invention can also be applied to generators and discriminators that learn in a competitive manner in this way. [Explanation of symbols]

[0146] 1A to 1D: data processing device, 11: calculation processing unit, 12A to 12D: learning processing unit, 121: acquisition unit, 124: update unit, 125: mixing control unit, 126: dropout control unit, 127: conversion unit

Claims

1. A data processing device (1D) for performing deep learning of a neural network, a calculation processing unit (11) having a neural network (30) for calculating predicted data from input data using parameters, and outputting the predicted data; A learning processing unit (12D) that executes a learning process to update the parameters using training data that is a pair of input data and output data associated in advance, The learning processing unit (12D) Prior to the learning process, a pre-processing is performed to give characteristics to the neural network (30), and in the pre-processing, the training data is acquired, and noise data, which is a pair of input data and output data different from the training data, is acquired according to the density of the training data, and the noise data is mixed with the training data to generate a training data group to be used in the learning process. Data processing device (1D).

2. The learning processing unit (12D) in the pre-processing Setting arbitrary parameters in the neural network (30); Initializing the arbitrary parameters using the noise data. A data processing device (1D) according to claim 1.

3. The learning processing unit (12D) in the pre-processing performing dropout of said neural network (30); updating the arbitrary parameters so that an error between output data of the noise data and predicted data calculated by the neural network (30) from input data of the noise data becomes small; The updated parameters are set as initial values ​​of the parameters to be updated in the learning process. A data processing device (1D) according to claim 2.

4. The neural network (30) has an input layer (4a), a hidden layer (4b) and an output layer (4c); The learning processing unit (12D) in the pre-processing performing dropout of said neural network (30); The dropout rate in each layer of the neural network (30) is gradually reduced from the input layer (4a) to the output layer (4c). A data processing device (1D) according to claim 1.

5. A method of deep learning for a neural network (30) for calculating predicted data from input data using parameters, comprising: A step of performing a learning process to update the parameters using training data that is a pair of input data and output data that are previously associated with each other; and performing a pre-processing step of giving characteristics to the neural network (30) before the learning process; In the step of performing the pre-processing, the training data is acquired, and noise data, which is a pair of input data and output data different from the training data, is acquired according to the density of the training data, and the noise data is mixed with the training data to generate a training data group to be used in the learning process. method.

6. A program executed by a computer to perform deep learning of a neural network (30) that calculates prediction data from input data using parameters, the program comprising: A step of performing a learning process to update the parameters using training data that is a pair of input data and output data that are previously associated with each other; A step of performing a pre-processing to give characteristics to the neural network (30) before the learning process; Run A program for acquiring the training data in the step of performing the preprocessing, acquiring noise data which is a pair of input data and output data different from the training data in accordance with the density of the training data, and mixing the noise data with the training data to generate a training data group to be used in the learning process.

Images