Recording medium, information processing apparatus, and computer-implemented method

Abstract

The present disclosure relates to a recording medium storing a machine learning program, an information processing apparatus, and a computer-implemented method. In one embodiment, the machine learning program controls machine learning of distributed neural network models generated by partitioning a neural network. The machine learning program includes instructions for causing a processor to perform processing including, for each of the distributed neural network models, adding individual noise for the distributed neural network model to non-parallel processing blocks in the distributed neural network model such that the individual noise of the distributed neural network model is different from individual noise of other distributed neural network models among the distributed neural network models, and distributing the distributed neural network model to which the individual noise is added to a plurality of processes to cause each of the plurality of processes to perform machine learning on the distributed neural network model to which the individual noise is added.

Description

TECHNICAL FIELD

[0001] Embodiments discussed herein relate to a machine learning program, an information processing apparatus, and a machine learning method. BACKGROUND

[0002] In recent years, in machine learning in neural networks, as the size of a machine learning model increases, there is a demand for improving the speed of learning.

[0003] For example, in a simulation of estimating cosmological parameters from dark matter data using CosmoFlow, the data amount is 5.1 TB, and machine learning by a single V100 graphics processing unit (GPU) takes a week.

[0004] Furthermore, data parallelism, which is a mainstream acceleration method in machine learning, has a limitation in terms of accuracy. In other words, for example, when parallelism increases, the batch size increases, and there is a possibility that this adversely affects the learning accuracy.

[0005] Therefore, in recent years, a model parallel method for dividing a machine learning model in a neural network and performing parallel processing by a plurality of computers is known. Hereinafter, there are cases where a machine learning model in a neural network is simply referred to as a neural network model or a model.

[0006] By performing parallel processing on each of the models created by dividing the neural network model by a plurality of computers, the learning accuracy is not affected, and the speed of machine learning can be improved.

[0007] Figure 8 is a diagram for explaining a conventional model parallel method in a neural network.

[0008] In Figure 8 , reference sign A indicates a neural network model that is not parallelized. Furthermore, reference sign B indicates a model-parallelized neural network model, and represents two models (process #0 and process #1) created by dividing the single model indicated by reference sign A.

[0009] In the model-parallelized neural network indicated by reference sign B, all layers (layers) including the convolution layer and the fully connected layer of the neural network indicated by reference sign A are divided and parallelized.

[0010] However, in the model-parallelized neural network indicated by reference sign B, communication (allgather and allreduce) frequently occurs between process #0 and process #1 before and after each layer. This increases the communication load and causes delays and the like due to waiting and the like for synchronization.

[0011] Therefore, we consider a method that parallelizes only the computationally intensive convolutional layers among the multiple layers included in the neural network.

[0012] Figure 9 This is a diagram used to illustrate the traditional parallel model approach in neural networks.

[0013] Figure 9 It shows a non-parallelized and... Figure 8 The neural network model indicated by reference numeral A in the attached figure is a parallel neural network model created by the model.

[0014] Figure 9 The neural network shown only includes non-parallelized and... Figure 8 The diagram, indicated by reference numeral A, shows a neural network model where the convolutional layer is divided into two. In other words, for example, the processing of the convolutional layer is performed in parallel by process #0 and process #1, while the processing of the fully connected layer is performed only by process #0.

[0015] Typically, while convolutional layer processing is computationally intensive, it only involves data exchange between adjacent parts as communication. Therefore, the drawbacks caused by dividing convolutional layers are minimal. Furthermore, because the number of neurons in fully connected layers in later stages is small, the computation time does not increase even without parallelization, and there are cases where the processing speed is higher than when the model is parallelized.

[0016] Examples of related technologies include: Japanese National Patent Publication No. 2017-514251; and U.S. Patent Application Publication No. 2020 / 0372337. Summary of the Invention

[0017] Technical issues

[0018] However, in Figure 9 In the traditional parallel neural network shown, process #1 does not perform any processing other than the convolutional layer. Process #1 wastes computational resources and is inefficient.

[0019] In addition, Figure 9 In the parallelized neural network model shown, data communication is performed from process #0 to process #1 to share the loss finally computed by process #0. To shorten the time spent on data communication, for example, in process #0, the loss could be computed by process #1 performing the computation of each fully connected layer. However, in this case, the same computation of the fully connected layers is repeated in both process #0 and process #1, which is inefficient.

[0020] In one aspect, the objective of the implementation is to efficiently utilize computational resources in machine learning of multiple distributed neural network models with parallel model processing.

[0021] Problem solution

[0022] According to an aspect of an embodiment, there is provided a non-transitory computer-readable recording medium storing a machine learning program that controls machine learning of a plurality of distributed neural network models generated by partitioning a neural network. In an example, the machine learning program includes instructions for causing a processor to execute a process including: for each of the plurality of distributed neural network models, adding individual noise for the distributed neural network model to non-parallel processing blocks in the distributed neural network model so that the individual noise of the distributed neural network model is different from individual noise of other distributed neural network models among the plurality of distributed neural network models; and distributing the plurality of distributed neural network models to which the individual noise is added to a plurality of processes to cause each of the plurality of processes to perform machine learning on the distributed neural network model distributed among the plurality of distributed neural network models to which the individual noise is added.

[0023] Advantages of the invention

[0024] According to an embodiment, in machine learning of a plurality of distributed neural network models of model parallel processing, computing resources can be effectively used. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a diagram schematically showing a hardware configuration of a computer system as an example of an embodiment;

[0026] Figure 2 is a functional configuration diagram of a management device of a computer system as an example of an embodiment;

[0027] Figure 3 is a conceptual diagram showing a neural network model generated by a computer system as an example of an embodiment;

[0028] Figure 4 is a flowchart for explaining a process by a model management unit of a computer system as an example of an embodiment;

[0029] Figure 5 is a diagram for explaining a machine learning process by a plurality of distributed models created by a computer system as an example of an embodiment;

[0030] Figure 6 is a diagram for explaining a machine learning process by a plurality of distributed models created by a computer system as an example of an embodiment;

[0031] Figure 7is a diagram for explaining a machine learning process by a plurality of distributed models created by a computer system as an example of an embodiment;

[0032] Figure 8 is a diagram for explaining a conventional model parallel method in a neural network; and

[0033] Figure 9 is a diagram for explaining a conventional model parallel method in a neural network. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of a machine learning program, an information processing apparatus, and a machine learning method will be described with reference to the accompanying drawings. Note that the embodiments to be described below are merely examples and are not intended to exclude application of various modifications and technologies that are not explicitly described in the embodiments. In other words, the present embodiments can be variously modified and implemented without departing from the spirit thereof, for example. Furthermore, each of the drawings is not intended to include only the components shown in the drawing and can include other functions and the like.

[0035] (A) Configuration

[0036] Figure 1 is a diagram schematically showing a hardware configuration of a computer system 1 as an example of an embodiment, and Figure 2 is a functional configuration diagram of a management apparatus of the computer system 1.

[0037] As shown in Figure 1 , the computer system 1 includes a management apparatus 10 and a plurality of computing nodes 2. The management apparatus 10 and each of the computing nodes 2 are connected to be communicable with each other via a network 3. The network 3 is, for example, a local area network (LAN).

[0038] In the computer system 1, a machine learning model in a neural network (neural network model) is divided, and the plurality of computing nodes 2 implement model parallel processing.

[0039] The computing node 2 is an information processing apparatus (computer) including a processor and a memory (not shown) and executes a process assigned by the management apparatus 10 described later. Each of the computing nodes 2 executes training (machine learning) of an assigned neural network model, inference using a corresponding neural network model, and the like.

[0040] The management apparatus 10 is, for example, an information processing apparatus (computer) having a server function and having a function for managing a neural network model.

[0041] As shown in Figure 1 , the management apparatus 10 includes, for example, a processor 11, a memory 12, and a storage apparatus 13.

[0042] The storage device 13 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a storage class memory (SCM) and stores various types of data.

[0043] The memory 12 is a storage memory including a read only memory (ROM) and a random access memory (RAM). In the ROM of the memory 12, a software program for managing a machine learning model and data for the program are written. The software program for managing a machine learning program includes a machine learning model.

[0044] The software program in the memory 12 is appropriately read and executed by the processor 11. Furthermore, the RAM of the memory 12 is used as a main storage memory or a work memory.

[0045] The processor (processing unit) 11 controls the entire management device 10. The processor 11 can also be a multi-processor. The processor 11 can also be, for example, any one of a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). Furthermore, the processor 11 can also be a combination of two or more types of elements of a CPU, an MPU, a DSP, an ASIC, a PLD, and an FPGA.

[0046] Then, the processor 11 executes a control program to function as Figure 2 the model management unit 100, the training control unit 102, and the inference control unit 103 illustrated in FIG. 1. The control program includes a machine learning program. The processor 11 executes the machine learning program to realize the function as the training control unit 102.

[0047] Note that, for example, a program (control program) for realizing the functions as the model management unit 100, the training control unit 102, and the inference control unit 103 is provided in the form of being recorded in a computer-readable recording medium such as a flexible disk, a compact disc (CD) (CD-ROM, CD-R, rewritable CD (RW), or the like), a digital versatile disc (DVD) (DVD-ROM, DVD-RAM, recordable DVD (R), DVD+R, DVD-RW, DVD+RW, high definition (HD) DVD, or the like), a Blu-ray disc, a magnetic disk, an optical disk, or a magneto-optical disk. Then, the computer reads the program from the recording medium, transfers the program to an internal storage device or an external storage device, and stores the program to use. Furthermore, for example, the program can also be recorded in a storage device (recording medium) such as a magnetic disk, an optical disk, or a magneto-optical disk, and provided to the computer from the storage device via a communication path.

[0048] When functioning as the model management unit 100, training control unit 102, and inference control unit 103, the program stored in the internal storage device (memory 12 in this embodiment) is executed by the computer's microprocessor (processor 11 in this embodiment). At this time, the computer can also read and execute programs recorded on a recording medium.

[0049] The model management unit 100 manages the neural network model.

[0050] In computer system 1, the neural network model is divided and parallel processing of the model is achieved through multiple computing nodes 2.

[0051] Figure 3 This is a conceptual diagram showing a neural network model generated by computer system 1.

[0052] exist Figure 3 The example shown illustrates two neural network models created by partitioning a single neural network model. Furthermore, in Figure 3 In the neural network model shown, only the convolutional layers are partitioned and parallelized, while other layers are not parallelized.

[0053] In the following text, there are cases where each of the multiple neural network models created by partitioning a single neural network model is referred to as a distributed neural network model or a distributed model. Furthermore, the single neural network model before partitioning can also be called the original neural network model.

[0054] Each created distributed model is processed by an individual computing node 2. In other words, for example, each distributed model is processed as a different process. Figure 3 The example shown illustrates the generation of two distributed models, with each of processes #0 and #1 processing a single distributed model.

[0055] Figure 3 Each distributed model shown includes convolutional layers and fully connected layers. Furthermore, the fully connected layer includes a dropout layer.

[0056] The dropout layer suppresses overtraining by performing machine learning, while simultaneously deactivating (disabling) a certain percentage of nodes. Note that in Figure 3 In the example shown, the fully connected layer includes a dropout layer. However, this embodiment is not limited to this, and dropout layers may also be included in convolutional layers, etc.

[0057] In computer system 1, the drop-out layer exists in multiple processes (in... Figure 3Different deactivation (invalidation) is performed between two of the examples shown in FIG. 1. Hereinafter, deactivating a specific node in a dropout layer can be referred to as adding noise. The dropout layer is used as a noise addition layer for adding noise to machine learning.

[0058] The model management unit 100 generates a parallelized neural network model as shown in FIG. 1. For example, the model management unit 100 adds noise different from the dropout layer included in another distributed model to each of the plurality of distributed models included in the parallelized neural network model. Figure 3

[0059] The convolution layer in the original neural network is divided into process #0 and process #1, each of which is processed in parallel by different computing nodes 2. In each distributed model, the convolution layer can also be referred to as a model parallelization unit. In addition, in each distributed model, the fully connected layer and the dropout layer, which are not processed in parallel by process #0 and process #1, can also be referred to as a non-model parallelization unit. In addition, the plurality of processes that perform processing of the convolution layer in parallel can also be referred to as model parallel processes.

[0060] The non-model parallelization unit of each distributed model includes a processing block that performs the same processing. Such a processing block is repeatedly included in the non-model parallelization units of the plurality of distributed models. In this way, the processing block repeatedly included in the non-model parallelization units of the plurality of distributed models can also be referred to as a repeated block. The repeated block is a group of layers that performs repeated processing between processes that perform model parallelization without performing model parallelization. In each distributed model, the dropout layer is included in the repeated block.

[0061] As shown in FIG. 1, the model management unit 100 has a function as a noise setting unit 101. Figure 2

[0062] The noise setting unit 101 sets various parameters configuring the dropout layer to perform different dropout processing for each dropout layer of the plurality of distributed models.

[0063] For example, the noise setting unit 101 can also set a different percentage of nodes to be deactivated (hereinafter, referred to as a dropout rate) for each distributed model. To set a different dropout rate for each distributed model, for example, an arbitrary dropout rate can also be selected from among a plurality of types of dropout rates using a random number for each dropout layer of each distributed model, and the selection can be appropriately changed and performed.

[0064] In addition, the noise setting method by the noise setting unit 101 is not limited to making the dropout rate different for each distributed model, and can be appropriately changed and performed. For example, the nodes to be deactivated can also be different for each distributed model, or the probability of dropout of input elements can also be different for each distributed model.​​

[0065] The noise setting unit 101 reads the data configuring the distributed model and determines whether the drop-out layer is included in the processing block of each layer of the corresponding distributed model. Then, if the distributed model includes a drop-out layer, it sets the parameters of each drop-out layer to perform different drop-out processes among multiple distributed models.

[0066] Setting various parameters for the drop-out layer to perform different drop-out processing for each drop-out layer in multiple distributed models can also be referred to as setting different noise for parallel processing of each model.

[0067] The noise setting unit 101 can also manage (store) the drop-off process (e.g., drop-off rate, nodes to be deactivated) set as actual implementation information to each distributed model, refer to the actual implementation information, and determine the drop-off process to be set to each distributed model so that the drop-off process is not repeated among multiple distributed models.

[0068] The training control unit 102 assigns each distributed model set by the model management unit 100 to each computing node 2 so that each distributed model performs training (machine learning).

[0069] According to instructions from the training control unit 102 for performing machine learning, multiple computing nodes 2 perform machine learning on multiple distributed neural network models created by parallel partitioning of the original neural network.

[0070] Each distributed model assigned to each compute node 2 includes a drop-out layer within a non-parallel block (repeated block). Therefore, as each of the multiple compute nodes 2 executes the machine learning process of the distributed model, different noise is added to each non-parallel processing block (repeated block, drop-out layer).

[0071] The inference control unit 103 enables each computing node 2 to perform inference through a distributed model.

[0072] (B) Operation

[0073] According to Figure 4 The flowcharts shown (steps S1 to S8) illustrate the processing of the model management unit 100 of the computer system 1, which is an example of an implementation of the configuration described above.

[0074] In step S1, the model management unit 100 reads information about the pre-created distributed models. For example, the model management unit 100 reads information about multiple distributed models created from the original neural network.

[0075] In step S2, the model management unit 100 selects a distributed model from multiple read distributed models, confirms the processing blocks sequentially starting from the beginning of the corresponding distributed model, and searches for duplicate blocks that are repeated among multiple distributed models (model parallel processes).

[0076] In step S3, the model management unit 100 confirms whether there are duplicate blocks (candidates). If duplicate blocks exist (refer to the path in step S3), the process proceeds to step S4. In step S4, the noise setting unit 101 confirms whether noise can be set for the corresponding duplicate block. In other words, for example, the noise setting unit 101 confirms whether the corresponding duplicate block is a discard layer.

[0077] As a result of the confirmation, if the corresponding repeating block can be configured with different noise for each model parallel process, in other words, for example, if the repeating block is a discard layer (refer to the path in step S4), the process proceeds to step S5.

[0078] In step S5, the noise setting unit 101 confirms whether the user has set different noises among multiple distributed models. For example, the noise setting unit 101 may also display a message asking the user whether different noises can be set among multiple distributed models on a display (not shown) or the like.

[0079] Users can use a mouse or keyboard (neither shown) to input responses to queries.

[0080] In step S6, the noise setting unit 101 confirms whether the user agrees to set different noise levels among the multiple distributed models. For example, the noise setting unit 101 confirms whether the user has used a mouse or keyboard to input an instruction indicating that the user agrees to set different noise levels among the multiple distributed models. As a result of the confirmation, if the user does not agree to set different noise levels among the multiple distributed models (refer to the no path in step S6), the process returns to step S2.

[0081] On the other hand, if the user agrees to set different noise levels among multiple distributed models (refer to the path in step S6), the process proceeds to step S7.

[0082] In step S7, the noise setting unit 101 sets (rewrites) the parameters of each dropout layer corresponding to the corresponding dropout layer in the multiple distributed models, so that the dropout process is different from each other. After that, the process returns to step S2.

[0083] Furthermore, if it is impossible to set noise as the result of confirmation in step S4 for the corresponding repeating block, in other words, for example, if the corresponding repeating block is not a discard layer (refer to the no path in step S4), the process returns to step S2.

[0084] Furthermore, if there is no duplicate block as the result of confirmation in step S3 (refer to the no path in step S3), the process proceeds to step S8.

[0085] In step S8, the configuration information for each distributed model is written (stored) in a predetermined storage area, such as storage device 13. The process then ends.

[0086] Note that in the flowchart above, the processes in steps S5 and S6 can also be omitted. In other words, for example, without confirming whether the user has set different noise levels across multiple distributed models, the parameters of each corresponding dropout layer can be rewritten in step S7 to execute dropout processes different from each other.

[0087] Next, we will refer to Figures 5 to 7 Describes machine learning processing using multiple distributed models created by computer system 1.

[0088] exist Figures 5 to 7 The example shown illustrates the implementation of model parallelization, which divides the original neural network into three distributed models and processes the three distributed models through processes #0 to #2.

[0089] Notice, Figure 5 The forward propagation process is shown. Figure 6 The backpropagation process is shown, and Figure 7 The weight update process is illustrated. Furthermore, Figure 5 The direction from top to bottom indicates the forward propagation data flow.

[0090] In forward propagation, such as Figure 5 As shown, the outputs of the individual processes executed by the model parallelization units of the distributed model, which are executed by processes #0 through #2 respectively, are combined (see figure P1). Each combined output is input to each non-model parallelization unit of the distributed model executed by each of processes #0 through #2. The same data is input to each non-model parallelization unit of each distributed model.

[0091] exist Figure 5 In the example shown, each non-model parallelization unit of each distributed model includes three processing blocks (repeated blocks), which include a discard layer (refer to figure labels P2 to P4).

[0092] Furthermore, different parameters are set for the three discard layers by the noise setting unit 101, and therefore, the discard rates of each discard layer are different.

[0093] Therefore, in the processing blocks downstream of these discard layers in the non-model parallelization unit of each distributed model, outputs different from each other are obtained.

[0094] Furthermore, the outputs of each processing block at the final stage of the non-model parallelization unit of each distributed model are combined (see attached figure P5).

[0095] Each combined output is fed into each subsequent model parallelization unit in each of the distributed models executed from process #0 to process #2. The same data is fed into each non-model parallelization unit of each distributed model.

[0096] Figure 6 The direction from bottom to top indicates the direction of the backpropagation data flow.

[0097] In backpropagation, such as Figure 6 As shown, the outputs of the individual processes executed by the model parallelization units of the distributed model, which are executed by processes #0 through #2 respectively, are combined (see figure numeral P6). Each combined output is input to each non-model parallelization unit of the distributed model executed by each of processes #0 through #2. The same data is input to each non-model parallelization unit of each distributed model.

[0098] In each non-model parallelization unit of the distributed model, in each processing block (repeated block) except for the dropout layer (see figure labels P7 to P9), for example, using gradient descent, weights Δw are computed in the direction used to reduce the loss function, which limits the error between the machine learning model's inferences about the training data and the correct answer data.

[0099] The noise setting unit 101 sets different parameters for each dropout layer included in the non-model parallelization unit of each distributed model, and therefore, the dropout rates of each dropout layer are different.

[0100] Therefore, in the processing blocks downstream of these discard layers in the non-model parallelization unit of each distributed model, outputs different from each other are obtained.

[0101] The outputs of the individual processing blocks at the final stage of the non-model parallelization units of each distributed model are combined (see attached figure, P10). Each combined output is input to each subsequent model parallelization unit in each of the distributed models executed from process #0 to process #2. The same data is input to each non-model parallelization unit of each distributed model.

[0102] In weight updates, such as Figure 7 As shown, each weight Δw, computed via backpropagation by each non-model parallelization unit of the distributed model executed by processes #0 to #2, is combined, and the combined weights (combined Δw) are used to update the weights of each processing block. The combination of weights Δw could also be, for example, the calculation of an average, and can be modified and executed as appropriate.

[0103] (C) Effect

[0104] In this way, according to the computer system 1 as an example of the implementation, the noise setting unit 101 sets various parameters for configuring the drop-out layer to perform different drop-out processes for each drop-out layer of multiple distributed models.

[0105] Therefore, in machine learning, each process used to process the distributed model performs different discarding processes in each discarding layer (repeating block) of the non-model parallelization unit.

[0106] Therefore, computational resources can be used efficiently by generating noise in a different way for each process in the non-model parallelization unit used to perform repetitive processing between processes.

[0107] Furthermore, by adding different noises to each distributed model, the robustness of the distributed models in parallel processing can be improved, as well as the learning accuracy.

[0108] The processing blocks included in the non-model parallelization unit are processed raw and repeatedly in parallel by multiple processes (distributed models). Therefore, in computer system 1, the computation time for performing discard processing by each of the multiple processes used to perform model parallelization is almost negligible. Learning accuracy can be improved.

[0109] (D) Other

[0110] Each configuration and each process in this embodiment can be selected or omitted as needed, or they can be combined appropriately.

[0111] Furthermore, the disclosed technology is not limited to the above-described embodiments, and various modifications can be made and implemented without departing from the spirit of these embodiments.

[0112] For example, in the above embodiment, the discard layer is used as a repeating block that can be set with noise. However, the embodiment is not limited to this and can be appropriately modified and implemented.

[0113] Furthermore, this embodiment can be implemented and manufactured by those skilled in the art based on the above disclosure.

Claims

1. A non-transitory computer-readable recording medium for storing a machine learning program, the machine learning program controlling machine learning of multiple distributed neural network models generated by dividing a neural network and executed in parallel on multiple information processing devices, the machine learning program including instructions for causing a processor to perform processing, the processing including: In each of the plurality of distributed neural network models, by setting the parameter of the dropout layer in the non-parallel processing block of the distributed neural network model, individual noise for the distributed neural network model is added. The parameter makes the individual noise added by the dropout layer in the non-parallel processing block of the distributed neural network model different from the individual noise of other distributed neural network models among the plurality of distributed neural network models. as well as The plurality of distributed neural network models with added individual noise are assigned to corresponding plurality of processes processed in parallel by the plurality of information processing devices, such that each of the plurality of processes performs the machine learning on the assigned distributed neural network model among the plurality of distributed neural network models with added individual noise.

2. An information processing apparatus, said information processing apparatus controlling machine learning of multiple distributed neural network models generated by dividing a neural network, said information processing apparatus comprising: Multiple information processing devices are configured to execute the multiple distributed neural network models in parallel; as well as Processing unit, the processing unit being configured to: In each of the plurality of distributed neural network models, by setting the parameter of the dropout layer in the non-parallel processing block of the distributed neural network model, individual noise for the distributed neural network model is added. The parameter makes the individual noise added by the dropout layer in the non-parallel processing block of the distributed neural network model different from the individual noise of other distributed neural network models among the plurality of distributed neural network models. as well as The plurality of distributed neural network models with added individual noise are assigned to corresponding plurality of processes processed in parallel by the plurality of information processing devices, such that each of the plurality of processes performs the machine learning on the assigned distributed neural network model among the plurality of distributed neural network models with added individual noise.

3. A computer-implemented method, said method controlling machine learning of multiple distributed neural network models generated by dividing a neural network and executed in parallel on multiple information processing devices, said machine learning program including instructions for causing a processor to perform processing, said processing including: In each of the plurality of distributed neural network models, by setting the parameter of the dropout layer in the non-parallel processing block of the distributed neural network model, individual noise for the distributed neural network model is added. The parameter makes the individual noise added by the dropout layer in the non-parallel processing block of the distributed neural network model different from the individual noise of other distributed neural network models among the plurality of distributed neural network models. as well as The plurality of distributed neural network models with added individual noise are assigned to corresponding plurality of processes processed in parallel by the plurality of information processing devices, such that each of the plurality of processes performs the machine learning on the assigned distributed neural network model among the plurality of distributed neural network models with added individual noise.

Images