Machine learning program, information processing device and machine learning method

Abstract

To allow efficient use of an operation resource, in machine learning of a plurality of distributed neural network models in which model parallel processing is performed.SOLUTION: In machine learning of a plurality of distributed neural network models which is created by dividing a neural network, and in which model parallel processing is performed in a plurality of processes, a different noise is given for each of the plurality of processes, in a non-parallel processing block processed in each of the plurality of processes.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a machine learning program, an information processing device, and a machine learning method. [Background technology]

[0002] In recent years, in machine learning in neural networks, there is a demand for faster learning due to the increasing size of machine learning models.

[0003] For example, in a simulation using CosmoFlow to estimate cosmological parameters from dark matter data, the amount of data was 5.1TB, and it took a week of machine learning on a single V100 GPU (Graphics Processing Unit).

[0004] In addition, data parallelism, which is the mainstream method for speeding up machine learning, has limitations in terms of accuracy. In other words, increasing the degree of parallelism increases the batch size, which may have a negative effect on learning accuracy.

[0005] In recent years, a model parallelization method has been known in which a machine learning model in a neural network is divided and processed in parallel on multiple computers. Hereinafter, a machine learning model in a neural network may be referred to as a neural network model or simply as a model.

[0006] By dividing the neural network model and processing each model in parallel on multiple computers, machine learning can be accelerated without affecting learning accuracy. FIG. 8 is a diagram for explaining a conventional model parallel method in a neural network.

[0007] In Fig. 8, symbol A indicates a neural network model without parallelization. Symbol B indicates a neural network model with parallelization, which represents two models (processes #0 and #1) created by dividing one model indicated by symbol A.

[0008] In the model-parallelized neural network shown by symbol B, all layers of the convolutional layer and the fully connected layer of the neural network shown by symbol A are divided and parallelized.

[0009] However, in the model-parallelized neural network shown in B, frequent communication (allgather, allreduce) occurs between process #0 and process #1 before and after each layer. This increases the communication load and causes delays due to synchronization waits. Therefore, a method can be considered in which only the convolutional layer, which requires a large amount of calculations, is parallelized among the multiple layers in a neural network. FIG. 9 is a diagram for explaining a conventional model parallel method in a neural network. FIG. 9 shows a model-parallelized neural network created based on the neural network model without parallelization shown by symbol A in FIG.

[0010] In the neural network shown in Fig. 9, only the convolution layer of the neural network model without parallelization shown by symbol A in Fig. 8 is divided into two. That is, the processing of the convolution layer is performed in parallel by processes #0 and #1, and the processing of the fully connected layer is performed only by process #0.

[0011] In general, the processing in the convolutional layer requires a large amount of calculation, but since communication is limited to data exchange between adjacent parts, there are few disadvantages to dividing the convolutional layer. Also, since the number of neurons in the fully connected layer at the latter stage is small, the calculation time does not increase significantly even without parallelization, and processing can be faster than model parallelization. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] Special table 2017-514251 publication [Patent Document 2] US Patent Application Publication No. 2020 / 0372337 Summary of the Invention [Problem to be solved by the invention]

[0013] However, in the conventional model-parallelized neural network shown in FIG. 9, process #1 does not perform any processing other than the convolutional layer, which results in wasting computational resources and is inefficient.

[0014] Also, in the model-parallelized neural network shown in Fig. 9, data communication is performed from process #0 to process #1 in order to share the loss finally calculated in process #0 with process #1. In order to reduce the time required for such data communication, it is conceivable that process #1 also calculates the loss by performing calculations on each fully connected layer in the same way as process #0. However, in this case, the same calculations on the fully connected layer are duplicated in process #0 and process #1, which is inefficient. In one aspect, the present invention aims to enable efficient use of computing resources in machine learning of multiple distributed neural network models that are processed in model parallelism. [Means for solving the problem]

[0015] For this reason, this machine learning program was created by splitting the neural network. Complex In machine learning of distributed neural network models of numbers, A non-model parallel unit that is not parallelized and is provided in each of the distributed neural network models, The plurality of Distributed Neural Network Model Every 、 Different noise of Grant Do The process is executed by the processor. The distributed neural network model includes a model parallel portion and the non-model parallel portion that are processed in parallel, and the application of noise is performed by deactivating specific nodes of the non-model parallel portion. Effect of the Invention

[0016] According to one embodiment, it is possible to efficiently utilize computing resources in machine learning of multiple distributed neural network models that are processed in model parallelism. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of a computer system according to an embodiment; [Diagram 2] FIG. 2 is a functional configuration diagram of a management device of a computer system according to an embodiment. [Diagram 3] FIG. 2 is a conceptual diagram illustrating a neural network model generated by a computer system as an example of an embodiment. [Figure 4] 11 is a flowchart illustrating a process performed by a model management unit of a computer system according to an embodiment. [Diagram 5] FIG. 1 is a diagram for explaining machine learning processing using multiple distribution models created by a computer system as an example of an embodiment. [Figure 6] FIG. 1 is a diagram for explaining machine learning processing using multiple distribution models created by a computer system as an example of an embodiment. [Figure 7] FIG. 1 is a diagram for explaining machine learning processing using multiple distribution models created by a computer system as an example of an embodiment. [Figure 8] FIG. 1 is a diagram for explaining a conventional model parallel method in a neural network. [Figure 9] FIG. 1 is a diagram for explaining a conventional model parallel method in a neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the embodiments of the present machine learning program, information processing device, and machine learning method will be described with reference to the drawings. However, the embodiments shown below are merely examples, and there is no intention to exclude the application of various modified examples and technologies not explicitly stated in the embodiments. In other words, the present embodiment can be implemented with various modifications within the scope of its purpose. In addition, each figure is not intended to include only the components shown in the figure, but can include other functions, etc.

[0019] (A) Configuration FIG. 1 is a diagram showing a schematic hardware configuration of a computer system 1 as an example of an embodiment, and FIG. 2 is a functional configuration diagram of a management device thereof.

[0020] 1, the computer system 1 includes a management device 10 and a plurality of computing nodes 2. The management device 10 and each computing node 2 are connected to each other so as to be able to communicate with each other via a network 3. The network 3 is, for example, a LAN (Local Area Network). In this computer system 1, a machine learning model (neural network model) in a neural network is divided to realize model parallel processing by a plurality of computing nodes 2.

[0021] The computation nodes 2 are information processing devices (computers) equipped with a processor and memory (not shown), and each executes a process assigned by a management device 10 (described later). Each computation node 2 trains an assigned neural network model (machine learning) and performs inference using the neural network model.

[0022] The management device 10 is, for example, an information processing device (computer) having a server function, and has a function of managing a neural network model. As shown in FIG. 1, the management device 10 includes, for example, a processor 11, a memory 12, and a storage device 13.

[0023] The storage device 13 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a storage class memory (SCM), and stores various data.

[0024] The memory 12 is a storage memory including a ROM (Read Only Memory) and a RAM (Random Access Memory). A software program related to the management of the machine learning model and data for this program are written in the ROM of the memory 12. The software program related to the management of the machine learning model includes a machine learning program.

[0025] The software programs on the memory 12 are appropriately read and executed by the processor 11. The RAM of the memory 12 is used as a primary storage memory or a working memory.

[0026] The processor (processing unit) 11 controls the entire management device 10. The processor 11 may be a multiprocessor. The processor 11 may be, for example, any one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). The processor 11 may also be a combination of two or more types of elements from the CPU, MPU, DSP, ASIC, PLD, and FPGA.

[0027] The processor 11 executes the control program to function as the model management unit 100, the training control unit 102, and the inference control unit 103 shown in Fig. 2. The control program includes a machine learning program. The processor 11 executes this machine learning program to realize the function of the training control unit 102.

[0028] The programs (control programs) for implementing the functions of the model management unit 100, the training control unit 102, and the inference control unit 103 are provided in a form recorded on a computer-readable recording medium, such as a flexible disk, a CD (CD-ROM, CD-R, CD-RW, etc.), a DVD (DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, HD DVD, etc.), a Blu-ray disk, a magnetic disk, an optical disk, a magneto-optical disk, etc. Then, the computer reads the programs from the recording medium, transfers them to an internal storage device or an external storage device, stores them, and uses them. The programs may also be recorded on a storage device (recording medium), such as a magnetic disk, an optical disk, a magneto-optical disk, etc., and provided to the computer from the storage device via a communication path.

[0029] When the functions of the model management unit 100, the training control unit 102, and the inference control unit 103 are realized, a program stored in an internal storage device (memory 12 in this embodiment) is executed by a microprocessor (processor 11 in this embodiment) of a computer. At this time, the computer may read and execute a program recorded on a recording medium. The model management unit 100 manages the neural network model. In this computer system 1, a neural network model is divided and model parallel processing is realized by a plurality of computing nodes 2. FIG. 3 is a conceptual diagram illustrating a neural network model generated by the computer system 1. As shown in FIG.

[0030] In the example shown in Fig. 3, two neural network models are created by dividing one neural network model. In addition, in the neural network model shown in Fig. 3, only the convolution layer is divided and parallelized, and the other layers are not parallelized.

[0031] Hereinafter, each of the multiple neural network models created by dividing one neural network model may be referred to as a distributed neural network model or a distributed model. Also, one neural network model before being divided may be referred to as an original neural network model.

[0032] Each of the created distributed models is processed by a different computation node 2. That is, each distributed model is processed as a different process. In FIG. 3, an example is shown in which two distributed models are created, and process #0 and process #1 each process one distributed model. Each distributed model illustrated in Fig. 3 includes a convolutional layer and a fully connected layer. The fully connected layer also includes a dropout layer.

[0033] The dropout layer suppresses overlearning by performing machine learning while deactivating (disabling) a certain percentage of nodes. In the example shown in Fig. 3, the dropout layer is provided in the fully connected layer, but this is not limited to this, and the dropout layer may be provided in the convolution layer, etc.

[0034] In the computer system 1, the dropout layer performs different deactivation (invalidation) between multiple processes (two in the example shown in FIG. 3). Hereinafter, deactivating a specific node in the dropout layer may be referred to as adding noise. The dropout layer functions as a noise adding layer that adds noise to machine learning.

[0035] The model management unit 100 generates a parallelized neural network model as exemplified in Fig. 3. For example, the model management unit 100 adds noise to each of a plurality of distributed models constituting the parallelized neural network model, the noise being different from the dropout layers provided in the other distributed models.

[0036] The convolution layer in the original neural network is divided into process #0 and process #1, and processed in parallel by different computation nodes 2. In each distributed model, the convolution layer may be called a model parallel part. Also, in each distributed model, the fully connected layer and the dropout layer that are not processed in parallel by process #0 and process #1 may be called a non-model parallel part. Furthermore, multiple processes that perform processing of the convolution layer in parallel may be called model parallel processes.

[0037] The non-model parallel part of each distributed model is provided with a processing block that performs the same processing. Such processing blocks are provided in duplicate in the non-model parallel parts of multiple distributed models. In this way, processing blocks provided in duplicate in the non-model parallel parts of multiple distributed models may be called duplicate blocks. A duplicate block is a group of layers that perform duplicate processing without performing model parallelism between processes that perform model parallelism. In each distributed model, the dropout layer is included in the duplicate block. As shown in FIG. 2, the model management unit 100 has a function as a noise setting unit 101.

[0038] The noise setting unit 101 sets various parameters constituting the dropout layer so that different dropout processes are performed for each of the dropout layers of the multiple distributed models.

[0039] For example, the noise setting unit 101 may set a different ratio of nodes to be deactivated (hereinafter, referred to as a dropout rate) for each distributed model. In order to set a different dropout rate for each distributed model, for example, an arbitrary dropout rate may be selected from among a plurality of types of dropout rates using a random number for each dropout layer of each distributed model, and the dropout rate may be appropriately changed and implemented.

[0040] In addition, the noise setting method by the noise setting unit 101 is not limited to differentiating the dropout rate for each distribution model, and can be modified as appropriate. For example, the nodes to be deactivated may be different for each distribution model, and the probability of dropping out an input element may be different for each distribution model.

[0041] The noise setting unit 101 reads data constituting a distributed model, and judges the presence or absence of a dropout layer for each processing block constituting the distributed model. If the distributed model includes a dropout layer, the noise setting unit 101 sets parameters for each dropout layer so that different dropout processing is performed between multiple distributed models.

[0042] Setting various parameters constituting the dropout layer so that different dropout processing is performed for each dropout layer of multiple distributed models can be said to be setting a different noise for each model parallel process.

[0043] The noise setting unit 101 may manage (store) the dropout processing (e.g., dropout rate, nodes to be deactivated) set for each distributed model as performance information, and may refer to this performance information to determine the dropout processing to be set for each distributed model so that the dropout processing is not duplicated among multiple distributed models. The training control unit 102 assigns each of the distributed models set by the model management unit 100 to each of the computation nodes 2 and causes them to perform training (machine learning).

[0044] In accordance with an instruction to execute machine learning from the training control unit 102, the multiple computation nodes 2 perform machine learning in parallel on multiple distributed neural network models created by dividing the original neural network.

[0045] The distributed model assigned to each computation node 2 includes a dropout layer in each non-parallel block (redundant block). Therefore, when multiple computation nodes 2 each perform a machine learning process of the distributed model, different noise is added to each non-parallel processing block (redundant block, dropout layer). The inference control unit 103 causes each calculation node 2 to perform inference based on the distributed model.

[0046] (B) Operation The process performed by the model management unit 100 of the computer system 1 as one example of the embodiment configured as described above will be described with reference to the flowchart (steps S1 to S8) shown in FIG.

[0047] In step S1, the model management unit 100 reads information constituting a distributed model created in advance. For example, the model management unit 100 reads information on a plurality of distributed models created from an original neural network.

[0048] In step S2, the model management unit 100 selects one of the multiple distributed models that have been read, checks the processing blocks of the selected distributed model, starting from the top of the model, and searches for overlapping blocks that overlap between multiple distributed models (model parallel processes).

[0049] In step S3, the model management unit 100 checks whether there is an overlapping block (whether there is a candidate), and if there is an overlapping block (see the YES route in step S3), the process proceeds to step S4. In step S4, the noise setting unit 101 checks whether the overlapping block can have noise set. That is, the noise setting unit 101 checks whether the overlapping block is a dropout layer.

[0050] As a result of the check, if the overlapping block is capable of setting different noise for each model parallel process, that is, if the overlapping block is a dropout layer (see the YES route of step S4), the process proceeds to step S5.

[0051] In step S5, the noise setting unit 101 confirms with the user whether or not to set different noises between the multiple distribution models. For example, the noise setting unit 101 may display a message on a display (not shown) or the like, inquiring of the user whether or not to set different noises between the multiple distribution models. The user may use a mouse or keyboard (neither of which are shown) to input a response to this inquiry.

[0052] In step S6, the noise setting unit 101 checks whether the user agrees to set different noises between multiple distribution models. For example, the noise setting unit 101 checks whether the user has input an indication of agreement to set different noises between multiple distribution models using a mouse or keyboard. If the result of this check shows that the user does not agree to set different noises between multiple distribution models (see the NO route in step S6), the process returns to step S2.

[0053] On the other hand, if the user agrees to set different noises among a plurality of distribution models (see the YES route in step S6), the process proceeds to step S7.

[0054] In step S7, the noise setting unit 101 sets (rewrites) the parameters of each dropout layer corresponding to the dropout layer in the plurality of distributed models so that different dropout processes are performed. Then, the process returns to step S2.

[0055] Also, if the check result in step S4 shows that noise cannot be set for the overlapping block, that is, if the overlapping block is not a dropout layer (see the NO route in step S4), the process returns to step S2. If it is determined in step S3 that there are no overlapping blocks (see the NO route in step S3), the process proceeds to step S8. In step S8, the information constituting each distribution model is written (stored) in a predetermined storage area such as the storage device 13. After that, the process ends.

[0056] In the above-mentioned flowchart, the processes of steps S5 and S6 may be omitted. That is, without asking the user whether or not to set different noises between multiple distribution models, the parameters of the corresponding dropout layers may be rewritten in step S7 so that different dropout processes are performed. Next, machine learning processing using a plurality of distributed models created by the computer system 1 will be described with reference to FIGS.

[0057] 5 to 7 show an example of implementing model parallelism in which the original neural network is divided into three distributed models and processed by processes #0 to #2.

[0058] Note that Fig. 5 shows the forward propagation process, Fig. 6 shows the backpropagation process, and Fig. 7 shows the weight update process. In Fig. 5, the direction from top to bottom indicates the data flow in the forward propagation.

[0059] In forward propagation, as shown in Fig. 5, the outputs of the processes performed in the model parallel parts of the distributed models executed by the processes #0 to #2 are combined (see symbol P1). The combined outputs are input to the non-model parallel parts of the distributed models executed by the processes #0 to #2. The same data is input to the non-model parallel parts of the distributed models.

[0060] In the example shown in FIG. 5, each of the non-model parallel parts of the distributed models includes three processing blocks (redundant blocks) including a dropout layer (see symbols P2 to P4).

[0061] Furthermore, different parameters are set for these three dropout layers by the above-mentioned noise setting unit 101, and as a result, the dropout rates of the respective dropout layers are different. Therefore, in the processing blocks downstream of these dropout layers in the non-model parallel parts of each distributed model, different outputs are obtained. Furthermore, the outputs of the processing blocks at the final stage of the non-model parallel portion of each distributed model are combined (see symbol P5).

[0062] The combined outputs are input to the subsequent model parallel parts in the distributed models executed by the processes #0 to #2, respectively. The same data is input to the non-model parallel parts of each distributed model. In FIG. 6, the direction from bottom to top in the figure indicates the direction of data flow for backpropagation.

[0063] In back propagation, as shown in Fig. 6, the outputs of the processes performed in the model parallel parts of the distributed models executed by the processes #0 to #2 are combined (see symbol P6). The combined outputs are input to the non-model parallel parts of the distributed models executed by the processes #0 to #2. The same data is input to the non-model parallel parts of the distributed models.

[0064] In the non-model parallel part of each distributed model, in each processing block (overlapping block) other than the dropout layer (see symbols P7 to P9), a weight Δw is calculated using, for example, gradient descent in the direction of reducing the loss function that defines the error between the inference result of the machine learning model for the training data and the ground truth data.

[0065] In each dropout layer provided in the non-model parallel portion of each distributed model, different parameters are set by the above-mentioned noise setting unit 101, and as a result, the dropout rate of each dropout layer is different. Therefore, in the processing blocks downstream of these dropout layers in the non-model parallel parts of each distributed model, different outputs are obtained.

[0066] The outputs of the processing blocks at the last stage of the non-model parallel part of each distributed model are combined (see symbol P10). Each combined output is input to the subsequent model parallel part in the distributed model executed by each of processes #0 to #2. The same data is input to each of the non-model parallel parts of each distributed model.

[0067] In weight updating, as shown in Fig. 7, the weights Δw calculated by back propagation in the non-model parallel parts of the distributed models executed by the processes #0 to #2 are combined, and the weights of the processing blocks are updated using the combined weights (combined Δw). The combination of the weights Δw may be, for example, a calculation of an average value, and can be implemented by modifying it appropriately.

[0068] (C) Effects In this manner, according to the computer system 1 as an example of the embodiment, the noise setting unit 101 sets various parameters constituting the dropout layer so that different dropout processing is performed for each dropout layer of a plurality of distributed models.

[0069] As a result, during machine learning, in each process processing a distributed model, different dropout processing is performed in each dropout layer in the non-model parallel part (overlapping block).

[0070] Therefore, in a non-model parallel portion in which processes perform overlapping processing, noise is generated using a different method for each process, thereby making it possible to use computation resources efficiently. In addition, by adding different noise to each distributed model, the robustness of the distributed models processed in parallel can be improved, and the learning accuracy can be improved.

[0071] The processing blocks constituting the non-model parallel part are originally processed in parallel and overlapping manner in multiple processes (distributed model). Therefore, in this computer system 1, there is almost no increase in calculation time when the dropout processing is performed in each of the multiple processes performing model parallelism. This can improve the learning accuracy.

[0072] (D) Other The configurations and processes of this embodiment may be selected as necessary, or may be combined as appropriate. The disclosed technology is not limited to the above-described embodiment, but can be implemented in various modifications without departing from the spirit of the present embodiment.

[0073] For example, in the above-described embodiment, a dropout layer is used as an overlapping block for which noise can be set, but this is not limited thereto and can be modified as appropriate. Furthermore, the above disclosure enables a person skilled in the art to implement and manufacture the present embodiment.

[0074] (E) Notes The following supplementary notes are further disclosed regarding the above embodiment. (Appendix 1) In machine learning of multiple distributed neural network models, which are created by dividing the neural network and processing the models in parallel in multiple processes, In a non-parallel processing block processed in each of the plurality of processes, a different noise is added to each of the plurality of processes. A machine learning program that causes a processor to execute processing.

[0075] (Appendix 2) The non-parallel processing block is a dropout layer, and different dropout processing is performed for each process. 2. The machine learning program of claim 1, which causes the processor to execute a process.

[0076] (Appendix 3) In machine learning of multiple distributed neural network models, which are created by dividing the neural network and processing the models in parallel in multiple processes, In a non-parallel processing block processed in each of the plurality of processes, a different noise is added to each of the plurality of processes. An information processing device comprising a processing unit.

[0077] (Appendix 4) The processing unit, The non-parallel processing block is a dropout layer, and different dropout processing is performed for each process. 4. An information processing device according to claim 3.

[0078] (Appendix 5) In machine learning of multiple distributed neural network models, which are created by dividing the neural network and processing the models in parallel in multiple processes, In a non-parallel processing block processed in each of the plurality of processes, a different noise is added to each of the plurality of processes. A machine learning method characterized in that processing is executed by a computer.

[0079] (Appendix 6) The non-parallel processing block is a dropout layer, and different dropout processing is performed for each process. 6. The machine learning method according to claim 5, wherein the processing is executed by the computer. [Explanation of symbols]

[0080] 1. Computer Systems 2 Compute Node 10 Management device 11 Processor (processing unit) 12. Memory 13 Storage device 100 Model Management Department 101 Noise setting section 102 Training Control Unit 103 Inference control section

Claims

1. In machine learning of a plurality of distributed neural network models created by dividing a neural network, a processor is caused to execute a process of adding different noise to a non-model parallel portion that is not subjected to parallel processing and that is provided in each of the distributed neural network models, for each of the plurality of distributed neural network models; the distributed neural network model includes a model parallel portion that is processed in parallel and the non-model parallel portion, The noise is added by deactivating a specific node of the non-model parallel part. A machine learning program characterized by:

2. The process of adding different noises is performed on a dropout layer included in the non-model parallel portion, The dropout layer functions as a layer that adds noise to the distributed neural network model. The machine learning program according to claim 1 .

3. The process of applying different noises includes varying a ratio of nodes to be inactivated for each of the plurality of distributed neural network models. The machine learning program according to claim 1 or 2.

4. In machine learning of a plurality of distributed neural network models created by dividing a neural network, a processing unit is provided for adding different noises to a non-model parallel unit that is not subjected to parallel processing and is provided in each of the distributed neural network models, for each of the plurality of distributed neural network models; the distributed neural network model includes a model parallel portion that is processed in parallel and the non-model parallel portion, 2. The information processing apparatus according to claim 1, wherein the noise is added by deactivating a specific node of the non-model parallel portion.

5. In machine learning of a plurality of distributed neural network models created by dividing a neural network, a computer executes a process of adding different noise to a non-model parallel portion that is not subjected to parallel processing and is provided in each of the distributed neural network models, for each of the plurality of distributed neural network models; the distributed neural network model includes a model parallel portion that is processed in parallel and the non-model parallel portion, The noise is added by deactivating a specific node of the non-model parallel part. A machine learning method comprising:

Images