Information processing device, information processing system, information processing method, and program

Abstract

To provide a device that allows reduction of an operation cost of a neural network.SOLUTION: An information processing device includes one or a plurality of memories, one or a plurality of first processors, and one or a plurality of second processors. The one or plurality of first processor executes first processing of transferring information on a parameter of a neural network between the one or plurality of second processors. The one or plurality of second processors executes second processing for updating the parameter of the neural network at a processing speed according to the execution time of the first processing.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] There are known techniques for constructing neural networks based on deep learning. In order to speed up deep learning, distributed deep learning, which executes learning processing in parallel within a node or in parallel between nodes, is sometimes used. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-18220 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an apparatus capable of reducing the operating costs of a neural network. [Means for solving the problem]

[0005] An information processing device according to one embodiment of the present disclosure includes one or more memories, one or more first processors, and one or more second processors, where the one or more first processors execute a first process for transferring information regarding parameters of a neural network between the one or more second processors, and the one or more second processors execute a second process for updating the parameters of the neural network at a processing speed corresponding to the execution time of the first process. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing an example of a system configuration of a neural network system. [Diagram 2] FIG. 13 is a diagram illustrating an example of a learning process by a single node. [Diagram 3] FIG. 11 illustrates an example of a learning process performed by a plurality of nodes. [Figure 4] FIG. 11 is a diagram illustrating an example of an execution time of a learning process in the prior art. [Diagram 5] FIG. 11 is a diagram illustrating an example of an execution time of a learning process in the embodiment. [Figure 6] 1 is a graph showing an example of a relationship between a clock frequency and a power supply voltage. [Figure 7] FIG. 11 is a sequence diagram illustrating an example of a learning process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0008] [Embodiment] An embodiment of the present disclosure may be a neural network system that performs information processing using a neural network based on deep learning (hereinafter, simply referred to as a "neural network"). In this embodiment, the neural network system may have a function of constructing a neural network by distributed deep learning.

[0009] Distributed deep learning is a technique in which the calculations for learning one neural network are distributed to multiple computation nodes (hereinafter also simply referred to as "nodes") and executed in parallel. In this embodiment, distributed deep learning includes both intra-node parallelism and inter-node parallelism. Intra-node parallelism means parallel processing by multiple computation components implemented in a node, and inter-node parallelism means parallel processing by multiple nodes implemented with computation components.

[0010] <System configuration> The system configuration of the neural network system in this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the system configuration of the neural network system.

[0011] As shown in Fig. 1, the neural network system 1000 includes a plurality of nodes 10 (10-1, 10-2). Although two nodes 10-1 and 10-2 are shown in Fig. 1 as an example, the number of nodes 10 is not limited. The number of nodes 10 may be one, or three or more.

[0012] The multiple nodes 10-1, 10-2 may be connected to each other so as to be able to communicate data with each other via a communication network N1 such as a local area network (LAN), Ethernet, or the Internet.

[0013] Hereinafter, when multiple nodes 10 are to be distinguished from one another, they will be referred to using sub-numbers such as "node 10-1," "node 10-2," etc. When simply referred to as "node 10," this applies to all nodes 10.

[0014] 1, the node 10-1 may include a processor 11-1, a memory 12-1, an accelerator 13-1, a memory 14-1, a bus switch 15-1, and a network interface 16-1. Similarly, the node 10-2 may include a processor 11-2, a memory 12-2, an accelerator 13-2, a memory 14-2, a bus switch 15-2, and a network interface 16-2.

[0015] The processor 11 may be a processing circuit capable of performing general-purpose calculations. The calculations performed by the processor 11 may include control of other devices or circuits in the node 10, and calculations on data input to the node 10. The processor 11 may be, for example, a central processing unit (CPU).

[0016] The memory 12 may be a storage device that stores data used in the calculations of the processor 11 and data obtained by the calculations of the processor 11. As an example, the memory 12 may be a dynamic random access memory (DRAM).

[0017] The accelerator 13 may be a processing circuit capable of performing a specific operation at high speed. The operation performed by the accelerator 13 includes, for example, a multiply-and-accumulate operation that is often used in training a neural network. For example, the accelerator 13 may be a graphics processing unit (GPU; Graphic Processing Unit), may be a processor with a different design from the processor 11, and may be formed on a chip different from the processor 11.

[0018] The memory 14 may be a storage device that stores data used in the calculations of the accelerator 13 and data obtained by the calculations of the accelerator 13. As an example, the memory 14 may be a dynamic random access memory (DRAM).

[0019] The bus switch 15 may be a switch that connects the processor 11, the accelerator 13, and the network interface 16 via a bus and controls data transfer among the processor 11, the accelerator 13, and the network interface 16. As an example, the bus switch 15 may be a PCIe (Peripheral Component Interconnect Express) switch.

[0020] The network interface 16 may be an interface that connects to the communication network N1 and performs data communication with other nodes 10. The network interface 16 may be, for example, a network interface card (NIC).

[0021] The system configuration of the neural network system 1000 shown in FIG. 1 is an example, and various configuration examples are possible depending on the application and purpose. For example, the node 10 may be one, or may be three or more. For example, the node 10 may have a plurality of processors 11 or accelerators 13. For example, the node 10 may have a plurality of memories 12 or memories 14 according to the number of processors 11 or accelerators 13. For example, the bus switch 15 and the network interface 16 may be integrated, and one may execute a part of the function of the other. The names of the processor, accelerator, memory, bus switch, network interface, etc. shown in FIG. 1 are examples.

[0022] <Background> In recent years, neural networks with a huge number of parameters have been used. One example of a neural network with a huge number of parameters is a deep neural network called a large language model (LLM). A large language model is a language model that is trained in advance using a large amount of text data and can perform various language processing tasks. Examples of language processing tasks that can be performed by a large language model include sentence generation, sentence summarization, machine translation, and question answering.

[0023] For example, assume that there is a large-scale language model with 355 billion parameters. If one parameter is 2 bytes, a memory capacity of 710 gigabytes is required to hold all the parameters of this large-scale language model. Furthermore, to train a large-scale language model, additional memory capacity is required to hold various data other than the parameters.

[0024] On the other hand, semiconductor devices capable of high-speed calculations such as product-sum calculations, which are often used in deep learning, have been used to train neural networks. This type of semiconductor device is sometimes called an "AI (Artificial Intelligence) accelerator (or simply "accelerator").

[0025] It is difficult to store the enormous number of parameters used in large-scale language models in the memory of a single accelerator. It is also difficult to calculate such an enormous number of parameters in a single accelerator in a realistic execution time. For this reason, multiple accelerators are connected in parallel, and neural networks are trained in parallel in a distributed manner on each accelerator.

[0026] When multiple accelerators are connected in parallel, multiple accelerators may be installed in one node, or multiple nodes each equipped with one or more accelerators may be connected. The neural network system 1000 shown in Fig. 1 is an example of the latter.

[0027] <Overview of the learning process> Fig. 2 is a diagram showing an example of a learning process by a single node. As shown in Fig. 2, in the learning process by one node 10, a forward process P1, a backward process P2, and an optimization process P3 are repeatedly executed in that order.

[0028] The forward process P1 is a process of propagating the learning data forward to the neural network to be trained and calculating the error between the inference result and the correct answer data. The backward process P2 is a process of propagating the error obtained in the forward process P1 backward to the neural network and calculating the gradient of each parameter. The optimization process P3 is a process of optimizing each parameter based on the gradient of each parameter obtained in the backward process P2.

[0029] Fig. 3 is a diagram showing an example of a learning process by a plurality of nodes. As shown in Fig. 3, in the learning process by a plurality of nodes 10-1 to 10-3, each of the nodes 10-1 to 10-3 may sequentially execute forward processes P1-1 to P1-3 and backward processes P2-1 to P2-3, and then all the nodes 10-1 to 10-3 may cooperate to execute an all-reduce process P4.

[0030] The all-reduce process P4 is a process for collecting gradients of each parameter obtained in the backward processes P2-1 to P2-3 from each of the nodes 10-1 to 10-3, averaging them, and distributing them to each of the nodes 10-1 to 10-3. After the all-reduce process P4 is completed, each of the nodes 10-1 to 10-3 may execute the optimization processes P3-1 to P3-3, respectively, based on the gradients obtained in the all-reduce process P4.

[0031] In the AllReduce process P4, the amount of data for all parameters may travel back and forth between the nodes 10-1 to 10-3. As an example, if the amount of data for all parameters is 710 gigabytes, the amount of data transferred between the nodes is 1420 gigabytes by simple calculation. Even if the amount of data actually transmitted and received can be reduced by various improvements in the efficiency of data transfer, it is still necessary to transfer a huge amount of data between the nodes.

[0032] The all-reduce process P4 is an example of a process for transferring information about each parameter of the neural network between the processor 11 and the accelerator 13. The forward process P1 and the backward process P2 are examples of a process for learning the parameters of the neural network. Each parameter and the gradient of each parameter are an example of information about each parameter of the neural network.

[0033] <Learning process execution time> The execution time of the learning process by multiple nodes will be described in more detail with reference to Figures 4 and 5. In the following, as an example, a neural network that needs to transfer 130 gigabytes of data between nodes in the all-reduce process P4 is considered. In addition, as an example, the data transfer rate between nodes is 100 GB / s, and the data transfer bandwidth between the processor and the accelerator is 65 GB / s on one side.

[0034] As an example of data transfer between nodes, parameters stored in the accelerator memory are moved to the processor memory, and then transmitted from the processor memory via a bus switch and a network interface.

[0035] Fig. 4 is a diagram showing an example of the execution time of the learning process in the conventional technology. As shown in Fig. 4, in the learning process in the conventional technology, the accelerator 13 performs a forward process P1 and a backward process P2. Also, the processor 11 performs an all-gather process P4-1, a reduce-scatter process P4-2, and an optimize process P3. However, the all-gather process P4-1 and the reduce-scatter process P4-2 are processing units obtained by subdividing the all-reduce process P4.

[0036] The all-gather process P4-1 is a process that collects parameters optimized in the optimization process P3 and distributes them to each node. The reduce-scatter process P4-2 is a process that collects the gradients of the parameters obtained in the backward process P2 from each node and averages the collected gradients for each parameter. The optimization process P3 optimizes each parameter based on the gradients averaged in the reduce-scatter process P4-2.

[0037] The reduce / scatter process P4-2 and the optimize process P3 are executed in parallel in the processor 11. The all-gather process P4-1 in the processor 11 and the forward process P1 in the accelerator 13 can be executed in parallel. The reduce / scatter process P4-2 in the processor 11 and the backward process P2 in the accelerator 13 can be executed in parallel.

[0038] In the example shown in Figure 4, the execution time of the all-gather process P4-1 and the reduce-scatter process P4-2 is 1.3 seconds, respectively, and the execution time of the optimization process P3 is 1.0 seconds. The execution times of the forward process P1 and the backward process P2 are 0.4 seconds and 0.8 seconds, respectively. Furthermore, the transfer times of the parameters and gradients are 0.5 seconds, respectively.

[0039] As shown in Fig. 4, in the learning process of the conventional technology, the execution time of the all-gather process P4-1 is longer than the execution time of the forward process P1. Also, the execution time of the reduce-scatter process P4-2 is longer than the execution time of the backward process P2. Note that in the case of a neural network with a relatively small number of parameters, the execution time of the all-gather process P4-1 is often shorter than the execution time of the forward process P1. Also, the execution time of the reduce-scatter process P4-2 is often shorter than the execution time of the backward process P2.

[0040] The execution time of the forward processing P1 in the accelerator 13 can be extended to the execution time of the all-gather processing P4-1 in the processor 11. Similarly, the execution time of the backward processing P2 in the accelerator 13 can be extended to the execution time of the reduce-scatter processing P4-2 in the processor 11.

[0041] 5 is a diagram showing an example of the execution time of the learning process in the embodiment. As shown in FIG. 5, in the learning process in the embodiment, the execution time of the forward process P1 in the accelerator 13 is extended to 1.3 seconds, which is the same as the execution time of the all-gather process P4-1, for example. Similarly, the execution time of the backward process P2 in the accelerator 13 is extended to 1.3 seconds, which is the same as the execution time of the reduce-scatter process P4-2, for example.

[0042] <How to adjust the processing speed> The execution time of the forward process P1 or the backward process P2 can be extended by reducing the processing speed of the accelerator 13. In this embodiment, the processing speed of the accelerator may be adjusted based on the execution time of the forward process and the all-gather process, or the execution time of the backward process and the reduce-scatter process. The processing speed of the accelerator may also be adjusted based on the number of parameters or the amount of learning data, or the processing speed of the accelerator may be adjusted by combining these.

[0043] Specifically, first, the forward process and the all-gather process may be executed at least once, and their execution times may be measured. Next, the ratio between the execution time of the forward process and the execution time of the all-gather process may be calculated. Then, the processing speed of the accelerator may be changed based on the calculated ratio of execution times. As an example, if the execution time of the forward process is half the execution time of the all-gather process, the execution time of the forward process can be doubled, and therefore the processing speed of the accelerator can be reduced to half.

[0044] The processing speed of an accelerator can be adjusted, for example, as follows. The first adjustment method is to change the clock frequency of the accelerator. For example, if the clock frequency is halved, the processing speed will be halved. Lowering the clock frequency reduces the amount of power required for operation, so the power supply voltage of the accelerator can be lowered. Lowering the power supply voltage of the accelerator can reduce the power consumption of the accelerator.

[0045] Figure 6 is a graph showing an example of the relationship between clock frequency and power supply voltage. The graph shown in Figure 6 shows the clock frequency [MHz] on the vertical axis and the power supply voltage [mV] on the horizontal axis, indicating the clock frequency that can be operated at a certain power supply voltage.

[0046] As shown in Fig. 6, when the clock frequency is reduced, the power supply voltage can be reduced according to the amount of reduction. For example, the power supply voltage for operating the accelerator at a reduced clock frequency can be obtained according to the graph shown in Fig. 6. The relationship between the clock frequency and the power supply voltage may be obtained in advance, for example, through an experiment or the like, or information released by the accelerator manufacturer or the like may be used.

[0047] The second adjustment method is, for example, thinning out or stretching the clock of the accelerator. Thinning out or stretching the clock of the accelerator reduces the number of calculations of the accelerator per unit time, so the processing speed decreases. If the processing speed decreases, the power supply voltage of the accelerator can be reduced, as in the first example, and the power consumption of the accelerator can be reduced.

[0048] <Learning process flow> The learning process executed by the neural network system 1000 will be described with reference to Fig. 7. Fig. 7 is a sequence diagram showing an example of the learning process.

[0049] In step S1, the accelerator 13 of the node 10 may acquire learning data and each parameter of the neural network. For example, the accelerator 13 may read out learning data pre-stored in the memory 14. Also, for example, the accelerator 13 may read out parameters pre-stored in the memory 14, or may generate initial values ​​of the parameters according to a predetermined rule.

[0050] In step S2, the accelerator 13 of the node 10 executes forward processing. Specifically, the accelerator 13 may set each parameter acquired in step S1 in a neural network, and propagate the learning data acquired in step S1 in the forward direction to the neural network. In this way, the accelerator 13 obtains an output result of the neural network for the learning data. Next, the accelerator 13 may calculate an error between the output result and the learning data.

[0051] In step S3, the accelerator 13 of the node 10 may execute backward processing. Specifically, the accelerator 13 may propagate the error calculated in step S2 backward through the neural network. In this way, the accelerator 13 obtains the gradient of each parameter. The accelerator 13 may store the gradient of each parameter in the memory 14.

[0052] In step S4, the processor 11 of the node 10 may execute a reduce scatter process. Specifically, the processor 11 may collect gradients of each parameter from the accelerator 13. The processor 11 may collect gradients from the accelerator 13 mounted on the same node 10, for example, by reading out gradients stored in the memory 14. The processor 11 may also collect gradients from the accelerator 13 mounted on another node 10, for example, by receiving gradients via the network interface 16.

[0053] The processor 11 may then average the collected gradients for each parameter, and the processor 11 may store the averaged gradients in the memory 12.

[0054] In step S5, the processor 11 of the node 10 may execute an optimization process. Specifically, the processor 11 may read out the gradient of each parameter from the memory 12. Then, the processor 11 may optimize each parameter based on the read out gradient. The processor 11 may store each optimized parameter in the memory 12.

[0055] In step S6, the accelerator 13 of the node 10 may adjust the processing speed. The adjustment of the processing speed may be performed autonomously by the node 10, or may be performed manually by an administrator of the neural network system 1000, etc.

[0056] When the processing speed is adjusted autonomously, for example, the processing speed may be adjusted as follows. First, the processor 11 determines the clock frequency and power supply voltage of the accelerator 13 based on the execution time of the backward processing and the execution time of the reduce scatter processing. Next, the processor 11 notifies the accelerator 13 of the determined clock frequency and power supply voltage. Then, the accelerator 13 changes the clock frequency and power supply voltage according to the notified clock frequency and power supply voltage. In this way, the processing speed of the accelerator 13 is adjusted autonomously.

[0057] When adjusting the processing speed manually, for example, the processing speed may be adjusted as follows. First, the processor 11 presents the execution time of the backward processing and the execution time of the reduce-scatter processing to an administrator or the like. For example, the processor 11 may display the execution time of the backward processing and the execution time of the reduce-scatter processing on a display or the like of the node 10. The administrator or the like determines the clock frequency and power supply voltage of the accelerator 13 based on the presented execution times. Then, the administrator or the like sets the determined clock frequency and power supply voltage in the accelerator 13. In this way, the processing speed of the accelerator 13 is manually adjusted.

[0058] In step S7, the processor 11 of the node 10 may execute all-gather processing. Specifically, the processor 11 may first read each parameter of the neural network from the memory 12. Next, the processor 11 may distribute each read parameter to the accelerator 13. The processor 11 may distribute each parameter to the accelerator 13 mounted on the same node 10 by storing each parameter in the memory 14, for example. Also, the processor 11 may distribute each parameter to the accelerator 13 mounted on another node 10 by transmitting each parameter via the network interface 16, for example.

[0059] In step S8, the accelerator 13 of the node 10 may execute forward processing. The contents of the forward processing are the same as those in step S2. However, since the processing speed of the accelerator 13 is adjusted in step S6, the execution time of the forward processing in step S8 is approximately the same as the execution time of the all-gather processing in step S7.

[0060] In step S9, the accelerator 13 of the node 10 may execute backward processing. The content of the backward processing is the same as that in step S3. However, since the accelerator 13 has its processing speed adjusted in step S6, the execution time of the backward processing in step S9 is approximately the same as the execution time of the reduce scatter processing in step S10.

[0061] In step S10, the processor 11 of the node 10 may execute a reduce scatter process. The contents of the reduce scatter process are similar to those of step S4.

[0062] In step S11, the processor 11 of the node 10 may execute an optimization process. The details of the optimization process are similar to those of step S5.

[0063] Thereafter, the node 10 may repeatedly execute steps S7 to S11 until the learning of the neural network is completed.

[0064] With the above configuration, it is possible to reduce the processing speed of the accelerator 13 without delaying the execution time of the entire learning process of the neural network. As a result, it is possible to reduce the power consumption and heat generation of the accelerator 13, and it is possible to reduce the operating cost of the neural network system 1000.

[0065] <Summary> As is clear from the above description, a node 10 according to an embodiment of the present disclosure includes one or more memories, one or more processors 11, and one or more accelerators 13. The processor 11 executes a first process of transferring information related to parameters of a neural network between the processor 11 and the accelerator 13. The accelerator 13 executes a second process for updating the parameters of the neural network at a processing speed according to the execution time of the first process. Note that "according to the execution time" may include determining the processing speed based on a condition that determines the execution time, such as the number of parameters or the amount of training data.

[0066] The accelerator 13 may execute the second process at a slower processing speed as the execution time of the first process becomes longer.

[0067] The first process may include an all-reduce process. The second process may include a forward process or a backward process. The all-reduce process may include an all-gather process and a reduce-scatter process. The accelerator 13 may execute the forward process at a processing speed according to the execution time of the all-gather process, and execute the backward process at a processing speed according to the execution time of the reduce-scatter process.

[0068] The accelerator 13 may reduce the processing speed by lowering the power supply voltage. The accelerator 13 may reduce the power supply voltage by lowering the clock frequency. The accelerator 13 may reduce the power supply voltage by thinning out the clock or stretching the clock.

[0069] As a result, according to one embodiment of the present disclosure, a device capable of reducing the operating costs of a neural network can be provided. For example, when learning a neural network with a huge number of parameters such as a large-scale language model, the processing speed of the accelerator can be reduced according to the execution time of the process of transferring parameters or gradients between the processor and the accelerator. Reducing the processing speed of the accelerator reduces, for example, the power consumption or heat generation of the accelerator. As a result, the operating costs of the neural network system are reduced.

[0070] [Hardware configuration of information processing device] A part or all of each device (node ​​10) in the above-described embodiment may be configured with hardware. When configured with software information processing, software for realizing at least a part of the functions of each device in the above-described embodiment may be stored in a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by reading the software into a computer. The software may also be downloaded via a communication network.

[0071] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed storage medium such as a hard disk, memory, etc. The storage medium may be provided inside the computer (main storage device, auxiliary storage device, etc.) or outside the computer.

[0072] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0073] In this specification (including claims), when expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes the use of data itself or data that has been processed in some way (e.g., data with noise added, normalized, feature values ​​extracted from data, intermediate representations of data, etc.). In addition, when it is stated that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), unless otherwise specified, this includes the use of the result based only on the data, or the use of the result influenced by other data, factors, conditions, and / or states other than the data. In addition, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes the use of data itself as output, or data that has been processed in some way (e.g., data with noise added, normalized, feature values ​​extracted from data, intermediate representations of various data, etc.) as output.

[0074] In this specification (including the claims), the terms "connected" and "coupled" are intended as open-ended terms including any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, physically connection / coupling, etc. The terms should be interpreted appropriately depending on the context in which the terms are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in the terms without any restriction.

[0075] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured / set to actually perform operation B. For example, when element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, when element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure, etc. of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0076] In this specification (including the claims), when terms implying containing or possessing (e.g., "comprising / including," "having," etc.) are used, they are intended to be open-ended terms that include cases in which the term contains or possesses things other than the object indicated by the object of the term. When the object of such terms implying containing or possessing does not specify a quantity or suggests a singular number (such as an expression using the article "a" or "an"), the expression should be construed as not being limited to a specific number.

[0077] In this specification (including the claims), even if expressions such as "one or more" and "at least one" are used in some places and expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0078] In this specification, when a particular advantage / result is described as being obtained from a particular configuration of an embodiment, it should be understood that the same effect can also be obtained from one or more other embodiments having the same configuration, unless there is a reason to the contrary. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states are satisfied, and the effect is not necessarily obtained in the invention according to the claim that specifies the configuration or a similar configuration.

[0079] In this specification (including claims), when a plurality of pieces of hardware perform a predetermined process, each piece of hardware may cooperate to perform the predetermined process, or a portion of the hardware may perform all of the predetermined process. Also, a portion of the hardware may perform a portion of the predetermined process, and another piece of hardware may perform the remainder of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. The hardware may include an electronic circuit, a device including an electronic circuit, and the like.

[0080] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0081] In this specification (including the claims), terms such as "first", "second", etc. are used merely as a way of distinguishing between two or more elements, and are not necessarily intended to impose technical meanings such as time aspect, space aspect, sequence, quantity, etc. on the subject matter. Thus, for example, a reference to a first element and a second element does not necessarily mean that only two elements may be employed therein, that the first element must precede the second element, that the first element must be present for the second element to be present, etc.

[0082] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the above-mentioned individual embodiments. Various additions, modifications, replacements, partial deletions, etc. are possible within the scope of the conceptual idea and intent of the present invention derived from the contents defined in the claims and their equivalents. For example, in the above-mentioned embodiments, when numerical values ​​or formulas are used for explanation, these are shown for illustrative purposes and do not limit the scope of the present disclosure. In addition, the order of each operation shown in the embodiment is also illustrative and does not limit the scope of the present disclosure. [Explanation of symbols]

[0083] 10 nodes 11 Processors 12. Memory 13. Accelerators 14. Memory 15 Bus Switch 16 Network Interfaces 1000 Neural Network System

Claims

1. one or more memories; one or more first processors; one or more second processors; The one or more first processors: performing a first process for transferring information regarding parameters of a neural network to and from the one or more second processors; The one or more second processors: executing a second process for updating a parameter of the neural network at a processing speed corresponding to an execution time of the first process; Information processing device.

2. the one or more second processors are one or more accelerators; The information processing device according to claim 1 .

3. the first process includes an all-reduce process; The second processing includes a forward processing or a backward processing. The information processing device according to claim 1 .

4. The all-reduce process includes an all-gather process and a reduce-scatter process, The one or more second processors: execute the forward process at a processing speed corresponding to an execution time of the all-gather process; Executing the backward process at a processing speed corresponding to an execution time of the reduce scatter process. The information processing device according to claim 3 .

5. The one or more second processors: Slowing down the processing speed by lowering the power supply voltage; 5. The information processing device according to claim 1.

6. The one or more second processors: Slowing down the processing speed by lowering the clock frequency; 5. The information processing device according to claim 1.

7. The one or more second processors: Slowing down the processing speed by thinning out or stretching the clock; 5. The information processing device according to claim 1.

8. An information processing system in which a plurality of information processing devices each including one or more memories, one or more first processors, and one or more second processors can communicate with each other via a network, The one or more first processors: performing a first process for transferring information regarding parameters of a neural network to and from the one or more second processors; The one or more second processors: executing a second process for updating a parameter of the neural network at a processing speed corresponding to an execution time of the first process; Information processing system.

9. An information processing method executed by an information processing device including one or more memories, one or more first processors, and one or more second processors, the method comprising: The one or more first processors: performing a first process for transferring information regarding parameters of a neural network to and from the one or more second processors; The one or more second processors: executing a second process for updating a parameter of the neural network at a processing speed corresponding to an execution time of the first process; Information processing methods.

10. A computer comprising one or more memories, one or more first processors, and one or more second processors, executing, by the one or more first processors, a first process for transferring information about parameters of a neural network to and from the one or more second processors; executing, by the one or more second processors, a second process for updating parameters of the neural network at a processing speed corresponding to an execution time of the first process; A program for executing the above.

Images