Data processing method and data processing device

Abstract

To reduce the memory bandwidth of a memory while maximizing effective efficiency of computations.SOLUTION: A data processing method according to an embodiment of the present invention involves determining whether or not to store output data in one or more first memories in forward processing of a neural network on the basis of computation cost of each of at least one or more layers of the forward processing and the size of the output data.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] This disclosure relates to a data processing method and a data processing apparatus.

Background Art

[0002] Generally, deep learning training is performed using a processor with a large number of cores such as a GPU (Graphics Processing Unit). When training is performed using this type of processor, usually, intermediate results of forward processing calculations are stored in an external memory such as DRAM (Dynamic Random Access Memory) for backward processing. Then, during backward processing, the intermediate results (intermediate data) of forward processing calculations required for backward processing calculations are read from the external memory. The reason why the intermediate results of forward processing calculations can be saved in the external memory every time is that the memory bandwidth (communication bandwidth) between the processor and the external memory is sufficiently large.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In the future, if the performance of the processor improves, it may become difficult to secure a memory bandwidth between the processor and the external memory that can transmit the intermediate results of forward processing calculations to the external memory every time. If it is difficult to improve the memory bandwidth using an external memory or a memory interface with an existing configuration, it becomes necessary to develop a new high-speed memory or a new high-speed memory interface, etc., and the system cost will increase significantly.

Means for Solving the Problems

[0004] The data processing method according to an embodiment of the present invention determines whether to save the forward processing at least one operation operation cost of the neural network and Intermediate data size of the above in one or more first memories Intermediate data or not. Then, using the intermediate data stored in the one or more first memories, one or more computers execute a process to perform a backward process following the forward process, and to obtain the intermediate data that was not stored in the one or more first memories by recalculating it during the backward process. . [Brief explanation of the drawing]

[0005] [Figure 1] A block diagram showing an example of a data processing unit in one embodiment of the present invention. [Figure 2] This is an explanatory diagram illustrating an example of neural network training performed on the data processing device shown in Figure 1. [Figure 3] This flowchart illustrates an example of forward processing in neural network training. [Figure 4] This flowchart illustrates an example of backward processing and optimization in neural network training. [Figure 5] Figure 1 is an explanatory diagram illustrating an example of neural network training using the data processing device shown in Figure 1. [Figure 6] This is an explanatory diagram illustrating another example of neural network training using the data processing device shown in Figure 1. [Figure 7] This is an explanatory diagram illustrating yet another example of neural network training using the data processing device shown in Figure 1. [Figure 8] This is an explanatory diagram illustrating another example of training a neural network using the data processing device shown in Figure 1. [Figure 9] This is a flowchart illustrating an example of how a data processing unit operates when training a neural network. [Figure 10] This block diagram shows an example of the hardware configuration of the data processing device 100 shown in Figure 1. [Modes for carrying out the invention]

[0006] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0007] Figure 1 is a block diagram showing an example of a data processing device in one embodiment of the present invention. The data processing device 100 shown in Figure 1 has at least one system board 10 including a processor 20 and a plurality of DRAMs (Dynamic Random Access Memory) 50 connected to the processor 20. For example, the data processing device 100 is a server. The processor 20 is an example of an arithmetic unit. The DRAMs 50 are an example of memory (external memory).

[0008] The processor 20 has multiple arithmetic units 30 and multiple SRAMs (Static Random Access Memory) 40 connected to each of the arithmetic units 30. The processor 20 is connected to the system bus. The processor 20 may be in the form of a chip or a package. Note that the memory connected to the processor 20 is not limited to DRAM 50, and the memory connected to the arithmetic units 30 is not limited to SRAM 40. SRAM 40 is an example of internal memory.

[0009] In this embodiment, the data processing device 100 includes multiple types of memory with different read / write speeds, namely SRAM 40 and DRAM 50, which generally has a slower read / write speed than SRAM 40. In this embodiment, for example, when training a neural network having multiple layers, it is possible to compensate for the insufficient read / write speed of the memory by not writing some of the calculation results of some layers to the DRAM 50. This makes it possible to improve the training speed.

[0010] Figure 2 is an explanatory diagram illustrating an example of neural network training performed on the data processing device 100 shown in Figure 1. In training a neural network with multiple intermediate layers between the input and output layers, forward processing, backward processing, and optimization processing are repeatedly performed multiple times while changing the training data. Forward processing, backward processing, and optimization processing are explained in Figures 3 and 4.

[0011] Forward processing is an example of the first type of processing, and the operations performed in multiple layers (multiple intermediate layers) during forward processing are examples of multiple types of the first type of processing. The first data is an example of the data used in the first type of processing, and the second data is an example of the data (operation result) obtained by performing the first type of processing. Backward processing is an example of the second type of processing, and the operations performed in multiple layers during backward processing are examples of multiple types of the second type of processing.

[0012] Figure 3 is a flowchart illustrating an example of forward processing in neural network training. In forward processing, data and parameters such as weights are input to the input layer and a predetermined number of intermediate layers. In the input layer, the input data and parameter 1 are processed to generate intermediate data 1. In the next intermediate layer after the input layer, intermediate data 1 and parameter 2 are processed to generate intermediate data 2.

[0013] In subsequent intermediate layers, the intermediate data generated by the previous intermediate layer and the parameters set for each intermediate layer are used in calculations, and the intermediate data generated by these calculations is output to the next intermediate layer. Note that there are also intermediate layers that do not use parameters. Examples of intermediate layers include convolutional layers, pooling layers, and fully connected layers.

[0014] In the output layer, the output data is obtained using the intermediate data N generated by the preceding intermediate layer N (the Nth layer). In the output layer, which calculates the error in a classification problem, the output data (solution) is obtained by, for example, using the softmax function as the activation function and cross-entropy as the error function. In the output layer, as explained in Figure 4, the error from the correct answer (loss function) is obtained by comparing the output data with the training data (correct answer data).

[0015] In this way, in the forward process, in each layer of the neural network, the input data and the parameters are calculated to obtain the data to be input to the next layer, and the output data is output from the final layer (forward propagation). Note that the forward process is used not only for training the neural network but also for inference using the neural network. The forward process can be represented by a computational graph such as a DAG (Directed Acyclic Graph).

[0016] FIG. 4 is a flowchart showing an example of the backward process and the optimization process in the training of a neural network. In the backward process, error backpropagation is performed to propagate the error in the reverse order of the forward process. In FIG. 4, the symbol Δ indicates the error of the data or the error of the parameters. The parameter update process performed in the optimization process is indicated by a dashed arrow.

[0017] First, in the backward process, in the layer where the error is obtained (output layer), the output data generated in the forward process and the teacher data are compared, and Δ intermediate data N, which is the error with respect to the intermediate data N input to the output layer, is generated. Δ intermediate data N is also the error of the output data output by the Nth intermediate layer.

[0018] Next, in each intermediate layer, starting from the intermediate layer closest to the output layer in order, the error with respect to the output data (Δ intermediate data) and the intermediate data, which is the input data, are calculated, and Δ parameter, which is the error with respect to the parameters of the intermediate layer, is generated. Δ parameter indicates the gradient of the parameter in the curve showing the change in the error with respect to the change in the parameter. For example, in the intermediate layer adjacent to the input layer, Δ intermediate data 2 and intermediate data 1 are calculated to obtain Δ parameter 2.

[0019] Also, in each intermediate layer, an error (Δintermediate data) with respect to the output data is calculated with the parameters of the intermediate layer, and Δintermediate data, which is the error with respect to the input data of the intermediate layer, is generated. The error (Δintermediate data) with respect to the input data of the intermediate layer is also the error of the output data of the previous intermediate layer (or the input layer). For example, in an intermediate layer adjacent to the input layer, Δintermediate data 2 and parameter 2 are calculated to obtain Δintermediate data 1.

[0020] In the input layer as well as in the intermediate layer, Δintermediate data 1 and the input data are calculated to obtain Δparameter 1, and Δintermediate data 1 and parameter 1 are calculated to obtain Δinput data, which is the error with respect to the input data. Thus, in the backward process, intermediate data, which is an intermediate result of the calculation by the forward process, is required.

[0021] In the optimization process, in each intermediate layer and the input layer, the parameters are corrected using the Δparameters (gradient of the error) obtained in the backward process. That is, the parameters are optimized. The optimization of the parameters is performed using a gradient descent method such as Momentum-SGD (Stochastic Gradient Descent) or ADAM.

[0022] Thus, in the backward process, the error of the data input to the output layer (the output data of the intermediate layer immediately before the output layer) from the output data and the teacher data is calculated. Then, the process of calculating the error of the intermediate data using the calculated error of the data and the process of calculating the error of the parameters using the error of the intermediate data are performed in order from the output-side layer (error backpropagation). In the parameter update process, the parameters are optimized based on the error of the parameters obtained in the backward process.

[0023] Figures 5 to 8 are explanatory diagrams illustrating an example of neural network training using the data processing device 100 in Figure 1. For the sake of clarity, the neural network to be trained is assumed to have layers L1, L2, L3, and a Loss layer. In the computation graph, layer L1 receives input data D0, and the output of layer L1 is connected to the input of layer L2. The output of layer L2 is connected to the input of layer L3, and layer L3 outputs output data. Layer Loss calculates the error (loss function) using the output data D3 from layer L3 and the training data. The explanation of Figures 5 to 8 is based on the following assumptions.

[0024] (1) Deep learning processing typically involves processing multiple data points in units called batches. In Figures 5 to 8, to examine the ratio of computation to memory access, it is sufficient to discuss the peak performance and memory bandwidth of the data processing unit 100 (processor 20) per data point included in one batch. Data size, FLOPS (Floating-point Operations Per Second), and memory bandwidth are implicitly assumed to be values ​​per data point.

[0025] (2) Assume that layer L1 is a convolutional layer with a kernel size of 1x1, input channels of "3", and output channels of "128". Assume that layer L2 is a convolutional layer with a kernel size of 3x3, input channels of "128", and output channels of "128". Assume that layer L3 is a fully connected layer with input channels of "128" and output channels of "10". Assume that the input and output image sizes of layer L1 and the output image size of layer L2 are 32 pixels wide and 32 pixels high.

[0026] (3) As additional layers, each layer L1 and L2 (convolutional layer) is to be followed by an appropriate activation function such as ReLU (Rectified Linear Unit). An Average Pooling layer is to be followed by layer L3 (fully connected layer). Layers L1 and L2 may each be layers that combine a convolutional layer and a ReLU (Rectified Linear Unit). Layer L3 may be a layer that combines Average Pooling and a fully connected layer.

[0027] As a result, the neural network trained in Figure 5 is capable of performing small but practical image recognition tasks. Forward and backward processing can be performed by fusing with layers L1, L2, and L3, respectively. In other words, no additional access to DRAM 50 occurs due to these layers. Furthermore, the computational load of the additional layers is sufficiently small to be ignored.

[0028] (4) The data used for training is assumed to be represented in 32-bit floating-point format. The peak performance of processor 20 is assumed to be 0.5 TFLOPS (Tera Floating-point Operations Per Second). The bandwidth of processor 20 to DRAM 50 is assumed to be 1 GB / s. The access to DRAM 50 and the calculations performed by arithmetic unit 30 can overlap to the maximum extent. In other words, the total time required for training will be the larger of the access time to DRAM 50 and the calculation time performed by arithmetic unit 30.

[0029] In Figures 5 to 8, the letters enclosed in circles within the rectangular frames representing each layer L1, L2, L3, and Loss indicate the processing performed in that layer. The leading letter "F" indicates forward processing, and the leading letter "B" indicates backward processing. The numbers in parentheses within the rectangular frames representing each layer L1, L2, and L3 show examples of the cost (computation cost or calculation cost) incurred for processing in each layer.

[0030] In forward processing, the input or output values ​​for each layer L1, L2, L3, and Loss represent the data size. In this example, for simplicity, we assume that the data size is equal to the number of channels between two adjacent layers.

[0031] In forward processing, the transfer index shown under each layer L1, L2, and L3 is obtained by dividing the computational cost of each layer by the data size output by each layer. The transfer index is one of the criteria for deciding whether or not to transfer the data obtained by the forward processing of each layer to DRAM50, and is an example of storage value.

[0032] It is preferable to transfer data obtained from calculations to the DRAM 50 for layers with high transfer value, and it is preferable not to transfer data obtained from calculations to the DRAM 50 for layers with low transfer value. By deciding whether or not to transfer data to the DRAM 50 for each layer based on the transfer value, the effective efficiency, which is expressed as the ratio of the calculation time by the processor 20 to the elapsed time during neural network training, can be improved. For example, the effective efficiency is calculated by dividing the minimum (fastest) calculation time by the processor 20 by the elapsed time during neural network training. In Figures 5 to 8, the amount of data temporarily stored in the SRAM 40 for use in the calculations of each layer is not limited. Also, data read from and written to the DRAM 50 is done via the SRAM 40.

[0033] In the training shown in Figure 5, no threshold is set for the transfer metric that determines whether or not to transfer the data obtained from calculations in each layer to DRAM 50. Therefore, in forward processing, all data D1, D2, and D3 obtained from calculations in layers L1, L2, and L3 are stored in DRAM 50. Data D0 used in layer L1 is transferred from DRAM 50. In backward processing, data D3, D2, D1, and D0 used in layers Loss, L3, L2, and L1 are transferred from DRAM 50.

[0034] If all data obtained from calculations in forward processing is stored in DRAM50 and all data used in backward processing is read from DRAM50, the total access time to DRAM50 for the entire training will be 2.122ms. The total calculation time by processor 20 for the entire training will be 1.817ms. The total calculation time is the minimum value because it does not include the data recalculation explained in Figure 6 and later. Therefore, the elapsed time required for training will be the total access time of DRAM50, which is the bottleneck (2.122ms), and the effective efficiency will be 85.6% (1.817 / 2.122).

[0035] In the training shown in Figure 6, the transfer threshold (first threshold) that determines whether or not to transfer the data obtained from calculations in each layer to the DRAM 50 is set to "0.1". Therefore, in the forward processing, data D2 and D3 obtained from calculations in layers L2 and L3, where the transfer threshold is greater than or equal to the threshold, are stored in the DRAM 50. Data D1 obtained from calculations in layer L1, where the transfer threshold is less than the threshold, is not stored in the DRAM 50. In other words, the data processing device 100 decimates a portion of the data used for training and stores it in the DRAM 50.

[0036] Then, in backward processing, the data D1 used for calculations in layer L2 is recalculated by the processor 20 performing forward processing on layer L1. The data D3, D2, and D0 used for calculations in layers Loss, L3, and L1 of the backward processing are transferred from DRAM 50.

[0037] In Figure 6, during backward processing, the relatively large data D1 is recalculated by the forward processing F1 of layer L1. Since there is no reading or writing of data D1 to DRAM 50, the access time of DRAM 50 throughout the entire training is significantly reduced compared to Figure 5, with a total access time of 1.073 ms. Also, the total computation time by processor 20 throughout the entire training increases slightly to 1.818 ms. Therefore, the elapsed time required for training is the total computation time by the bottleneck processor 20 (1.818 ms), and the effective efficiency improves from Figure 5 to 99.9% (1.817 / 1.818).

[0038] In the training shown in Figure 7, the threshold for the transfer index, which determines whether or not to transfer data obtained from calculations in each layer to DRAM 50, is set to "1.0". However, in Figure 7, in order to evaluate the effective efficiency, data D1 obtained from calculations in layer L1, where the transfer index is less than the threshold, is transferred to DRAM 50. Therefore, in the forward processing, data D3 obtained from calculations in layer L3, where the transfer index is greater than or equal to the threshold, and data D1 obtained from calculations in layer L1 are stored in DRAM 50. Data D2 obtained from calculations in layer L2, where the transfer index is less than the threshold, is not stored in DRAM 50.

[0039] Then, in backward processing, the data D2 used for calculations in layer L3 is recalculated by the processor 20 performing forward processing on layer L2. The data D3, D1, and D0 used for calculations in layers Loss, L2, and L1 of the backward processing are transferred from DRAM 50.

[0040] In Figure 7, during backward processing, the relatively large data D2 is recalculated by the forward processing F2 of layer L2. Since there is no reading or writing of data D2 to DRAM 50, the access time of DRAM 50 throughout the entire training decreases as in Figure 6, and the total access time becomes approximately 1 ms. On the other hand, the computational cost of the forward processing F2 of layer L2 is greater than the computational cost of the forward processing F1 of layer L1. Therefore, the total computation time by processor 20 throughout the entire training increases compared to Figures 5 and 6, reaching 2.423 ms. Consequently, the elapsed time becomes the total computation time by the bottleneck processor 20 (2.423 ms), and the effective efficiency worsens from Figure 5 to 75.0% (1.817 / 2.423).

[0041] In the training shown in Figure 8, the transfer index threshold, which determines whether or not to transfer data obtained from calculations in each layer to DRAM 50, is set to "1.0". Therefore, in forward processing, data D3 obtained from calculations in layer L3, where the transfer index is greater than or equal to the threshold, is stored in DRAM 50. Data D1 and D2 obtained from calculations in layers L1 and L2, respectively, where the transfer index is less than the threshold, are not stored in DRAM 50.

[0042] In the backward processing, the data D2 used for calculations in layer L3 is recalculated by the processor 20 by sequentially executing forward processing of layers L1 and L2. In the backward processing, the data D1 used for calculations in layer L2 is the one held in SRAM 40 during the forward processing of layer L1. The data D0 used for layer Loss and calculations in L1 in the backward processing is transferred from DRAM 50.

[0043] The operation shown in Figure 8 is a combination of the operations in Figures 6 and 7, and the computational cost of processor 20 in backward processing is higher than in Figure 7. Therefore, the effective efficiency is lower than 75.0% in Figure 7.

[0044] As shown in the examples from Figures 5 to 8, it can be seen that the effective efficiency in Figure 6 (99.9%) is the highest. In this way, by setting the threshold of the transfer index, which determines whether or not to transfer the data obtained from calculations in each layer to the DRAM 50, to an appropriate value, it is possible to maximize the effective efficiency of the processor 20 while reducing the memory bandwidth of the DRAM 50.

[0045] Since the amount of data transferred to the DRAM 50 can be reduced, the capacity of the DRAM 50 installed in the data processing unit 100 can be reduced. For example, a less expensive DRAM 50 with a smaller capacity can be used. As a result, the cost of the data processing unit 100 can be reduced. In other words, even with a data processing unit 100 with a reduced DRAM 50 capacity, the effective efficiency of the processor 20 can be improved.

[0046] Furthermore, since the capacity of the DRAM 50 can be reduced, the power consumption of the data processing unit 100 can be reduced. Also, if the memory bandwidth of the DRAM 50 is not reduced, a more powerful processor 20 can be used.

[0047] Figure 9 is a flowchart illustrating an example of the operation of a data processing device 100 that performs neural network training. For example, the flow shown in Figure 9 may be realized by the data processing device 100 (processor 20) executing a data processing program. Figure 9 shows an example of a data processing method and a data processing program.

[0048] First, in step S10, the data processing device 100 selects a layer on which to perform forward processing. Next, in step S12, the data processing device 100 performs forward processing on the selected layer.

[0049] Next, in step S14, the data processing device 100 determines whether the data obtained by forward processing at the layer is worth saving to the DRAM 50. For example, if the transfer index (Figures 5 to 8) at the layer targeted for forward processing is greater than or equal to a preset threshold, the data processing device 100 executes step S16 to save the data to the DRAM 50. If the transfer index at the layer targeted for forward processing is less than the threshold, the data processing device 100 executes step S18 without saving the data to the DRAM 50.

[0050] In step S16, the data processing device 100 stores the data obtained by the forward processing in the DRAM 50 and executes step S18. In step S18, if there is a next layer to perform forward processing, the data processing device 100 executes step S10; otherwise, it executes step S20.

[0051] In step S20, the data processing device 100 selects a layer on which to perform backward processing. Next, in step S22, the data processing device 100 determines whether the data to be used for backward processing is stored in the DRAM 50. If the data is stored in the DRAM 50, the data processing device 100 executes step S24; if the data is unstored data not stored in the DRAM 50, it executes step S26.

[0052] In step S24, the data processing device 100 reads data to be used for backward processing from the DRAM 50 and executes step S28. In step S26, the data processing device 100 performs forward processing to generate data (non-stored data) to be used for backward processing and executes step S28.

[0053] In step S28, the data processing device 100 uses the data obtained in step S24 or step S26 to perform backward processing on the layer to be processed. Next, in step S30, if there is another layer to perform backward processing on, the data processing device 100 executes step S20; otherwise, it terminates the operation shown in Figure 9.

[0054] In the embodiments described above, part or all of the data processing device 100 may be composed of hardware, or it may be composed of information processing by software (programs) executed by a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), etc. If it is composed of information processing by software, the software that realizes at least some of the functions of each device in the embodiments described above may be stored on a non-temporary storage medium (non-temporary computer-readable medium) such as a flexible disk, CD-ROM (Compact Disc-Read Only Memory), or USB (Universal Serial Bus) memory, and the information processing of the software may be executed by having a computer read it. Alternatively, the software may be downloaded via a communication network. Furthermore, the information processing may be executed by hardware by implementing the software on a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0055] The type of storage medium used to store data processing programs and other software is not limited. The storage medium is not limited to removable media such as magnetic disks or optical disks; it may also be a fixed storage medium such as a hard disk or memory. Furthermore, the storage medium may be located inside or outside the computer.

[0056] Figure 10 is a block diagram showing an example of the hardware configuration of the data processing device 100 shown in Figure 1. The data processing device 100 may, for example, be implemented as a computer comprising a processor 20, a DRAM (main memory) 50, an auxiliary storage device 60 (memory), a network interface 70, and a device interface 80, all connected via a bus 90. For example, the training described in Figures 5 to 8 is performed when the processor 20 executes a data processing program.

[0057] The data processing device 100 has one of each component, but it may have multiple identical components. Also, although Figure 10 shows one data processing device 100, the software may be installed on multiple devices including the data processing device 100, and each of the multiple data processing devices 100 may execute the same or different parts of the software. In this case, it may be a distributed computing configuration in which each data processing device 100 communicates via a network interface 70 or the like to execute processing. In other words, it may be configured as a computer system that realizes functionality by having one or more data processing devices 100 execute instructions stored in one or more storage devices. Alternatively, it may be configured so that information transmitted from a terminal is processed by one or more data processing devices 100 located on the cloud, and the processing results are transmitted to the terminal.

[0058] The operations described in Figures 5 to 8 and the flow shown in Figure 9 may be executed in parallel using one or more processors 20, or using multiple computers via a communication network 200. Furthermore, various calculations may be distributed to multiple arithmetic cores within the processor 20 and executed in parallel. Additionally, some or all of the processing and means of this disclosure may be executed by at least one of a processor and a storage device located on a cloud that can communicate with the data processing device 100 via a network. Thus, the computer system including the data processing device 100 may take the form of parallel computing using one or more computers.

[0059] The processor 20 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, or ASIC, etc.) including a computer control unit and arithmetic unit. Alternatively, the processor 20 may be a semiconductor device including a dedicated processing circuit. The processor 20 is not limited to an electronic circuit using electronic logic elements, but may also be implemented using an optical circuit with optical logic elements. Furthermore, the processor 20 may include computational functions based on quantum computing.

[0060] The processor 20 performs calculations based on data and software (programs) input from each of the internal components of the data processing device 100, and can output calculation results and control signals to each of the internal components. The processor 20 may also control each component of the data processing device 100 by executing the OS (Operating System) or applications of the data processing device 100.

[0061] The data processing device 100 may be implemented by one or more processors 20. Here, the processor 20 may refer to one or more electronic circuits provided on one chip, or one or more electronic circuits provided on two or more chips or two or more devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless.

[0062] The main memory 50 is a storage device that stores instructions executed by the processor 20 and various data, and the information stored in the main memory 50 is read by the processor 20. The auxiliary storage device 60 is a storage device other than the main memory 50. These storage devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. In the data processing device 100, the storage device for saving various data may be implemented by the main memory 50 or the auxiliary storage device 60, or by built-in memory such as SRAM 40 built into the processor 20.

[0063] The data processing device 100 is not limited to the configuration shown in Figure 1. Multiple processors 20 may be connected to one memory device, or only one processor 20 may be connected to one memory device. Multiple memory devices may be connected to one of the processors 20. If the data processing device 100 consists of at least one memory device and multiple processors 20 connected to this at least one memory device, it may include a configuration in which at least one of the multiple processors 20 is connected to at least one memory device. Furthermore, this configuration may be realized by the memory devices and processors 20 included in multiple data processing devices 100. In addition, it may include a configuration in which the memory device is integrated with the processor 20 (for example, a cache memory including an L1 cache and an L2 cache).

[0064] The network interface 70 is an interface for connecting to the communication network 200 wirelessly or via a wired connection. The network interface 70 can be any appropriate interface, such as one conforming to existing communication standards. Information may be exchanged between the data processing device 100 and an external device 210 connected via the communication network 200 through the network interface 70. The communication network 200 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, as long as information is exchanged between the data processing device 100 and the external device 210. An example of a WAN is the Internet; an example of a LAN is IEEE 802.11 or Ethernet®; and an example of a PAN is Bluetooth® or NFC (Near Field Communication).

[0065] The device interface 80 is an interface such as USB that connects directly to the external device 220.

[0066] The external device 220 may be connected to the data processing device 100 via a network, or it may be connected directly to the data processing device 100.

[0067] External device 210 or external device 220 may, for example, be an input device. The input device may be a camera, microphone, motion capture device, various sensors, keyboard, mouse, or touch panel, and will provide the acquired information to the data processing device 100. Alternatively, it may be a device equipped with an input unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

[0068] Furthermore, the external device 210 or external device 220 may, for example, be an output device. The output device may be a display device such as an LCD (Liquid Crystal Display), CRT (Cathode Ray Tube), PDP (Plasma Display Panel), or organic EL (Electro Luminescence) panel, or it may be a speaker that outputs sound, etc. It may also be a device equipped with an output unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

[0069] Furthermore, the external device 210 or external device 220 may be a storage device (memory). For example, the external device 210 may be network storage, and the external device 220 may be storage such as an HDD. The external device 220, which is a storage device (memory), is an example of a recording medium that can be read by a computer such as the processor 20.

[0070] Furthermore, the external device 210 or external device 220 may be a device that has some of the functions of the components of the data processing device 100. In other words, the data processing device 100 may transmit or receive some or all of the processing results of the external device 210 or external device 220.

[0071] In this embodiment, by setting an appropriate threshold value for the transfer index that determines whether or not to transfer data obtained from calculations in each layer to the DRAM 50, the effective efficiency of the processor 20 can be maximized while reducing the memory bandwidth of the DRAM 50. This reduces the amount of DRAM 50 installed in the data processing device 100, and thus reduces the cost of the data processing device 100.

[0072] The transfer metric is calculated by dividing the computational cost of each layer by the data size output by each layer. Therefore, the transfer metric can be easily determined regardless of the complexity of the neural network (computation graph). Furthermore, the method of determining whether or not to transfer the computation results from the processor 20 to the DRAM 50 based on the transfer metric is applicable not only to neural network training but also to other data processing.

[0073] Where the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used herein, it includes any of a, b, c, ab, ac, bc, or abc. It also includes multiple instances of any one element, such as aa, abb, aabbcc, etc. Furthermore, it includes adding other elements other than the enumerated elements (a, b, and c), such as abcd having d.

[0074] In this specification (including the claims), when expressions such as "data as input / based on / according to / in accordance with data" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used as input, or where the data has been processed in some way (e.g., data with added noise, normalized data, intermediate representations of the data, etc.) is used as input. Furthermore, when it is stated that some result is obtained "based on / according to / in accordance with data", this includes cases where the result is obtained based solely on the data in question, as well as cases where the result is also influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output", unless otherwise specified, this includes cases where the data itself is used as output, or where the data has been processed in some way (e.g., data with added noise, normalized data, intermediate representations of the data, etc.) is used as output.

[0075] In this specification (including the claims), the terms “connected” and “coupled” are intended to be non-restrictive terms that include any direct connection / coupling, indirect connection / coupling, electrical connection / coupling, communicative connection / coupling, operational connection / coupling, physical connection / coupling, etc. The terms should be interpreted as appropriate in the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted non-restrictively as being included in the terms.

[0076] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and that it is configured to actually perform operation B by the setting of a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0077] Wherever terms meaning "comprising" or "possessing" (e.g., "comprising / including" and "having") are used in this specification (including the claims), they are intended to be open-ended terms, including cases where the subject matter of such terms is not the object of the term. Where the object of such terms meaning "comprising" or "possessing" is an expression that does not specify a quantity or suggests a singular number (an expression with the article "a" or "an"), such expression should be interpreted as not being limited to a specific number.

[0078] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in one place, and expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) are used in another place, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) should not necessarily be interpreted as not being limited to a specific number.

[0079] In this specification, if a particular configuration of an embodiment is described as yielding a specific advantage or result, it should be understood that, unless otherwise stated, the same advantage or result can also be obtained from one or more other embodiments having the same configuration. However, it should be understood that the presence or absence of such an advantage or result generally depends on various factors, conditions, and / or states, and that the configuration does not necessarily guarantee that the advantage or result can be obtained. The advantage or result can only be obtained from the configuration described in the embodiment when various factors, conditions, and / or states are met, and the advantage or result cannot necessarily be obtained in the claimed invention that defines the configuration or a similar configuration.

[0080] In this specification (including the claims), when terms such as "maximize" are used, they include finding the global maximum value, finding an approximation of the global maximum value, finding the local maximum value, and finding an approximation of the local maximum value, and should be interpreted appropriately depending on the context in which the term is used. They also include finding approximations of these maximum values ​​probabilistically or heuristically. Similarly, when terms such as "minimize" are used, they include finding the global minimum value, finding an approximation of the global minimum value, finding the local minimum value, and finding an approximation of the local minimum value, and should be interpreted appropriately depending on the context in which the term is used. They also include finding approximations of these minimum values ​​probabilistically or heuristically. Similarly, when terms such as "optimize" are used, they include finding the global optimal value, finding an approximation of the global optimal value, finding the local optimal value, and finding an approximation of the local optimal value, and should be interpreted appropriately depending on the context in which the term is used. They also include finding approximations of these optimal values ​​probabilistically or heuristically.

[0081] In this specification (including the claims), when multiple hardware components perform a predetermined process, each component may cooperate to perform the predetermined process, or some components may perform all of the predetermined process. Alternatively, some components may perform part of the predetermined process, while other components perform the remainder. In this specification (including the claims), when expressions such as "one or more hardware components perform a first process, and the one or more hardware components perform a second process" are used, the hardware component performing the first process and the hardware component performing the second process may be the same or different. In other words, it is sufficient that the hardware component performing the first process and the hardware component performing the second process are included in the one or more hardware components. Hardware may include electronic circuits or devices containing electronic circuits.

[0082] In this specification (including the claims), when multiple memory devices store data, each of the multiple memory devices may store only a portion of the data or the entire data.

[0083] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible, provided that they do not depart from the conceptual idea and spirit of the present invention derived from the claims and their equivalents. For example, where numerical values ​​or mathematical formulas are used in the description in all of the embodiments described above, they are provided as examples only and are not limited thereto. Also, the order of operations in the embodiments is provided as examples only and is not limited thereto. [Explanation of Symbols]

[0084] 20 processors 30 Arithmetic unit 40 SRAM 50 DRAM 60 Auxiliary storage 70 Network Interfaces 80 device interfaces 90 bus 100 Data Processing Devices 200 Communication Networks 210 External device 220 External device

Claims

1. Based on the computational cost of at least one operation in the forward processing of the neural network and the size of the intermediate data, it is determined whether to store the intermediate data in one or more first memories. Using the intermediate data stored in the one or more first memories, the backward processing following the forward processing is executed. The intermediate data that was not stored in the one or more first memories is obtained by recalculating it during the backward processing. A data processing method performed by one or more computers.

2. Based on the value obtained by dividing the computation cost by the size of the intermediate data, it is determined whether or not to store the intermediate data in one or more first memories. The data processing method according to claim 1.

3. The aforementioned one or more first memories are DRAMs. The data processing method according to claim 1 or claim 2.

4. The intermediate data stored in the one or more first memories is used in the backward processing without being recalculated. The data processing method according to any one of claims 1 to 3.

5. Of the aforementioned intermediate data, at least a portion of the intermediate data that was not stored in the one or more first memories is stored in one or more second memories. The data processing method according to any one of claims 1 to 4.

6. The recalculation is performed using at least a portion of the intermediate data stored in the one or more second memories. The data processing method according to claim 5.

7. The aforementioned one or more second memories are SRAMs. The data processing method according to claim 5 or claim 6.

8. The update of the neural network parameters based on the backward processing is performed by ADAM. The data processing method according to any one of claims 1 to 7.

9. The forward processing is performed by one or more GPUs. The data processing method according to any one of claims 1 to 8.

10. The forward processing is performed by multiple processors. The one or more first memories are connected to the plurality of processors. The data processing method according to any one of claims 1 to 9.

11. A data processing device comprising one or more processors, The one or more processors mentioned above are: Based on the computational cost of at least one operation in the forward processing of the neural network and the size of the intermediate data, it is determined whether to store the intermediate data in one or more first memories. Using the intermediate data stored in the one or more first memories, the backward processing following the forward processing is executed. The intermediate data that was not stored in the one or more first memories is obtained by recalculating it during the backward processing. Data processing device.

12. The one or more processors determine whether to store the intermediate data in the one or more first memories based on a value obtained by dividing the computation cost by the size of the intermediate data. The data processing device according to claim 11.

13. The aforementioned one or more first memories are DRAMs. The data processing apparatus according to claim 11 or claim 12.

14. The intermediate data stored in the one or more first memories is used in the backward processing without being recalculated. A data processing device according to any one of claims 11 to 13.

15. Of the aforementioned intermediate data, at least a portion of the intermediate data that was not stored in the one or more first memories is stored in one or more second memories. A data processing device according to any one of claims 11 to 14.

16. The recalculation is performed using at least a portion of the intermediate data stored in the one or more second memories. The data processing device according to claim 15.

17. The aforementioned one or more second memories are SRAMs. The data processing device according to claim 15 or claim 16.

18. The update of the neural network parameters based on the backward processing is performed by ADAM. A data processing device according to any one of claims 11 to 17.

19. The forward processing is performed by one or more GPUs. A data processing device according to any one of claims 11 to 18.

20. The forward processing is performed by multiple processors. The one or more first memories are connected to the plurality of processors. A data processing device according to any one of claims 11 to 19.

21. The data processing method described in any one of Claims 1 to 10 is performed by one or more computers. program.

Images