Inference device, calculation device, setting method, calculation method, and calculation program

Abstract

In an inference device (10) provided with a plurality of inference units (330, 331,…, 33N) for performing convolution processing on input data by a layer integration method, a calculation unit (31) calculates, regarding a layer integration section obtained by integrating a plurality of layers of a convolution neural network, a different segment for each of the inference units (330, 331,…, 33N), and a setting unit (32) sets, to each of the plurality of inference units (330, 331,…, 33N), a segment that is of the layer integration section and that is calculated by the calculation unit (31).

Description

Technical Field

[0001] The disclosed technology relates to an inference device, a calculation device, a setting method, a calculation method, and a calculation program.

Background Art

[0002] In recent years, for applying image recognition and object recognition using a convolutional neural network (CNN) to use cases that require real-time performance, power saving, area saving, etc., such as surveillance cameras and drones, research and development on CNN inference processing accelerators has been actively conducted. Furthermore, efforts have been made to enable inference processing with high-definition video such as 4K and 8K as input. Since there is a limit to the input image size of the CNN model, a method of parallel processing divided input images with a plurality of inference cores is common. However, parallelization of the inference cores causes an increase in the external memory bandwidth, so the external memory bandwidth is likely to become a bottleneck in processing performance.

[0003] As a method for reducing the external memory bandwidth in CNN inference processing, the Layer fusion method can be mentioned (Non-Patent Document 1). In the method described in Non-Patent Document 1, as shown in FIG. 1, a plurality of layers are continuously processed in units of tiles obtained by dividing the input data into a grid pattern. As a result, the amount of output data of each layer can be reduced to such an extent that it can be stored in the internal memory. As a result, it becomes possible to hold the data that was transferred to the external memory in the internal memory, and it becomes possible to reduce the external memory bandwidth. Here, the section of the layers to be continuously processed will be referred to as the "layer integration section". Generally, the processing is performed by dividing the inside of the CNN model into several layer integration sections. Data transfer to the external memory occurs at the start and end layers of the layer integration section.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the method described in Non-Patent Document 1, the external memory bandwidth per inference core can be reduced. However, when the inference cores are parallelized for high-definition video processing or the like, there remains a problem that the external memory bandwidth increases in proportion to the number of inference cores.

[0006] The disclosed technology has been made in view of the above points, and an object thereof is to reduce the external memory bandwidth when inference cores are parallelized in a CNN inference processing accelerator.

Means for Solving the Problems

[0007] The first aspect of the present disclosure is an inference device including: a plurality of inference units that perform convolution processing on input data in a layer integration manner for each layer integration section in which a plurality of layers of a convolutional neural network are integrated; and a setting unit that sets a delimiter of the layer integration section different for each of the plurality of inference units for each of the plurality of inference units.

[0008] The second aspect of the present disclosure is a calculation device including: a calculation unit that calculates a delimitation method of the layer integration section different for each of the plurality of inference units, which is set for each of the plurality of inference units that perform convolution processing on input data in a layer integration manner for each layer integration section in which a plurality of layers of a convolutional neural network are integrated; and an output unit that outputs the delimitation of the layer integration section calculated by the calculation unit to an inference device including the plurality of inference units.

[0009] The third aspect of the present disclosure is a setting method in which a setting unit sets a different delimiter of the layer integration section for each of a plurality of inference units that perform convolution processing on input data by a layer integration method for each layer integration section in which a plurality of layers of a convolutional neural network are integrated.

[0010] The fourth aspect of the present disclosure is a calculation method in which a calculation unit calculates a different delimiter method of the layer integration section for each of a plurality of inference units that perform convolution processing on input data by a layer integration method for each layer integration section in which a plurality of layers of a convolutional neural network are integrated, and an output unit outputs the delimiter of the layer integration section calculated by the calculation unit to an inference device including the plurality of inference units.

[0011] The fifth aspect of the present disclosure is a calculation program for causing a computer to function as each part of the above calculation device.

Advantages of the Invention

[0012] According to the disclosed technology, it is possible to reduce the external memory bandwidth when the inference cores are parallelized by a CNN inference processing accelerator.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for the convenience of explanation and may be different from the actual ratios.

[0015] <First Embodiment> FIG. 2 is a block diagram showing the hardware configuration of the inference device 10 according to the first embodiment. As shown in FIG. 2, the inference device 10 includes a CPU (Central Processing Unit) 11, an external memory 12, a storage 13, an input / output I / F (Interface) 14, a communication I / F 15, and a plurality of inference cores 16. Each component is connected to be communicable with each other via a bus 20.

[0016] The CPU 11 is a central arithmetic processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the storage 13 and executes the program using the external memory 12 as a work area. The CPU 11 performs the control of the above components and various arithmetic processes according to the program stored in the storage 13. In the present embodiment, a setting program for executing the setting process described later is stored in the storage 13.

[0017] The external memory 12 temporarily stores programs or data as a working area. The external memory 12 is implemented by, for example, DDR SDRAM (Double-Data-Rate Synchronous Dynamic Random Access Memory) or the like. The storage 13 stores various programs and various data. The storage 13 is implemented by, for example, HDD (Hard Disk Drive), SSD (Solid State Drive), or the like.

[0018] The input / output I / F 14 is an interface for connecting to external devices such as input devices like a mouse and a keyboard, and output devices like a display and a printer. The communication I / F 15 is an interface for communicating with other devices. For such communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0019] The plurality of inference cores 16 each have the same configuration. In the example of FIG. 2, the inference device 10 includes N + 1 inference cores 16, and numbers 0, 1, ···, N are assigned to each inference core 16. The number of inference cores 16 included in the inference device 10 may be an arbitrary number. Hereinafter, when explaining each inference core without distinction, it is denoted as "inference core 16", and when explaining each inference core separately, it is denoted as "inference core #m". m is the number of the inference core 16.

[0020] The inference core 16 is an integrated circuit that performs inference processing in a CNN. The inference core 16 is implemented by, for example, FPGA (Field-Programmable Gate Array) or the like. The inference core 16 includes a setting holding unit 17, an internal memory 18, and a MAC (Multiply Accumulation) circuit 19.

[0021] The setting retention unit 17 retains the settings necessary for the inference process executed by the corresponding inference core 16. The setting retention unit 17 is realized by, for example, a register or the like. The internal memory 18 is a memory that retains data and calculation results necessary for various calculations, and is an on-chip memory module mounted inside the inference core 16. Specifically, the input data of the CNN inference process stored in the external memory 12 is temporarily transferred to the internal memory 18 for processing by the inference core 16. Further, the internal memory 18 temporarily retains the input / output data of the intermediate layers in the layer integration section (details will be described later) when processing in the layer integration method. The MAC circuit 19 is an arithmetic circuit designed to perform the convolution process in the CNN inference process.

[0022] As described above, the inference device 10 has an internal memory 18 for each inference core 16, and the external memory 12 is shared among the inference cores 16. Although not shown in FIG. 2, when the CNN model includes layers other than the convolutional layers such as the Maxpooling layer and the Residual layer, an arithmetic unit for performing the arithmetic process of that layer is separately provided in the inference core 16.

[0023] Next, the functional configuration of the inference device 10 will be described.

[0024] FIG. 3 is a block diagram showing an example of the functional configuration of the inference device 10. As shown in FIG. 3, the inference device 10 includes, as a functional configuration, an arithmetic unit 31, a setting unit 32, and a plurality of inference units 330, 331,..., 33N. The inference unit 330 is realized by the inference core #0 shown in FIG. 2, the inference unit 331 is realized by the inference core #1, and the inference unit 33N is realized by the inference core #N. Hereinafter, when the inference units 330, 331,..., 33N are described without distinction, they will be referred to as "inference unit 33".

[0025] The calculation unit 31 calculates how to divide the layer integration intervals differently for each inference unit 33. The layer integration interval is the integration of multiple layers of the CNN. Fig. 4 shows an example of how to divide the layer integration intervals. In the example of Fig. 4, the CNN model is composed of a total of 7 layers from layer 0 to layer 6, and this is an example when the inference cores 16 corresponding to the inference units 33 are three, namely inference core #0, inference core #1, and inference core #2. As shown in Fig. 4, by making the division of the layer integration intervals different for each inference unit 33, that is, setting the layer integration intervals so that the start layer and the end layer of the layer integration intervals of each inference core 16 do not overlap, the timing of data transfer to the external memory 12 is shifted. Thereby, the bandwidth of the external memory 12 when the inference cores 16 are parallelized is smoothed, and as a result, the bandwidth of the external memory 12 can be reduced.

[0026] Specifically, the calculation unit 31 calculates the bandwidth of the external memory 12 used for each layer integration interval set for each inference unit 33. More specifically, the calculation unit 31 obtains the bandwidth used for reading the input data of the first layer in the layer integration interval from the external memory 12 and writing the output data of the last layer in the layer integration interval to the external memory 12. Then, the calculation unit 31 calculates the bandwidth of the external memory 12 used for each layer integration interval based on each obtained bandwidth. Also, the calculation unit 31 calculates the way of dividing the layer integration intervals such that the maximum value of the total bandwidth obtained by summing up the bandwidths of the external memory 12 calculated for each inference unit 33 for each layer is equal to or less than a predetermined target value. The calculation unit 31 notifies the setting unit 32 of the different divisions of the layer integration intervals for each calculated inference unit 33.

[0027] The setting unit 32 sets, for each of the plurality of inference units 33, the different divisions of the layer integration intervals notified from the calculation unit 31.

[0028] Based on the division of the layer integration intervals set by the setting unit 32, the inference unit 33 performs convolution processing on the input data by the layer integration method.

[0029] Next, the operation of the inference device 10 according to the first embodiment will be described.

[0030] FIG. 5 is a flowchart showing the flow of the setting process by the inference device 10. The setting process is performed by the CPU 11 reading the setting program from the storage 13, expanding it in the external memory 12, and executing it. Note that the setting process is an example of the setting method of the disclosed technology.

[0031] In step S11, the CPU 11, as the calculation unit 31, sets an initial value for the method of delimiting the layer integration section in each inference core 16 (inference unit 33). The initial value may be given from the outside, or the initial value previously stored in a predetermined storage area of the inference device 10 may be read and used.

[0032] Next, in step S12, the CPU 11, as the calculation unit 31, calculates the external memory bandwidth per inference core 16 for all the inference cores 16 based on the currently set method of delimiting the layer integration section. Specifically, the CPU 11, as the calculation unit 31, calculates the external memory read bandwidth, which is the bandwidth of the external memory 12 from which data is read by the inference core 16, and the external memory write bandwidth, which is the bandwidth of the external memory 12 to which data is written by the inference core 16.

[0033] With reference to FIGS. 6 and 7, a specific example of the calculation of the external memory bandwidth will be described. FIG. 6 is an example of the method of delimiting the layer integration section in the CNN model. In FIG. 6, n in the layer integration section #n is the number of the layer integration section. The same applies to the following figures. In the example of FIG. 6, the CNN model is composed of 7 layers, and layer 0 and layer 1 are delimited as layer integration section #0, layer 2, layer 3, and layer 4 are delimited as layer integration section #1, and layer 5 and layer 6 are delimited as layer integration section #2. Also, it is assumed that the number of convolution operation cycles, the input data capacity, and the output data capacity in each layer are given from the outside in advance.

[0034] The upper diagram of FIG. 7 shows the processing image in the case of the example of FIG. 6. Here, only the processing image for one inference core 16 is illustrated. The processing in each layer integration period is roughly divided into the following three types. The first is the read of the input data of the first layer in the layer integration period from the external memory 12, the second is the convolution operation processing of all the layers within the layer integration period, and the third is the write of the output data of the final layer in the layer integration period to the external memory 12. It is assumed that the processing in each layer integration period performs pipeline processing with the above three types of processing overlapping each other.

[0035] The calculation unit 31 calculates, among the three types of processing, the number of cycles required for the convolution operation processing by adding up the number of cycles for the convolution operation processing in each layer given in advance from the outside. The number of cycles available for the read of the input data of the first layer in the layer integration period from the external memory 12 and the write of the output data of the final layer in the layer integration period to the external memory 12 is assumed to be the same as the number of cycles required for the overlapping convolution operation processing. Also, the calculation unit 31 determines the transfer data capacity that needs to be transferred in each of the read from the external memory 12 and the write to the external memory 12 based on the input data capacity and output data capacity of each layer given in advance from the outside. As shown in the lower diagram of FIG. 7, the calculation unit 31 calculates the value obtained by dividing the determined transfer data capacity by the number of cycles available for each of the read from the external memory 12 and the write to the external memory 12 as the external memory read bandwidth and the external memory write bandwidth.

[0036] Note that for the bandwidth indicated as "-" in the lower diagram of FIG. 7, since it depends on the processing before and after the processing range shown in this example, it means that it is omitted and actually needs to be calculated. However, for the first input read and the last output write of the entire processing, since the number of cycles available for each processing cannot be obtained from the number of cycles of other processing, the number of cycles available for each processing is given in advance.

[0037] Next, in step S13, the CPU 11, acting as the calculation unit 31, adds up the external memory read bandwidth and the external memory write bandwidth calculated for all the inference cores 16 for each layer, and calculates it as the total bandwidth of the external memory 12 for each layer.

[0038] Referring to FIG. 8, a specific example of calculating the total bandwidth of the external memory 12 in a plurality of inference cores 16 will be described. Here, as shown in FIG. 7, the external memory read bandwidth and the external memory write bandwidth for each layer integration section of each inference core 16 calculated are respectively represented as R mn , W mn . m represents the number of the inference core 16, and n represents the number of the layer integration section. Here, the case where there are three inference cores 16, namely inference core #0, inference core #1, and inference core #2, is exemplified. As shown in FIG. 8, the way of dividing the layer integration section is different for each inference core 16. The calculation unit 31 adds up the external memory read bandwidth R mn and the external memory write bandwidth W mn calculated for each inference core 16 for each layer (in the vertical direction in FIG. 8), and calculates it as the total bandwidth for each layer. In the example of FIG. 8, for the second and third layers, and the fifth and sixth layers, the total bandwidth for each layer is the same respectively.

[0039] Next, in step S14, the CPU 11, acting as the calculation unit 31, determines whether the maximum value among the total bandwidths for each layer is less than or equal to a target value given from the outside in advance. If it is less than or equal to the target value, it proceeds to step S16. If it exceeds the target value, it proceeds to step S15. Also, when the above processing loop is repeated a specified number of times, it proceeds to step S16.

[0040] In step S15, the CPU 11, acting as the calculation unit 31, changes the way of dividing the layer integration section. The method of changing the way of dividing is not particularly specified. As an example, it can be changed randomly. Then, it returns to step S12.

[0041] On the other hand, in step S16, the CPU 11, acting as the calculation unit 31, notifies the setting unit 32 of the method of delimiting the current layer integration interval, and via the setting unit 32, notifies each inference core 16 of the settings necessary for operation, thereby ending the setting process.

[0042] As described above, for each of the plurality of inference units that perform convolution processing on input data by a layer integration method for each layer integration interval obtained by integrating a plurality of layers of the CNN, the inference device according to the first embodiment calculates a different method of delimiting the layer integration interval for each inference unit. Then, the inference device sets the calculated method of delimiting the layer integration interval for each inference unit. As a result, when the inference cores are parallelized by the CNN inference processing accelerator, the timing of data transfer to the external memory that occurs between the start layer and the end layer of the layer integration interval of each inference core is shifted, thereby smoothing the external memory bandwidth. Therefore, the total external memory bandwidth can be reduced.

[0043] <Second Embodiment> Next, the second embodiment will be described. The second embodiment differs from the first embodiment in that the calculation of the method of delimiting the layer integration interval is performed outside the device. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

[0044] As shown in FIG. 9, in the second embodiment, the method of delimiting the layer integration interval calculated by the calculation device 40 is set for each inference unit 33 of the inference device 210, that is, for each inference core 16.

[0045] FIG. 10 is a block diagram showing the hardware configuration of the calculation device 40 according to the second embodiment. As shown in FIG. 10, the calculation device 40 includes a CPU 51, a memory 52 realized by a RAM or the like, a storage 53, an input / output I / F 54, and a communication I / F 55. Each component is connected to be communicable with each other via a bus 60. Since the details of each component are the same as the corresponding components of the hardware configuration of the inference device 10 shown in FIG. 2, detailed description thereof is omitted. Further, since the hardware configuration of the inference device 210 according to the second embodiment is the same as the hardware configuration of the inference device 10 according to the first embodiment shown in FIG. 2, description thereof is omitted.

[0046] Next, the functional configuration of the calculation device 40 will be described. As shown in FIG. 9, the calculation device 40 includes, as a functional configuration, a calculation unit 41 and an output unit 42. The function of the calculation unit 41 is the same as that of the calculation unit 31 of the inference device 10 according to the first embodiment. The output unit 42 outputs to the inference device 210 the way of dividing the layer integration section calculated by the calculation unit 41.

[0047] Next, the functional configuration of the inference device 210 will be described. As shown in FIG. 9, the inference device 210 includes, as a functional configuration, a setting unit 32 and a plurality of inference units 330, 331, ···, 33N. The functional configuration of the inference device 210 according to the second embodiment is the same as that of the inference device 10 according to the first embodiment except that the calculation unit 31 is not provided.

[0048] Next, the operation of the calculation device 40 according to the second embodiment will be described. The calculation device 40 executes a calculation process similar to the setting process shown in FIG. 5. However, in step S16, the CPU 11, as the output unit 42, executes a process of outputting to the inference device 210 the way of dividing the currently calculated layer integration section. Note that the calculation process is an example of the calculation method of the disclosed technology.

[0049] Next, the operation of the inference device 210 according to the second embodiment will be described. When the inference device 210 receives the way of dividing the layer integration section from the calculation device 40, it sets the way of dividing the layer integration section for each inference core 16 in the same manner as the process of step S16 of the setting process shown in FIG. 5.

[0050] As described above, for each of the plurality of inference units that perform convolution processing on input data by the layer integration method for each layer integration interval obtained by integrating a plurality of layers of the CNN, the calculation device according to the second embodiment calculates a way of dividing the layer integration interval that is different for each inference unit. Then, the calculation device outputs the calculated way of dividing the layer integration interval to the inference device. The inference device sets the way of dividing the layer integration interval received from the calculation device for each inference unit. Thereby, in the case where the inference cores are parallelized by the CNN inference processing accelerator, the timing of data transfer to the external memory that occurs between the start layer and the end layer of the layer integration interval of each inference core is shifted, and the external memory bandwidth is smoothed. Therefore, the total external memory bandwidth can be reduced.

[0051] Note that the setting process and the calculation process in which the CPU reads and executes software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of the processor in this case include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing, such as an FPGA, and a dedicated electric circuit which is a processor having a circuit configuration designed specifically for executing a specific process, such as an ASIC (Application Specific Integrated Circuit). Further, the setting process and the calculation process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). Further, the hardware structure of these various processors is, more specifically, an electric circuit combining circuit elements such as semiconductor elements.

[0052] Also, in each of the above embodiments, the mode in which the setting program and the calculation program are pre-stored (installed) in the storage 13 has been described, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory. Further, the program may be in a form downloaded from an external device via a network.

[0053] Regarding the above embodiments, the following additional remarks are further disclosed.

[0054] (Supplementary Note 1) A memory, At least one processor connected to the memory, Including, The processor is For each of a plurality of inference units that perform convolution processing on input data by a layer integration method for each layer integration section integrating a plurality of layers of a convolutional neural network, set a delimiter of the different layer integration sections for each inference unit An inference device configured as described above.

[0055] (Supplementary Note 2) A non-transitory recording medium storing a program executable by a computer to execute a calculation process, The calculation process is For each of a plurality of inference units that perform convolution processing on input data by a layer integration method for each layer integration section integrating a plurality of layers of a convolutional neural network, calculate a delimitation method of the different layer integration sections for each inference unit, Output the delimiter of the layer integration section calculated by the calculation unit to an inference device including the plurality of inference units A non-transitory recording medium including this.

Explanation of Signs

[0056] 10. 210 Inference Device 11. 51 CPU 12. External Memory 52. Memory 13. 53 Storage 14. 54 Input / Output I / F 15. 55 Communication I / F 16. Inference Core 17. Setting Retention Unit 18. Internal Memory 19. MAC Circuit 20. 60 Bus 31. Calculation Unit 32. Setting Unit 330, 331, 33N Inference Unit 40. Calculation Device 41. Calculation Unit 42. Output Unit

Claims

**Claim 1**: A plurality of inference units that process in parallel a plurality of divided data obtained by dividing input data, each of which performs convolution processing on the divided data by a layer integration method for each layer integration interval obtained by integrating a plurality of layers of a convolutional neural network. A setting unit that sets, for each of the plurality of inference units, a boundary of the layer integration interval that is different for each inference unit. An inference device including the above. **Claim 2** The inference device according to claim 1, further including a calculation unit that calculates a boundary setting method of the layer integration interval for each inference unit. **Claim 3** The calculation unit calculates a bandwidth of an external memory used for each layer integration interval set for each inference unit, and calculates a boundary setting method of the layer integration interval such that a maximum value of a total bandwidth obtained by summing up the calculated external memory bandwidths for each layer for each inference unit is equal to or less than a predetermined target value. The inference device according to claim 2. **Claim 4** The calculation unit calculates, as a bandwidth of the external memory used for each layer integration interval, a bandwidth of the external memory used for reading input data of the first layer in the layer integration interval from the external memory and writing output data of the last layer in the layer integration interval to the external memory. The inference device according to claim 3. **Claim 5**: A calculation device including: a plurality of inference units that process in parallel a plurality of divided data obtained by dividing input data, each of which performs convolution processing on the divided data by a layer integration method for each layer integration interval obtained by integrating a plurality of layers of a convolutional neural network; and a calculation unit that calculates, for each of the plurality of inference units, a boundary setting method of the layer integration interval that is different for each inference unit. An output unit that outputs the boundary of the layer integration interval calculated by the calculation unit to an inference device including the plurality of inference units. Including the above. **Claim 6** A setting method in which a setting unit sets, for each of a plurality of inference units that process in parallel a plurality of divided data obtained by dividing input data, a boundary of the layer integration interval that is different for each inference unit, where each of the plurality of inference units performs convolution processing on the divided data by a layer integration method for each layer integration interval obtained by integrating a plurality of layers of a convolutional neural network. **Claim 7** The calculation unit is a plurality of inference units that process a plurality of divided data obtained by dividing input data in parallel, and each of them performs convolution processing on the divided data by a layer integration method for each layer integration interval obtained by integrating a plurality of layers of a convolutional neural network. For each of the plurality of inference units, calculation of a different way of delimiting the layer integration interval is performed, The output unit outputs the delimitation of the layer integration interval calculated by the calculation unit to an inference device including the plurality of inference units. Calculation method.

8. An arithmetic program for causing a computer to function as each unit described in claim 5.

Images