Convolutional neural network inference processing device, convolutional neural network inference processing method, and convolutional neural network inference processing program

Abstract

A determination unit (33) determines a processing mode based on a sliding method when a processing range in input data input to each layer of a CNN is collectively processed by a plurality of convolutional calculators and the entire input data is processed by sliding the processing range. For each layer of the CNN, the determination unit (33) determines a processing mode in which an external memory band usage becomes minimum, from among a first mode in which input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in a channel direction, and a third mode in which the kernel is fixed and the input data is slid in vertical and horizontal directions. A setting unit (34) sets the determined processing mode in a calculation unit (35). The calculation unit (35) performs a convolutional calculation with respect to the input data on the basis of the set processing mode.

Description

[Technical Field]

[0001] The disclosed technology relates to a convolutional neural network inference processing device, a convolutional neural network inference processing method, and a convolutional neural network inference processing program. [Background technology]

[0002] In recent years, image recognition, object recognition, and other applications using convolutional neural networks (CNNs) have been applied to use cases that require real-time performance, low power consumption, and small footprint, such as surveillance cameras and drones. For this reason, research and development into hardware used for CNN inference processing has been actively pursued. For example, when considering use at the edge where memory resources are limited, reducing external memory bandwidth usage is important.

[0003] In hardware used for CNN inference processing, a common method is to use multiple convolutional processors deployed in parallel to process a certain range of the input image as a processing range, and then to process the entire input image by sliding the processing range. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] F. Indirli, A. Erdem and C. Silvano, "A Tile-based Fused-layer CNN Accelerator for FPGAs," 2020 27th IEEE International Conference on Electronics, Circuits and Systems (ICECS), 2020, pp. 1-4, doi: 10.1109 / ICECS49266.2020.9294981. Summary of the Invention [Problem to be solved by the invention]

[0005] When multiple parallel convolutional processors process a certain range of an input image as a processing range, there are multiple patterns for setting the processing range and the slide order, and the amount of external memory bandwidth used varies depending on the pattern.While which pattern is optimal depends on the CNN layer, a pre-fixed pattern is generally applied, and it is not necessarily the optimal pattern that minimizes external memory bandwidth usage.

[0006] The disclosed technology has been made in consideration of the above points, and aims to reduce the amount of external memory bandwidth used during CNN processing. [Means for solving the problem]

[0007] A first aspect of the present disclosure is a convolutional neural network inference processing device, comprising: a setting unit that sets, for each layer of the convolutional neural network, a processing mode that minimizes external memory bandwidth usage among a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, based on a sliding method for processing the entire input data, the processing mode being input to each layer of the convolutional neural network, the processing mode being based on a sliding method for processing the entire input data by collectively processing a processing range of the input data having values ​​in the vertical, horizontal, and channel directions using a plurality of convolutional computing units; and a computing unit that performs a convolution operation on the input data based on the processing mode set by the setting unit.

[0008] Furthermore, a second aspect of the present disclosure is a convolutional neural network inference processing method, in which a setting unit sets, for the plurality of convolutional computing units for each layer of the convolutional neural network, a processing mode that minimizes external memory bandwidth usage from among a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, and a computing unit performs a convolutional operation on the input data based on the processing mode set by the setting unit.

[0009] Furthermore, a third aspect of the present disclosure is a convolutional neural network inference processing program that causes a computer to function as a setting unit that sets, for the multiple convolutional computing units for each layer of the convolutional neural network, a processing mode that minimizes external memory bandwidth usage from among a first mode in which the input data is fixed and a kernel is slid, a second mode in which the output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, based on a sliding method when input data is input to each layer of a convolutional neural network, the processing range of the input data having values ​​in the vertical, horizontal, and channel directions, is collectively processed by multiple convolutional computing units, and the entire input data is processed by sliding the processing range, the processing mode being based on a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, [Effects of the Invention]

[0010] According to the disclosed technology, it is possible to reduce the amount of external memory bandwidth used during CNN processing. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram illustrating an example of input data, a kernel, and output data. [Figure 2] FIG. 1 is a block diagram showing the hardware configuration of a convolutional neural network inference processing device according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram illustrating an example of the functional configuration of a convolutional neural network inference processing device according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram illustrating a calculation formula for calculating the external memory bandwidth usage of input data, output data, and a kernel. [Figure 5] FIG. 2 is a diagram illustrating an example of input data, a kernel, and output data. [Figure 6] 10A to 10C are diagrams illustrating examples of processing in each mode. [Figure 7] 10A and 10B are diagrams illustrating an example of calculation of external memory bandwidth usage of input data, output data, and a kernel. [Figure 8] 1 is a flowchart showing the flow of a convolutional neural network inference process according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0013] <About processing modes> Figure 1 shows an example of an input feature map (hereinafter referred to as "input data") input to a convolutional layer of a CNN, a kernel applied in that convolutional layer, and an output feature map (hereinafter referred to as "output data") output from that convolutional layer.

[0014] When multiple parallel convolution processors process a certain range of input data collectively as a processing range, as shown in Figure 1, the processing range can have multiple combinations of vertical, horizontal, and channel lengths under the condition that the volume is constant. In Figure 1, the shaded area represents the processing range. Also in Figure 1, the vertical size of the input data is iH, the horizontal size is iW, and the channel size is iCh. The vertical size of the processing range in the input data is iH_t, the horizontal size is iW_t, and the channel size is iCh_t. Also, the number of kernels is oCh, the number of kernels applied to the processing range is iCh, and the channel size of the kernel applied to the processing range is icH_t. Also, the vertical size of the output data is oH, the horizontal size is oW, and the channel size is oCh. The vertical size of the output data corresponding to the processing range is oH_t, oW_t, and oCh_t. The same applies to the following figures.

[0015] In addition, the methods for sliding the processing range can be broadly divided into the following three types. (1) Fixing the input data and sliding with kernel priority (2) Fix the output data and slide the input data and kernel in the channel direction. (3) Fix the kernel and slide the input data vertically and horizontally.

[0016] In this embodiment, processing modes based on each of the three sliding methods are provided. Hereinafter, the processing mode based on sliding method (1), i.e., the processing mode that prioritizes reuse of input data, will be referred to as "mode 1." Furthermore, the processing mode based on sliding method (2), i.e., the processing mode that prioritizes reuse of output data, will be referred to as "mode 2." Furthermore, the processing mode based on sliding method (3), i.e., the processing mode that prioritizes reuse of kernels, will be referred to as "mode 3." Note that "reuse" refers to the repeated use of data loaded into internal memory during computation.

[0017] In this embodiment, a processing range and a processing mode that minimize the external memory bandwidth usage are set for each layer of the CNN, and convolution operations are performed.

[0018] <Configuration of a convolutional neural network inference processing device> FIG. 2 is a block diagram showing the hardware configuration of the convolutional neural network inference processing device 10 according to this embodiment.

[0019] 2, the convolutional neural network inference processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0020] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a convolutional neural network inference processing program for executing the convolutional neural network inference processing described below.

[0021] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of storage devices such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including the operating system and various data.

[0022] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information. The display unit 16 is, for example, a liquid crystal display, and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0023] The communication I / F 17 is an interface for communicating with other devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0024] Next, a description will be given of the functional configuration of the convolutional neural network inference processing device 10. FIG. 3 is a block diagram showing an example of the functional configuration of the convolutional neural network inference processing device 10.

[0025] 3, the convolutional neural network inference processing device 10 has, as its functional components, an input image holding unit 31, a kernel holding unit 32, a determination unit 33, a setting unit 34, a calculation unit 35, and a calculation result holding unit 36. Each functional component is realized by the CPU 11 reading out a convolutional neural network inference processing program stored in the ROM 12 or storage 14, expanding the program in the RAM 13, and executing the program.

[0026] The input image storage unit 31 stores, in units of frames, input video images input to the convolutional neural network inference processing device 10. Hereinafter, one frame of the input video image will be referred to as an "input image." The input image storage unit 31 is provided in an external memory.

[0027] The kernel storage unit 32 stores kernel data, i.e., weighting coefficients for convolution operations, and is provided in an external memory.

[0028] The determination unit 33 determines the processing mode that minimizes the external memory bandwidth usage from among the three processing modes. Specifically, the determination unit 33 determines the processing range and processing mode that minimizes the external memory bandwidth usage under constraints according to the internal memory capacity.

[0029] More specifically, the determination unit 33 solves a minimization problem in which the external memory bandwidth usage (input data + output data + kernel) in each mode is used as an objective function. The determination unit 33 calculates an objective function for each mode by adding up the external memory bandwidth usage of the input data, output data, and kernel calculated based on the calculation formula shown in FIG. 4. In the objective function, iW_t, iH_t, iCh_t, oW_t, oH_t, and oCh_t are variables. Also, iW, iH, iCh, oW, oH, and oCh are parameters of each layer of the CNN model. The determination unit 33 externally acquires CNN model information including these parameters.

[0030] The determination unit 33 also externally acquires constraint conditions for minimizing the objective function. The constraint conditions limit the domain of each variable of the objective function depending on the internal memory capacity for input data, the internal memory capacity for output data, and the internal memory capacity for the kernel. Specifically, the constraint conditions are as follows:

[0031] 1. iW_t × iH_t × iCh_t × number of bits per pixel ≦ internal memory capacity for input data 2. oW_t × oH_t × oCh_t × number of bits per pixel ≦ Internal memory capacity for output 3. k × k × iCh_t × oCh_t × number of bits per pixel ≦ kernel internal memory capacity The internal memory capacity for input data, the internal memory capacity for output data, and the internal memory capacity for the kernel are given externally, and k is the size of the kernel (number of vertical and horizontal pixels).

[0032] The determination unit 33 compares the minimum values ​​of the objective function (total external memory bandwidth usage) in each mode, and determines the mode with the smallest minimum value of the objective function as the processing mode.

[0033] For example, in mode 1, the input data is fixed and the output data and kernel are slid, so the input data only needs to be read from external memory once, and the external memory bandwidth usage is only the data capacity of the original input data. On the other hand, the output data and kernel require multiple reads and writes to external memory depending on the processing range, so the external memory bandwidth usage is a constant multiple of the data capacity of the original output data and kernel.

[0034] An example of processing in each mode will be described using an example of input data, kernels, and output data shown in Fig. 5. In the example of Fig. 5, the input data and output data are divided into four tiles in the vertical and horizontal directions, and each tile is numbered 0 to 3. The channel direction is divided into two halves, and each tile is numbered 0 and 1. Below, each processing range of the input data and output data will be represented by connecting the tile number and the channel direction number with a hyphen (for example, "0-0"). Furthermore, for kernels, the first kernel will be numbered 0 and the second kernel will be numbered 1, and these numbers and the channel direction number will be represented by connecting them with a hyphen (for example, "0-0"). In the example of FIG. 5, iW=64, iW_t=32, iH=64, iH_t=32, iCh=64, iCh_t=32, and oW=64, oW_t=32, oH=64, oH_t=32, oCh=64, oCh_t=32.

[0035] FIG. 6 shows an example of processing in each mode in the example of FIG. 5, and FIG. 7 shows an example of calculation of input data, output data, and external memory bandwidth usage of the kernel.

[0036] In mode 1, as shown in Fig. 6, for example, processing is performed on all data in the processing range (0-0) of iCH=0 to 31 of tile 0 of the input data, and then processing moves on to the processing range (0-1) of iCH=32 to 63 of tile 0 of the input data. This means that each piece of input data only needs to be read from the external memory once. That is, as shown in Fig. 7, in mode 1, the external memory bandwidth usage for input data is (original input data capacity) x 1.

[0037] On the other hand, for kernel data, for example, the same kernel data (0-0) is required when processing iCH=0 to 31 and oCH=0 to 31 of tile 0 and when processing iCH=0 to 31 and oCH=0 to 31 of tile 1, so it needs to be re-read from the external memory. That is, as shown in Fig. 7, the external memory bandwidth usage of the kernel is (original kernel data capacity) x (number of tiles).

[0038] On the other hand, for output data, for example, when processing iCH=0 to 31 and oCH=0 to 31 of tile 0, and when processing iCH=32 to 63 and oCH=0 to 31 of tile 0, the respective calculation results (0-0) must be added together. Therefore, it is necessary to read from external memory, add the results, and then write them. In this example, oCH is processed in two separate steps, resulting in a total of three reads and writes: two writes and one read. In other words, as shown in Figure 7, the external memory bandwidth usage for output data is (original output data capacity) x (2 x number of divisions in the channel direction - 1). The external memory bandwidth usage is calculated similarly for modes 2 and 3.

[0039] The setting unit 34 sets the processing range and processing mode determined for each layer by the determination unit 33 in the calculation unit 35 .

[0040] The calculation unit 35 reads into its internal memory the input data and kernel data required for the convolution calculation from the input image storage unit 31 and the kernel storage unit 32. The calculation unit 35 then performs the convolution calculation on the input image based on the processing range and processing mode set for each layer by the setting unit 34, and stores the calculation result in the calculation result storage unit 36.

[0041] The calculation result holding unit 36 ​​holds the calculation results obtained by the calculation unit 35 and outputs the held calculation results to the outside. The input image holding unit 31 is provided in an external memory.

[0042] <Function of the convolutional neural network inference processing device> Next, the operation of the convolutional neural network inference processing device 10 according to this embodiment will be described.

[0043] 8 is a flowchart showing the flow of convolutional neural network inference processing by the convolutional neural network inference processing device 10. The convolutional neural network inference processing is performed by the CPU 11 reading a convolutional neural network inference processing program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. Note that the convolutional neural network inference processing shown in FIG. 8 is processing for one frame of the input video sequence, i.e., for one input image. By repeatedly executing the convolutional neural network inference processing shown in FIG. 8 on a frame-by-frame basis, processing is performed for the entire input video sequence.

[0044] In step S11, the CPU 11, functioning as the input image storage unit 31, reads and stores one frame of input image from the input video image input to the convolutional neural network inference processing device 10. The CPU 11, functioning as the kernel storage unit 32, also reads and stores kernel data.

[0045] Next, in step S12, the CPU 11, functioning as the determination unit 33, acquires CNN model information and constraint conditions, and determines, for each CNN layer, a processing range and a processing mode that minimizes external memory bandwidth usage under constraint conditions according to the internal memory capacity. Next, in step S13, the CPU 11, functioning as the setting unit 34, sets, in the calculation unit 35, the processing range and processing mode determined for each layer by the determination unit 33.

[0046] Next, in step S14, the CPU 11, functioning as the calculation unit 35, performs a convolution calculation on the input image based on the processing range and processing mode set for each layer by the setting unit 34, and stores the calculation result in the calculation result holding unit 36. Next, in step S15, the CPU 11, functioning as the calculation result holding unit 36, outputs the calculation result to the outside, and the convolutional neural network inference process ends.

[0047] As described above, the convolutional neural network inference processing device according to this embodiment processes the entire input data, which is input to each layer of the CNN and has values ​​in the vertical, horizontal, and channel directions, by collectively processing the processing range of the input data using multiple convolutional computing units and sliding the processing range. The convolutional neural network inference processing device selects a processing mode based on the sliding method in this case, which minimizes external memory bandwidth usage from among a first mode in which the input data is fixed and the kernel is slid; a second mode in which the output data is fixed and the input data and the kernel are slid in the channel direction; and a third mode in which the kernel is fixed and the input data is slid vertically and horizontally. The processing mode is then selected for each layer of the CNN and set to multiple convolutional computing units. This reduces the external memory bandwidth usage during CNN processing.

[0048] In the above embodiment, the determination unit provided inside the convolutional neural network inference processing device determines the processing mode that minimizes the external memory bandwidth usage, but the present invention is not limited to this. The determination unit may be provided in an external device, and the setting unit may read the processing mode determined by the determination unit of the external device and set it in the calculation unit.

[0049] Furthermore, the convolutional neural network inference process executed by the CPU after reading the software (program) in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) whose circuit configuration can be changed after fabrication, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. The convolutional neural network inference process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0050] In the above embodiment, the convolutional neural network inference processing program is pre-stored (installed) in the ROM 12 or the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0051] The following additional notes are provided regarding the above-described embodiments.

[0052] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: a processing mode based on a sliding method in which a processing range in input data input to each layer of a convolutional neural network, the input data having values ​​in vertical, horizontal, and channel directions, is collectively processed by a plurality of convolutional operators, and the entire input data is processed by sliding the processing range, the processing mode being one that minimizes external memory bandwidth usage among a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, and the processing mode is set in the plurality of convolutional operators for each layer of the convolutional neural network; Based on the set processing mode, a convolution operation is performed on the input data. 1. A convolutional neural network inference processing device configured to:

[0053] (Additional note 2) A non-transitory recording medium storing a program executable by a computer to perform a convolutional neural network inference process, The convolutional neural network inference process includes: a processing mode based on a sliding method in which a processing range in input data input to each layer of a convolutional neural network, the input data having values ​​in vertical, horizontal, and channel directions, is collectively processed by a plurality of convolutional operators, and the entire input data is processed by sliding the processing range, the processing mode being one that minimizes external memory bandwidth usage among a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, and the processing mode is set in the plurality of convolutional operators for each layer of the convolutional neural network; Based on the set processing mode, a convolution operation is performed on the input data. Non-transitory recording media, including [Explanation of symbols]

[0054] 10. Convolutional Neural Network Inference Processing Unit 11 CPU 12 ROM 13 RAM 14. Storage 15 Input section 16 Display section 17 Communication I / F 19 Bus 31 Input image storage unit 32 Kernel storage unit 33 Decision Section 34 Setting section 35 Arithmetic section 36 Operation result holding section

Claims

1. a setting unit that sets, for each layer of the convolutional neural network, a processing mode that minimizes external memory bandwidth usage among a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, based on a sliding method when input data is input to each layer of the convolutional neural network, the input data having values ​​in vertical, horizontal, and channel directions, and a processing range in the input data is collectively processed by a plurality of convolutional computing units, and the entire input data is processed by sliding the processing range; a calculation unit that performs a convolution operation on the input data based on the processing mode set by the setting unit; a convolutional neural network inference processing unit comprising:

2. The convolutional neural network inference processing device according to claim 1 , wherein the setting unit sets a processing range and a processing mode that minimize the amount of bandwidth used in the external memory under constraints according to an internal memory capacity.

3. 3. The convolutional neural network inference processing device according to claim 2, wherein the constraints are that the data amount of the processing range of the input data is equal to or less than an internal memory capacity for the input data, the data amount of the kernel corresponding to the processing range is equal to or less than an internal memory capacity for the kernel, and the data amount of the output data for the processing range is equal to or less than an internal memory capacity for the output data.

4. 4. The convolutional neural network inference processing device according to claim 1, wherein the external memory bandwidth usage is a value calculated by a calculation formula based on the data amounts of the input data, the kernel, and the output data, and the number of times each of the input data, the kernel, and the output data is read from the external memory and the number of times each of the input data, the kernel, and the output data is written to the external memory according to the processing mode.

5. a determination unit that determines the processing mode, The setting unit sets the processing mode determined by the determination unit. The convolutional neural network inference processing device according to any one of claims 1 to 3.

6. 4. The convolutional neural network inference processing device according to claim 1, wherein the setting unit accepts and sets the processing mode designated from outside.

7. the setting unit sets, for each layer of the convolutional neural network, to the plurality of convolutional operators a processing mode that minimizes external memory bandwidth usage, from among a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, the processing mode being based on a sliding method when input data is input to each layer of the convolutional neural network, the input data having values ​​in vertical, horizontal, and channel directions, and the entire input data is processed by sliding the processing range; A calculation unit performs a convolution operation on the input data based on the processing mode set by the setting unit. Convolutional neural network inference processing method.

8. Computer, a setting unit that sets, for each layer of the convolutional neural network, a processing mode that minimizes external memory bandwidth usage among a first mode in which the input data is fixed and a kernel is slid, a second mode in which output data is fixed and the input data and the kernel are slid in the channel direction, and a third mode in which the kernel is fixed and the input data is slid in the vertical and horizontal directions, based on a sliding method when input data is input to each layer of the convolutional neural network, the input data having values ​​in vertical, horizontal, and channel directions, and a processing range in the input data is collectively processed by a plurality of convolutional computing units, and the entire input data is processed by sliding the processing range; a calculation unit that performs a convolution operation on the input data based on the processing mode set by the setting unit; A convolutional neural network inference processing program to function as a

Images