Convolutional neural network inference processing device, convolutional neural network inference processing method, and convolutional neural network inference processing program

Abstract

A region extraction unit (32) extracts regions each having the maximum processable size as an input image and each obtained when an input image to be inputted to a convolutional neural network including a plurality of convolutional layers is divided into a plurality of regions in which the regions overlap at an overlap amount according to a shrinkage amount of the input image by the convolutional layers. A calculation unit (33) performs convolutional calculation using the plurality of convolutional layers which are layer-integrated by using data of the extracted regions as input data. An output unit (35) combines and outputs the calculation results respectively for the plurality of regions.

Description

[Technical Field]

[0001] The disclosed technology relates to a convolutional neural network inference processing device, a convolutional neural network inference processing method, and a convolutional neural network inference processing program. [Background technology]

[0002] In recent years, image recognition, object recognition, and other applications using convolutional neural networks (CNNs) have been applied to use cases that require real-time performance, low power consumption, and small footprint, such as surveillance cameras and drones. For this reason, active research and development has been conducted on the hardware used for CNN inference processing. For example, when considering use at the edge where memory resources are limited, reducing external memory bandwidth usage becomes important. One technology that can reduce external memory bandwidth usage is called the layer integration method. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] F. Indirli, A. Erdem and C. Silvano, "A Tile-based Fused-layer CNN Accelerator for FPGAs," 2020 27th IEEE International Conference on Electronics, Circuits and Systems (ICECS), 2020, pp. 1-4, doi: 10.1109 / ICECS49266.2020.9294981. Summary of the Invention [Problem to be solved by the invention]

[0004] The layer merging method, one of the conventional techniques, is useful in that it can reduce external memory bandwidth usage, but it has the problem of low scalability for input image size because it is limited by the internal memory capacity.

[0005] The disclosed technology has been made in consideration of the above points, and aims to expand the processable input image size in the layer integration method of CNN without increasing the internal memory capacity. [Means for solving the problem]

[0006] A first aspect of the present disclosure is a convolutional neural network inference processing device including: a region extraction unit that extracts each region when an input image to be input to a convolutional neural network including a plurality of convolutional layers is divided into a plurality of regions of the largest size that can be processed as the input image and with an overlap amount corresponding to the reduction of the input image by the convolutional layers; a calculation unit that performs a convolutional operation using the plurality of layer-integrated convolutional layers with data of the region extracted by the region extraction unit as input data; and an output unit that combines and outputs the calculation results by the calculation unit for each of the plurality of regions.

[0007] Furthermore, a second aspect of the present disclosure is a convolutional neural network inference processing method, in which an area extraction unit extracts each area when an input image to be input to a convolutional neural network including a plurality of convolution layers is divided into a plurality of areas of the largest size that can be processed as the input image and with an overlap amount corresponding to the reduction of the input image by the convolutional layers, an arithmetic unit uses data of the areas extracted by the area extraction unit as input data and performs a convolution operation using the plurality of layer-integrated convolutional layers, and an output unit combines and outputs the operation results by the arithmetic unit for each of the plurality of areas.

[0008] Furthermore, a third aspect of the present disclosure is a convolutional neural network inference processing program that causes a computer to function as: a region extraction unit that extracts each region when an input image to be input to a convolutional neural network including a plurality of convolutional layers is divided into a plurality of regions of the largest size that can be processed as the input image and in which the regions overlap with each other by an amount of overlap corresponding to the reduction of the input image by the convolutional layers; a calculation unit that performs a convolutional calculation using the plurality of layer-integrated convolutional layers, using data of the region extracted by the region extraction unit as input data; and an output unit that combines and outputs the calculation results by the calculation unit for each of the plurality of regions. [Effects of the Invention]

[0009] According to the disclosed technology, it is possible to expand the processable input image size in the layer integration method of CNN without increasing the internal memory capacity. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining tiles in a layer merging method of a conventional CNN. [Figure 2] FIG. 1 is a block diagram showing the hardware configuration of a convolutional neural network inference processing device according to the first and second embodiments. [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of a convolutional neural network inference processing device according to a first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of a region extracted from an input image. [Figure 5] FIG. 2 is a diagram for explaining the processing of a calculation unit and an output unit. [Figure 6] 1 is a flowchart showing the flow of a convolutional neural network inference process according to the first embodiment. [Figure 7] FIG. 10 is a block diagram showing an example of the functional configuration of a convolutional neural network inference processing device according to a second embodiment. [Figure 8]A diagram showing an example of dividing a region into nine tiles. [Figure 9] A diagram for explaining the processes of the generation unit, the calculation unit, and the margin extraction unit. [Figure 10] A flowchart showing the flow of convolutional neural network inference processing according to the second embodiment.

Modes for Carrying Out the Invention

[0011] Hereinafter, an example of an embodiment of the disclosed technology will be described while referring to the drawings. In each of the drawings, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for the convenience of explanation and may be different from the actual ratios.

[0012] <Regarding the Limitation of the Input Image Size in the CNN Layer Integration Method> Before explaining the details of the embodiment, the limitation of the input image size in the layer integration method of the prior art will be described.

[0013] The CNN layer integration method continuously performs processing in multiple layers in tile units obtained by dividing an input image into a grid. In the example of FIG. 1, it is an example of dividing the input image into nine tiles, tiles #0 to #8. For example, taking the tile located at the upper left of the input image as a starting point, the arithmetic processing of each tile is performed in the order from left to right and from top to bottom, that is, in the order of tiles #0 to #8.

[0014] Here, when performing a convolution operation on the boundary portion of a tile using, for example, a 3-pixel × 3-pixel kernel for each tile, a part of the data of the tile adjacent to the tile to be processed is used. This data is hereinafter referred to as "margin data". The margin data needs to be held in the internal memory during the convolution operation on the boundary portion of the tile. For example, as shown in FIG. 1, when tile #5 is the tile to be processed, the portion S indicated by the mesh is required as the margin data.

[0015] The internal memory stores not only the margin data required for the convolution operation of the tile being processed, but also the margin data required for the convolution operation of subsequent tiles to be processed. For example, when tile #5 is being processed, the portion T required for processing subsequent tiles #6 and #7 is also stored as margin data. While it is sufficient to store vertical margin data for the tile immediately preceding the tile being processed, it is necessary to store horizontal margin data up to the width of the input image.

[0016] However, internal memory capacity is limited. Therefore, if an input image has a width where the amount of horizontal extra data exceeds the internal memory capacity, the necessary extra data cannot be stored and convolution calculations cannot be performed. In other words, the input image size is limited to the size that can be calculated using the extra data that can be stored in the internal memory.

[0017] In the following embodiments, the maximum input size that can be conventionally processed is used as an area unit, and each area is processed with overlapping, thereby making it possible to process input images of a size equal to or larger than the maximum size that can conventionally be processed. Each embodiment will be described below.

[0018] First Embodiment FIG. 2 is a block diagram showing the hardware configuration of the convolutional neural network inference processing device 10 according to the first embodiment.

[0019] 2, the convolutional neural network inference processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0020] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a convolutional neural network inference processing program for executing the convolutional neural network inference processing described below.

[0021] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of storage devices such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including the operating system and various data.

[0022] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information. The display unit 16 is, for example, a liquid crystal display, and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0023] The communication I / F 17 is an interface for communicating with other devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0024] Next, a description will be given of the functional configuration of the convolutional neural network inference processing device 10. FIG. 3 is a block diagram showing an example of the functional configuration of the convolutional neural network inference processing device 10.

[0025] 3, the convolutional neural network inference processing device 10 has, as its functional components, an input image holding unit 31, a region extraction unit 32, a calculation unit 33, a calculation result holding unit 34, and an output unit 35. Each functional component is realized by the CPU 11 reading out a convolutional neural network inference processing program stored in the ROM 12 or storage 14, expanding the program in the RAM 13, and executing the program.

[0026] The input image storage unit 31 stores, in units of frames, input video images input to the convolutional neural network inference processing device 10. Hereinafter, one frame of the input video image will be referred to as an "input image." The input image storage unit 31 is provided in an external memory.

[0027] The region extraction unit 32 extracts each region when an input image to be input to a CNN including multiple convolution layers is divided into a plurality of regions of the maximum size that can be processed as a conventional input image and that overlap each other by a predetermined amount of overlap. The maximum size that can be processed as a conventional input image is the maximum size of an input image determined according to the internal memory capacity when this embodiment is not adopted.

[0028] The overlap amount must be determined taking into account that the output size is reduced relative to the input size due to the convolution operation. That is, the overlap amount is set to an amount corresponding to the reduction in the input image due to the convolution layer. Specifically, the region extraction unit 32 acquires CNN model information including information such as the number of layers included in the CNN, the layer type, and the kernel size, as well as layer merging application conditions including information on the layers to be merged using the layer merging method. Based on the acquired CNN model information and layer merging application conditions, the region extraction unit 32 calculates the overlap amount based on the number of convolution layers to be merged and the kernel size of each of the convolution layers to be merged.

[0029] More specifically, the region extraction unit 32 extracts the i-th convolution layer from among the 0 to N-1 layers to be integrated, and the kernel size of the i-th convolution layer is k i pixels x k i Pixels (k iIf , the overlap amount is calculated using the following formula.

[0030]

number

[0031] For example, when integrating two layers, convolutional layer A with a kernel size of 3 pixels x 3 pixels and convolutional layer B with a kernel size of 5 pixels x 5 pixels, the input image is reduced by one pixel's worth of rows and columns in convolutional layer A, and by two pixels' worth of rows and columns in convolutional layer B. In other words, the total is reduced by 1 + 2 = 3 pixels' worth of rows and columns. Therefore, by working backwards, the first layer of the convolutional layers to be integrated must perform calculations on a range that is three pixels larger in number of rows and columns than the input image. The same applies to all regions, so the amount of overlap between adjacent regions is 3 pixels x 2, as shown in the above formula.

[0032] FIG. 4 shows an example of a region extracted by the region extraction unit 32. As shown in the middle diagram of FIG. 4, the regions extracted from the input image overlap by the amount of overlap calculated as described above. The bottom diagram of FIG. 4 shows each region in the input image separately. Hereinafter, the j-th region extracted from the input image will be referred to as "region #j."

[0033] Note that the area extraction unit 32 only determines the range of each area, and does not need to copy the data of each area, etc. The subsequent processing is executed for each area.

[0034] The calculation unit 33 reads the data of the region extracted by the region extraction unit 32 into an internal memory, and performs convolution calculations using the read data of the region as input data through a plurality of layer-integrated convolution layers.

[0035] The output unit 35 combines and outputs the results of the calculations performed by the calculation unit 33 for each of the multiple regions.

[0036] The processing of the calculation unit 33 and the output unit 35 will be specifically described with reference to Fig. 5. The example of Fig. 5 shows a case where convolutional layers A, B, and C are integrated and the input image is divided into regions #0, #1, #2, and #3.

[0037] As process #0, the calculation unit 33 inputs area #0 read into the internal memory to convolutional layer A, executes a convolutional calculation by convolutional layer A, and stores calculation result #0A in the calculation result storage unit 34 provided in the internal memory. Next, the calculation unit 33 inputs calculation result #0A to convolutional layer B and deletes it from the calculation result storage unit 34, executes a convolutional calculation by convolutional layer B, and stores calculation result #0B in the calculation result storage unit 34. Next, the calculation unit 33 inputs calculation result #0B to convolutional layer C and deletes it from the calculation result storage unit 34, executes a convolutional calculation by convolutional layer C, and stores calculation result #0C in the calculation result storage unit 34.

[0038] The output unit 35 writes the operation result #0C of the final layer, convolutional layer C, to a predetermined area in the external memory.

[0039] When the processing #0 for region #0 is completed, the calculation unit 33 and the output unit 35 similarly execute processing #1 for region #1, processing #2 for region #2, and processing #3 for region #3 in that order, whereby the final calculation result obtained by combining the calculation results for each region is stored in the external memory.

[0040] Next, the operation of the convolutional neural network inference processing device 10 according to the first embodiment will be described.

[0041] 6 is a flowchart showing the flow of convolutional neural network inference processing by the convolutional neural network inference processing device 10. The convolutional neural network inference processing is performed by the CPU 11 reading a convolutional neural network inference processing program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. Note that the convolutional neural network inference processing shown in FIG. 6 is processing for one frame of the input video, i.e., one input image. By repeatedly executing the convolutional neural network inference processing shown in FIG. 6 on a frame-by-frame basis, processing is performed for the entire input video.

[0042] In step S11, the CPU 11 functions as the input image holding unit 31 to read and hold one frame of an input image from the input video image input to the convolutional neural network inference processing device 10.

[0043] Next, in step S12, the CPU 11, functioning as the region extraction unit 32, acquires CNN model information and layer merging application conditions, and calculates the amount of overlap based on the number of convolutional layers to be merged and the kernel size of each of the convolutional layers to be merged.

[0044] Next, in step S13, the CPU 11, functioning as the region extraction unit 32, extracts one region when an input image to be input to a CNN including multiple convolution layers is divided into multiple regions. The multiple regions are regions of the maximum size that can be processed as a conventional input image, and are regions that overlap with each other by the calculated overlap amount.

[0045] Next, in step S14, CPU 11, functioning as calculation unit 33, reads data of the extracted region into internal memory, performs convolution calculations using the read region data as input data through the multiple integrated convolution layers, and stores the calculation results in calculation result storage unit 34. Next, in step S15, CPU 11, functioning as output unit 35, writes the calculation results of the convolution layer that is the final layer of the multiple integrated convolution layers for the region to be processed into a predetermined area of ​​external memory.

[0046] Next, in step S16, the CPU 11, functioning as the region extraction unit 32, determines whether all regions have been extracted from the input image. If there are any regions that have not been extracted, the process returns to step S13 to extract the next region. If all regions have been extracted, the convolutional neural network inference process ends.

[0047] As described above, the convolutional neural network inference processing device according to the first embodiment calculates the overlap amount based on the number of convolutional layers to be merged and the kernel size of each of the convolutional layers to be merged. The convolutional neural network inference processing device also extracts each region when the input image is divided into multiple regions of the largest size that can be processed as a conventional input image, with the regions overlapping by the calculated overlap amount. The convolutional neural network inference processing device then loads the regions into internal memory and performs convolution operations using the layer merging method. While this increases the number of operations due to the calculation of the overlapping portions, it eliminates the need to store extra data at the boundary portions of the regions, allowing the processable input image size in the CNN layer merging method to be expanded without increasing the internal memory capacity.

[0048] Second Embodiment In the second embodiment, a case will be described in which the area in the first embodiment is further divided into tiles.

[0049] The hardware configuration of the convolutional neural network inference processing device 210 according to the second embodiment is similar to the hardware configuration of the convolutional neural network inference processing device 10 according to the first embodiment shown in FIG. 2, and therefore a description thereof will be omitted.

[0050] Next, we will explain the functional configuration of the convolutional neural network inference processing device 210. Fig. 7 is a block diagram showing an example of the functional configuration of the convolutional neural network inference processing device 210. Note that in the convolutional neural network inference processing device 210, the same functional configuration as that of the convolutional neural network inference processing device 10 according to the first embodiment will be assigned the same reference numerals and detailed explanations will be omitted.

[0051] 7, the convolutional neural network inference processing device 210 has, as its functional components, an input image holding unit 31, a region extraction unit 32, a tile extraction unit 236, a generation unit 237, a calculation unit 33, and a calculation result holding unit 34. The convolutional neural network inference processing device 210 also has a margin extraction unit 238, a margin holding unit 239, and an output unit 35. Each functional component is realized by the CPU 11 reading out a convolutional neural network inference processing program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

[0052] The tile extraction unit 236 extracts each tile from a plurality of tiles obtained by dividing the area extracted by the area extraction unit 32 into a grid. FIG. 8 shows an example in which area #0 is divided into nine tiles #0-0 to #0-8. Note that the tile extraction unit 236 only determines the range of each tile, and does not need to copy the data of each tile. Subsequent processing is performed for each tile.

[0053] The generation unit 237 reads the data of the tiles extracted by the tile extraction unit 236 into an internal memory, and combines the read tile data with the margin data held in the margin holding unit 239 to generate input data for the calculation unit 33.

[0054] The margin extraction unit 238 extracts margin data that will be required for processing the next and subsequent tiles from the calculation results of each of the convolutional layers to be merged, which are stored in the calculation result storage unit 34.

[0055] The margin holding unit 239 holds the margin data extracted by the margin extraction unit 238 .

[0056] The processing of the generation unit 237, the calculation unit 33, and the margin extraction unit 238 will be specifically described with reference to Fig. 9. In the example of Fig. 9, a case where tiles #0-4 are to be processed among tiles extracted from area #0 will be described, focusing on convolutional layer B in the same example as Fig. 5.

[0057] The generation unit 237 combines the calculation result #0-4A of the convolutional layer A for the tile #0-4 with the margin data stored in the margin storage unit 239 to generate input data for the convolutional layer B. The margin data here is the portion (the dotted portion in FIG. 9) of each of the calculation results #0-1A and #0-3A of the convolutional layer A for the tiles #0-1 and #0-3 adjacent to the tile #0-4 that is adjacent to the calculation result #0-4A. Note that in FIG. 9, the horizontal margin data of the calculation result #0-3 (the diagonal grid portion in FIG. 9) is the margin data used when the tile to be processed is tile #0-6.

[0058] The calculation unit 33 executes convolution processing using the convolution layer B on the input data generated by the generation unit 237, and stores the calculation result #0-4B in the calculation result storage unit 34. The margin extraction unit 238 extracts the portion of the calculation result #0-4B stored in the calculation result storage unit 34 that contacts adjacent tiles as margin data for the next or subsequent tiles to be processed, and stores this in the margin storage unit 239. Note that the vertical margin data of the calculation result #0-4B (the diagonally shaded portion in FIG. 9) is the margin data used when the tile to be processed is tile #0-5. Also, the horizontal margin data of the calculation result #0-4B (the grid portion in FIG. 9) is the margin data used when the tile to be processed is tile #0-7.

[0059] Next, the operation of the convolutional neural network inference processing device 210 according to the second embodiment will be described.

[0060] 10 is a flowchart showing the flow of convolutional neural network inference processing by the convolutional neural network inference processing device 210. The convolutional neural network inference processing is performed by the CPU 11 reading out a convolutional neural network inference processing program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. In the convolutional neural network inference processing according to the second embodiment, processes similar to those in the convolutional neural network inference processing according to the first embodiment (FIG. 6) are assigned the same reference numerals, and detailed descriptions thereof will be omitted.

[0061] After steps S11 to S13, an area is extracted from the input image. In the next step S211, the CPU 11, functioning as the tile extraction section 236, extracts one tile from a plurality of tiles obtained by dividing the extracted area into a grid pattern.

[0062] Next, in step S212, the CPU 11, as the generation unit 237, reads the extracted tile data into internal memory, and synthesizes the read tile data with the margin data held in the margin holding unit 239 to generate input data.

[0063] Next, in step S213, the CPU 11, functioning as the calculation unit 33, inputs the generated input data to the first layer of the plurality of integrated convolutional layers, performs a convolution calculation, and stores the calculation result in the calculation result storage unit 34. Next, in step S214, the CPU 11, functioning as the margin extraction unit 238, extracts margin data required for processing the next and subsequent tiles from the calculation results stored in the calculation result storage unit 34, and stores the data in the margin storage unit 239. The processes of steps S213 and S214 are sequentially executed for the next and subsequent layers of the plurality of integrated convolutional layers.

[0064] Next, in step S215, the CPU 11, as the output unit 35, writes the calculation results of the convolutional layer, which is the final layer of the multiple layer-merged convolutional layers, for the tile extracted in step S211 above, to a predetermined area in the external memory.

[0065] Next, in step S216, the CPU 11, functioning as the tile extraction unit 236, determines whether all tiles have been extracted from the region extracted in step S13. If there are tiles that have not been extracted, the process returns to step S211 to extract the next tile. If all tiles have been extracted, the process proceeds to step S16. If it is determined in step S16 that all regions have been extracted from the input image, the convolutional neural network inference process ends.

[0066] As described above, according to the convolutional neural network inference processing device of the second embodiment, as in the first embodiment, tiles are extracted by dividing an area extracted from an input image, and arithmetic processing is performed for each tile using layer merging. In the convolution processing for each tile, the extra data required for the convolution processing is extracted from the processed tile and the calculation results, stored, and used in the calculation processing for the next or subsequent tile. As a result, as in the first embodiment, the input image size that can be processed in the CNN layer merging method can be expanded without increasing the internal memory capacity.

[0067] In the first embodiment, since an extra storage unit is not required, the memory capacity can be reduced compared to the second embodiment.

[0068] Furthermore, the convolutional neural network inference process executed by the CPU after reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) whose circuit configuration can be changed after fabrication, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. The convolutional neural network inference process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0069] In addition, in each of the above embodiments, the convolutional neural network inference processing program is described as being pre-stored (installed) in the ROM 12 or the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0070] The following additional notes are provided regarding the above-described embodiments.

[0071] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: extracting each region when an input image to be input to a convolutional neural network including a plurality of convolutional layers is divided into a plurality of regions each having a maximum size that can be processed as the input image and overlapping with an amount of overlap corresponding to the reduction of the input image by the convolutional layers; performing a convolution operation using the data of the extracted region as input data and the plurality of layer-integrated convolution layers; The calculation results for each of the plurality of regions are combined and output. 1. A convolutional neural network inference processing device configured to:

[0072] (Additional note 2) A non-transitory recording medium storing a program executable by a computer to perform a convolutional neural network inference process, The convolutional neural network inference process includes: extracting each region when an input image to be input to a convolutional neural network including a plurality of convolutional layers is divided into a plurality of regions each having a maximum size that can be processed as the input image and overlapping with an amount of overlap corresponding to the reduction of the input image by the convolutional layers; performing a convolution operation using the data of the extracted region as input data and the plurality of layer-integrated convolution layers; The calculation results for each of the plurality of regions are combined and output. Non-transitory recording media, including [Explanation of symbols]

[0073] 10, 210 Convolutional Neural Network Inference Processing Unit 11 CPU 12 ROM 13 RAM 14. Storage 15 Input section 16 Display 17 Communication I / F 19 Bus 31 Input image storage unit 32 Region extraction part 33 Arithmetic section 34 Operation result holding section 35 Output section 236 Tile Extraction Unit 237 Generation part 238 Margin Extraction 239 Glue holder

Claims

1. a region extraction unit that extracts each region when an input image to be input to a convolutional neural network including a plurality of convolution layers is divided into a plurality of regions each having a maximum size that can be processed as the input image and overlapping with an amount of overlap corresponding to an amount of reduction of the input image by the convolution layers; a calculation unit that performs a convolution calculation using the plurality of layer-integrated convolution layers, using data of the region extracted by the region extraction unit as input data; an output unit that combines and outputs the calculation results by the calculation unit for each of the plurality of regions; 1. A convolutional neural network inference processing unit comprising:

2. 2. The convolutional neural network inference processing device according to claim 1, wherein the region extraction unit calculates the amount of overlap based on the number of the convolutional layers to be merged and a kernel size of each of the convolutional layers to be merged.

3. The region extraction unit is configured such that the kernel size of the i-th convolutional layer among the 0 to N-1 layers to be integrated is k ｉ pixels x k ｉ Pixels (k ｉ 3. The neural network inference processing device according to claim 2, wherein when n is an odd number, the amount of overlap is calculated by the following formula: [Equation 1]

4. a tile extraction unit that extracts each tile from a plurality of tiles obtained by dividing the area into a grid, The calculation unit uses data obtained by combining data of the tile to be processed extracted by the tile extraction unit and margin data of a tile adjacent to the tile to be processed as input data, and performs a convolution calculation using the plurality of layer-integrated convolution layers. The convolutional neural network inference processing device according to any one of claims 1 to 3.

5. 5. The convolutional neural network inference processing device according to claim 4, further comprising a margin extraction unit that extracts data of a portion of the calculation result for the tile to be processed that contacts an adjacent tile as the margin data for subsequent tiles to be processed.

6. the input image is stored in an external memory; the calculation unit reads the region extracted by the region extraction unit from the external memory into an internal memory, stores the calculation results of each of the plurality of convolutional layers to be integrated in the internal memory, and deletes the calculation results that are input to the next convolutional layer from the internal memory; The output unit sequentially executes a process of storing the operation results of the final layer of the plurality of convolution layers in an external memory for each of the plurality of regions, thereby combining the operation results by the operation unit for each of the plurality of regions. The convolutional neural network inference processing device according to any one of claims 1 to 3.

7. an area extraction unit extracts each area when an input image to be input to a convolutional neural network including a plurality of convolution layers is divided into a plurality of areas each having a maximum size that can be processed as the input image and overlapping with an amount of overlap corresponding to an amount of reduction of the input image by the convolutional layers; a calculation unit performs a convolution calculation using the plurality of layer-integrated convolution layers, using data of the region extracted by the region extraction unit as input data; An output unit combines and outputs the calculation results by the calculation unit for each of the plurality of regions. Convolutional neural network inference processing method.

8. Computer, a region extraction unit that extracts each region when an input image to be input to a convolutional neural network including a plurality of convolution layers is divided into a plurality of regions each having a maximum size that can be processed as the input image and overlapping with an amount of overlap corresponding to an amount of reduction of the input image by the convolution layers; a calculation unit that performs a convolution calculation using the plurality of layer-integrated convolution layers, using data of the region extracted by the region extraction unit as input data; an output unit that combines and outputs the calculation results by the calculation unit for each of the plurality of regions; A convolutional neural network inference processing program to function as a

Images