Convolutional Neural Network Processing Unit

Abstract

To enhance processing efficiency of hardware which performs both of conventional convolution processing or the convolution processing in a point direction, and the convolution processing in a depth direction.SOLUTION: A convolutional neural network processing device comprises: data holding means which holds at least a part of data of a plurality of channels in a processing object layer among a plurality of layers; and a plurality of calculation means which perform multiply-accumulate operation using the data on one channel of the processing object layer, and a factor corresponding to the processing object layer in parallel, respectively. The convolutional neural network processing device selects whether first processing which inputs the data on one channel in the processing object layer stored in the data holding means into one calculation means among the plurality of calculation means is performed, or second processing which inputs the data into each of the plurality of calculation means in parallel is performed, based on information which identifies processing in the processing object layer.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a convolutional neural network processing device, and more particularly to hardware for performing processing according to a convolutional neural network. [Background technology]

[0002] In recent years, advances in deep learning have improved the accuracy of image recognition. Convolutional Neural Networks (CNN) are known as a method used in deep learning. CNNs are used in a variety of applications, but processing using CNNs requires many product-sum operations. For this reason, when implementing CNNs in embedded systems such as mobile terminals or in-vehicle devices, it is necessary to shorten the processing time by reducing the amount of transfer of feature data and weighting coefficients, or by efficiently performing product-sum operations.

[0003] Non-Patent Document 1 proposes using a combination of depthwise convolution and pointwise convolution as a technique for reducing the amount of calculation. In addition, Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3 propose a method for implementing depthwise convolution by hardware. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Pat. No. 10,360,470 [Non-patent literature]

[0005] [Non-Patent Document 1] Andrew G. Howard, et al., "MobileNets: Efficient convolutional neural networks for mobile vision applications.", CoRR, abs / 1704.04861 (2017). [Non-Patent Document 2] Bing Liu, et al. "An FPGA-Based CNN Accelerator Integrating Depthwise Separable Convolution.", Electronics, 2019, vol. 8, p. 281. [Non-Patent Document 3] Yunxuan Yu, et al. "Light-OPU: An FPGA-based overlay processor for lightweight convolutional neural networks." Proceedings of the 2020 ACM / SIGDA International Symposium on Field-Programmable Gate Arrays, p. 122 (2020). Summary of the Invention [Problem to be solved by the invention]

[0006] There is still a demand for hardware that can perform both conventional convolution processing and depthwise convolution processing in a short time. For example, in the methods described in Patent Document 1 and Non-Patent Document 2, depthwise convolution processing is realized by performing conventional convolution processing with some of the weighting coefficients set to zero, so the advantage of depthwise convolution processing, which can reduce the amount of calculation, is not fully utilized.

[0007] An object of the present invention is to improve the processing efficiency of hardware that performs both conventional convolution processing or point-wise convolution processing, and depth-wise convolution processing. [Means for solving the problem]

[0008] In order to achieve the object of the present invention, a processing apparatus according to an embodiment of the present invention comprises the following components: A processing device for performing calculations in a convolutional neural network having multiple layers, a control means for sequentially selecting each of the plurality of hierarchical layers as a processing target hierarchical layer; a data storage means for storing at least a portion of data of a plurality of channels in the hierarchical layer to be processed; a plurality of calculation means each performing a multiply-and-accumulate operation in parallel using data of one channel of the hierarchical layer to be processed and a coefficient corresponding to the hierarchical layer to be processed; a selection means for selecting, based on information specifying a process in the hierarchical layer to be processed, whether to perform a first process in which data of each of one channel in the hierarchical layer to be processed stored in the data holding means is input in parallel to a corresponding one of the plurality of calculation means and the plurality of calculation means performs parallel calculations, or a second process in which data of the plurality of channels in the hierarchical layer to be processed stored in the data holding means is read out sequentially and data of each of one channel in the hierarchical layer to be processed is input in parallel to each of the plurality of calculation means; In the first process, one of the plurality of calculation means calculates a value corresponding to data of one channel of a layer next to the layer to be processed. death, the data holding means is a memory, the calculation means comprises an arithmetic core including a multiplier and an adder; the processing device comprises a chip on which the memory and the arithmetic core are provided, The selection means is An address issuing unit provided in the chip that specifies an address in the memory of data to be input to the arithmetic core It is characterized by: Effect of the Invention

[0009] The processing efficiency of hardware that performs both conventional convolution processing or point-wise convolution processing and depth-wise convolution processing can be improved. [Brief description of the drawings]

[0010] [Figure 1] 1 is a flowchart of a convolution process according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the structure of a convolutional neural network according to an embodiment. [Diagram 3] FIG. 1 is a block diagram showing an example of the arrangement of an image processing device according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of the arrangement of a processing device according to an embodiment. [Diagram 5] 5A to 5C are diagrams for explaining a procedure of a convolution process according to an embodiment. [Figure 6] 1 is a flowchart of a convolution process according to an embodiment. [Figure 7] FIG. 2 is a block diagram showing an example of the arrangement of a processing device according to an embodiment. [Figure 8] 5A to 5C are diagrams for explaining a procedure of a convolution process according to an embodiment. [Figure 9] FIG. 1 is a diagram for explaining the relationship between a convolutional neural network and convolution processing. [Figure 10] FIG. 2 is a block diagram showing an example of the arrangement of a processing device according to an embodiment. [Figure 11] FIG. 2 is a block diagram showing an example of the arrangement of a processing device according to an embodiment. [Figure 12] A diagram explaining pipeline processing of a convolutional neural network. [Figure 13] FIG. 13 is a diagram showing an example of pseudocode for block-based convolution processing. [Figure 14] FIG. 2 is a block diagram showing an example of the arrangement of a processing device according to an embodiment. [Figure 15] 5A to 5C are diagrams for explaining a convolution process performed in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [Embodiment 1] An embodiment of the present invention relates to a processing device that performs calculations in a convolutional neural network. First, an image processing device that uses such a processing device to perform image processing will be described. Fig. 3 is a block diagram showing an example of the configuration of such an image processing device 300.

[0013] The storage unit 302 can store image data. The storage unit 302 may store programs or other data in addition to image data. The storage unit 302 may be, for example, a hard disk, a flexible disk, a CD-ROM, a CD-R or a DVD, a memory card, a CF card, a smart media, an SD card, a memory stick, an xD picture card, or a USB memory. Also, a part of a RAM 308 described later may be used as the storage unit 302. Furthermore, the storage unit 302 may be configured virtually, and for example, a storage device of an external device connected via a communication unit 303 described later may be used as the storage unit 302.

[0014] The display unit 304 is a device that displays an image before or after image processing, or an image such as a GUI. For example, a CRT or a liquid crystal display can be used as the display unit 304. The display unit 304 may be a display device that is external to the image processing device 300 and is connected via a cable or the like.

[0015] The input unit 301 is a device that accepts input of instructions or data from a user. The input unit 301 is, for example, a keyboard, a pointing device, or a button. The display unit 304 and the input unit 301 may be the same device. For example, a touch screen device may function as the display unit 304 and the input unit 301. In this case, an input to the touch screen can be treated as an input to the input unit 301.

[0016] The CPU 306 controls the overall operation of the image processing device 300. The ROM 307 and the RAM 308 provide the CPU 306 with programs, data, a working area, and the like required for processing by the image processing device 300. When a program required for processing described below is stored in the storage unit 302 or the ROM 307, the program is read into the RAM 308 and then executed. When the image processing device 300 receives a program via the communication unit 303, the program is recorded in the storage unit 302 and then read into the RAM 308, or is read directly from the communication unit 303 into the RAM 308 and then executed. Although only one CPU 306 is shown in FIG. 3, the image processing device 300 may have multiple CPUs.

[0017] In accordance with commands from the CPU 306, the image processing unit 309 can read image data written to the storage unit 302, perform processing to adjust the range of pixel values ​​for the image data, and write the resulting image data to the RAM 308.

[0018] The CNN processing unit 305 corresponds to a processing device that performs calculations in a convolutional neural network. The CNN processing unit 305 can perform calculations in the convolutional neural network on a processing target image stored in the RAM 308 or the like. Specifically, the CNN processing unit 305 can perform convolution processing (filter processing, steps S101 to S116) including product-sum calculations on image data stored in the RAM 308 according to the flowchart of FIG. 1 described later. Then, the CNN processing unit 305 outputs the obtained results to the storage unit 302 (or the RAM 308).

[0019] The CPU 306 can perform image processing on the processing target image based on the processing result obtained by performing an operation in the convolution neural network on the processing target image. For example, the CPU 306 can perform image processing such as image correction processing or image recognition processing on image data (still image or moving image having multiple frame images) based on the result of the convolution processing. The CPU 306 can store the obtained image processing result in the RAM 308. According to this embodiment, since the processing by the CNN processing unit 305 can be efficiently performed, the time from when the processing target image is input to the image processing device 300 until the CPU 306 completes image processing on the processing target image can be shortened. Therefore, the image processing device according to this embodiment is advantageous for performing real-time processing on images.

[0020] The communication unit 303 is an interface (I / F) for communication between devices. In Fig. 3, the input unit 301, the storage unit 302, and the display unit 304 are all included in one image processing device 300. However, each of the units arranged in a distributed manner may be connected by a communication path according to a predetermined communication method to form an image processing system. The image processing device 300 may have various components other than those described above, but the description thereof will be omitted.

[0021] (Network configuration) FIG. 2 shows an example of the structure of a convolutional neural network (hereinafter, sometimes simply referred to as a network). The network has multiple hierarchical layers, and each layer contains data of one or more channels (for example, one or more feature images). Then, data of the previous layer is processed to generate data of the subsequent layer. In the convolutional neural network, data of the next layer of the layer to be processed is generated by performing processing including convolution processing on data of at least one layer to be processed. Although the data of each layer is not limited to an image, the following describes a case where the calculation in the convolutional neural network is performed on the image to be processed. In this case, the data of the layer to be processed, which is in the middle of the network, is a feature image obtained in the process of the calculation in the convolutional neural network.

[0022] A processing device according to an embodiment of the present invention can perform calculations in a network having such a structure. The structure of the network can be represented by information on each layer (the amount of calculation for multiply-and-accumulate, the size and number of feature images, the type of processing, etc.). The number of layers in the network shown in FIG. 2 is four, and each layer is called layers 1 to 4. Each layer has four channels, and each channel contains one feature image. Furthermore, one feature image contains multiple feature data (for example, pixel data for each pixel of the feature image).

[0023] In the computation of the network, convolution processing is performed using the weight coefficients obtained by learning and the pixel data of the feature image. The convolution processing is a product-sum operation, which includes multiple multiplications and cumulative additions. In Figure 2, the arrows represent product-sum operations.

[0024] The convolution process for calculating the feature image of the next layer of the processing target layer is performed using the feature image of the processing target layer and the coefficients corresponding to the processing target layer. In many convolution processes, information on multiple feature images of the processing target layer is used to calculate one feature image. Such a product-sum operation can be performed according to formula (1).

number

[0025] In the point-wise convolution process, the weighting coefficient (W 0,0 (m,n)~W X-1,Y-1 The number of elements (m, n) is 1 × 1. In the depthwise convolution process, the product-sum calculation formula is expressed by formula (2).

number

[0026] FIG. 2 also shows the type of convolution processing corresponding to each layer of the network and the size of the filter used in the convolution processing. The convolution processing in layer 1 is a convolution processing using a filter having a size of 3×3. The convolution processing in layer 1 is not a convolution processing in the depth direction, but each feature image in layer 1 is used to generate one feature image in layer 2. Specifically, in layer 1, multiple feature images 201 and weighting coefficients are used to perform convolution processing based on equation (1), thereby multiple feature images 202 in layer 2 are generated. FIG. 9 shows an example of a network and convolution processing. Feature data is extracted from the same position of four feature images 901 in layer 1, and a product-sum operation is performed using the extracted feature data. The result is the feature data in the same position of the feature image 902 in layer 2.

[0027] The convolution process in layer 2 is also a convolution process using a filter having a size of 3×3. On the other hand, the convolution process in layer 2 is a depthwise convolution process, and one feature image in layer 2 is used to generate one feature image in layer 3. In this way, the amount of calculation for the convolution process in layer 2 is less than that in layer 1. Specifically, in layer 2, a convolution process is performed using multiple feature images 202 and weighting coefficients based on equation (2), thereby generating multiple feature images 203 in layer 3.

[0028] The convolution process in layer 3 is not a depthwise convolution process, but the size of the filter used is 1×1. Specifically, in layer 3, a convolution process is performed using a plurality of feature images 203 and a weighting factor based on equation (1), thereby generating a plurality of feature images 204 in layer 4. In this way, each filter and feature image is configured hierarchically, and the type of convolution process and the filter size may differ for each layer. Furthermore, arithmetic processing in the network can be performed in hierarchical order.

[0029] (Processing device configuration) An example of the configuration of a processing device according to an embodiment of the present invention will be described below with reference to Fig. 4. The lower part of Fig. 4 shows the configuration of a CNN processing unit 305, which is a processing device according to an embodiment of the present invention. The CNN processing unit 305 includes a control unit 401, a storage unit 402, a feature data storage unit 403, a coefficient storage unit 404, a readout unit 405, a convolution processing unit 406, a post-processing unit 407, and a result storage unit 408.

[0030] The storage unit 402 can store feature images (feature data), weighting coefficients, and network structure information. For example, the storage unit 402 can read all input feature images (for example, all feature images 201 in layer 1) from the storage unit 302 or the RAM 308 and store them. In addition, the storage unit 402 can read feature images (for example, all feature images 202 in layer 2) obtained by the convolution processing unit 406 and the post-processing unit 407 from the result storage unit 408 and store them. In addition, the storage unit 402 can read all weighting coefficients used in the convolution process from the storage unit 302 and store them. The network structure information is information that represents the structure of the network, and can be read from the storage unit 302. In the following description, the network structure information has information on the number of channels in each layer and information that specifies the process in each layer. The information that specifies the process in each layer can indicate the type of convolution process, that is, whether or not to perform convolution process in the depth direction.

[0031] The feature data storage unit 403 stores at least a portion of the data of a plurality of channels in the hierarchical layer to be processed. In this embodiment, the feature data storage unit 403 stores a feature image I(m) in the hierarchical layer to be processed. A feature image to be used in one step of processing, which will be described later, is transferred from the storage unit 402 to the feature data storage unit 403. The coefficient storage unit 404 can store at least a portion of the coefficients to be used in the product-sum operation in the hierarchical layer to be processed. In this embodiment, the coefficient storage unit 404 stores the weighting coefficient W(m) of a filter to be used in the convolution process. x,y 2, the size of the filter can be set for each layer. Weighting coefficients used in one step of processing, which will be described later, are transferred from the storage unit 402 to the coefficient storage unit 404. At least one of the feature data storage unit 403 and the coefficient storage unit 404 may be a memory, and this memory may be provided on the same chip as the convolution processing unit 406 or the calculation core 416.

[0032] The convolution processing unit 406 calculates the result of the convolution processing using the weighting coefficient and the feature data based on the formula (1) or (2). The upper part of FIG. 4 shows a detailed configuration of the convolution processing unit 406. The convolution processing unit 406 has a plurality of convolution subunits 410. Each of the convolution subunits 410 performs a multiply-and-accumulate operation in parallel using data of one channel of the processing target layer and a coefficient corresponding to the processing target layer. Each of the convolution subunits 410 includes a calculation core 416, a feature data storage unit 411, a coefficient storage unit 412, and a result storage unit 415, and has a function of accumulating the product of the feature data and the weighting coefficient. The calculation core 416 has a multiplier 413 and an adder 414 used for the multiply-and-accumulate operation.

[0033] For example, when performing a convolution process on one feature image, the feature data of the feature image and the weighting coefficients used in the convolution process are sequentially supplied to the feature data storage unit 411 and the coefficient storage unit 412 by the readout unit 405. That is, the feature data and weighting coefficients required for calculating data of one pixel of the image representing the result of the convolution process are supplied to the feature data storage unit 411 and the coefficient storage unit 412. The feature data storage unit 411 and the coefficient storage unit 412 supply the feature data and weighting coefficients one by one to the multiplier 413. Then, the product of the feature data and the weighting coefficients calculated by the multiplier 413 is accumulated by the adder 414, thereby calculating data of one pixel of the image representing the result of the convolution process. The result storage unit 415 can store the accumulated value of the products calculated by the multiplier 413 for each pixel of the image representing the result of the convolution process. By repeating such a process for each pixel of the image representing the result of the convolution process, data of each pixel of the image representing the result of the convolution process is calculated.

[0034] The post-processing unit 407 generates a next-layer feature image I(n) based on the result of the convolution processing calculated by the convolution processing unit 406. For example, the post-processing unit 407 can perform activation processing and pooling processing on the result of the convolution processing. Note that, instead of the post-processing unit 407, another processing unit such as the CPU 306 may perform the activation processing or pooling processing. Also, one or both of the activation processing and the pooling processing may be omitted. The result storage unit 408 stores the processing result by the post-processing unit 407, i.e., the next-layer feature image I(n).

[0035] The reading unit 405 selects whether to perform a first processing in which the data is input to one of the multiple convolution subunits 410, or a second processing in which the data is input in parallel to each of the multiple convolution subunits 410. The reading unit 405 makes such a selection based on information specifying the processing in the processing target layer. The information specifying the processing in the processing target layer is indicated in the network structure information that the reading unit 405 acquires from the storage unit 402. For example, as indicated in a frame 420, the reading unit 405 selects whether to perform the convolution processing according to Equation (1) or the depthwise convolution processing according to Equation (2) depending on the network structure.

[0036] Furthermore, the read unit 405 can control data supply from the feature data storage unit 403 and the coefficient storage unit 404 to the multiple convolution subunits 410. When performing convolution processing according to equation (1), the read unit 405 selects the second processing. In this case, the read unit 405 controls inputting data of one channel in the processing target layer (e.g., feature image I(m)) to each of the multiple convolution subunits 410. When performing convolution processing according to equation (2), the read unit 405 selects the first processing. In this case, the read unit 405 controls inputting data of one channel in the processing target layer (e.g., feature image I(m)) to one of the multiple convolution subunits 410. The read unit 405 can specify an address in the feature data storage unit 403 of data to be input to the convolution subunit 410 or the arithmetic core 416. Specifically, the reading unit 405 can control the reading of the feature data and the weighting coefficients by generating an in-memory address for accessing specific data in the feature data storage unit 403 and the coefficient storage unit 404 and transferring the in-memory address to the convolution subunit 410. The reading unit 405 functioning as such an address issuing unit may be provided on the same chip as the convolution subunit 410 or the arithmetic core 416.

[0037] Next, the flow of processing performed by the CNN processing unit 305 will be described with reference to the flowchart in Fig. 1. The following steps S101 to S116 can be performed by a CPU or a sequencer included in the control unit 401. In the following, each of the multiple feature images in the hierarchical layer to be processed is referred to as an input feature image. Also, each of the multiple feature images in the next hierarchical layer to the hierarchical layer to be processed, which are generated by performing processing on the hierarchical layer to be processed, is referred to as an output feature image.

[0038] In step S101, the control unit 401 reads out from the RAM 308 an input image (eg, a plurality of feature images in layer 1) to be subjected to calculation in the network, weighting coefficients, and network structure information, and stores them in the storage unit 402.

[0039] In step S102, the control unit 401 starts a loop of layers. In the first loop, the first layer (layer 1) is selected as the layer to be processed. In the subsequent loops, the next layer (layer 2 or later) is selected as the layer to be processed.

[0040] In step S103, the control unit 401 determines whether or not to perform depthwise convolution processing based on the network structure information stored in the storage unit 402. If depthwise convolution processing is not to be performed, the process proceeds to step S105. If depthwise convolution processing is to be performed, the process proceeds to step S114.

[0041] In step S105, the control unit 401 initializes each of the result storage units 415 that accumulate the results of the convolution process. The control unit 401 can perform the initialization by setting the process results stored in the result storage units 415 to zero.

[0042] In step S106, the control unit 401 starts a loop of input feature images. In the first loop, a process is performed using the first input feature image as the processing target. In the subsequent loops, a process is performed using the next input feature image (second or later).

[0043] In step S107, the control unit 401 reads out some of the input feature images from the storage unit 402 and transfers them to the feature data storage unit 403. The control unit 401 also reads out some of the weighting coefficients from the storage unit 402 and transfers them to the coefficient storage unit 404. As described above, the control unit 401 can transfer the feature images and weighting coefficients used in one step of processing to the feature data storage unit 403 and the coefficient storage unit 404. Here, one step of processing refers to the convolution processing in step S108 or the convolution processing in step S116 in one input feature image loop. When the convolution processing in the depth direction is not performed, one common input feature image and multiple sets of weighting coefficients corresponding to each convolution processing are used in step S108. Therefore, the control unit 401 can transfer one input feature image that is the processing target in the loop of steps S106 to S109 to the feature data storage unit 403. Furthermore, the control unit 401 can transfer weighting coefficients for convolution processing performed on one input feature image to be processed to the coefficient holding unit 404. To shorten the transfer time, the weighting coefficients and the feature image can be transferred in parallel, but when there is a sufficient transfer band, the weighting coefficients and the input image may be transferred in sequence.

[0044] In step S108, the convolution processing unit 406 performs convolution processing using the input feature image stored in the feature data storage unit 403 and the weighting coefficients stored in the coefficient storage unit 404 in accordance with the control signal from the control unit 401. In this embodiment, a common input feature image to be processed is input to each of the multiple convolution subunits 410. Also, different weighting coefficients can be input to each of the multiple convolution subunits 410. A weighting coefficient of the convolution processing to be performed to generate a corresponding output feature image among the multiple output feature images is input to one convolution subunit 410. In this way, each convolution subunit 410 can perform convolution processing using different weighting coefficients in parallel on the common input feature image.

[0045] More specifically, the feature data storage unit 411 and the coefficient storage unit 412 store specific feature data and weighting coefficients stored in the feature data storage unit 403 and the coefficient storage unit 404 under the control of the readout unit 405. Each of the arithmetic cores 416 can perform a product-sum operation by calculating the product of one input data and one coefficient and accumulating the calculated products. Therefore, the data supply from the feature data storage unit 411 and the coefficient storage unit 412 is controlled so that data and coefficients are input one by one to each of the multiple arithmetic cores 416. By repeating the product-sum operation using the feature data and weighting coefficients stored in the feature data storage unit 411 and the coefficient storage unit 412, data of one pixel of an image representing the result of the convolution process can be calculated. The readout unit 405 can control the readout order of the feature data and weighting coefficients from the feature data storage unit 403 and the coefficient storage unit 404 so as to obtain the result of the convolution process for the input feature image.

[0046] In step S109, the control unit 401 determines whether the loop of the input feature images has ended. If the processing of all the input feature images has not ended, the process returns to step S107, and the processing using the next input feature image is performed. In this manner, the input of one common channel of data from the feature data storage unit 403 to the multiple convolution subunits 410 in parallel in step S107 is repeated for each of the multiple channel of data in the hierarchical layer to be processed.

[0047] In step S109, if the processing of all input feature images is completed, the process proceeds to step S110. At this time, each of the convolution subunits 410 outputs a calculation result corresponding to one channel in the next layer of the layer to be processed, using the data of each channel in the layer to be processed that is sequentially input. That is, one convolution subunit 410 performs convolution processing on each of the multiple input feature images in each loop, and accumulates the results in the result storage unit 415. In this way, the result storage unit 415 can store data in which the results of the convolution processing on each of the multiple input feature images are accumulated. The data stored in the result storage unit 415 corresponds to the result of convolution processing on multiple input feature images using a filter having a dimension in the channel (depth) direction (for example, point-wise convolution processing, or three-dimensional convolution processing using a 3×3×3 filter).

[0048] In step S110, the post-processing unit 407 performs activation processing based on the result of the convolution processing stored in the result storage unit 415 according to a control signal from the control unit 401. The post-processing unit 407 can calculate the result of the activation processing according to, for example, equation (3). f(x) = 0(x<0) x(x≧0) ……(3) In formula (3), f(x) is an activation function and x is input data. In this example, ReLU (Rectified Linear Unit) is used as the activation function, but the activation function is not limited to ReLU. For example, other nonlinear functions or quantization functions may be used as the activation function. In addition, the post-processing unit 407 may adjust the size of the output feature image by performing a pooling process based on the result of the activation process according to the information of the hierarchy indicated in the network structure information, as necessary.

[0049] The post-processing unit 407 can generate an output feature image by performing such processing on the results of the convolution processing stored in the result storage unit 415, and store the output feature image in the result storage unit 408. The post-processing unit 407 can perform such processing on the results of the convolution processing stored in each of the multiple result storage units 415. As a result, the result storage unit 408 stores multiple output feature images respectively corresponding to the results of the convolution processing stored in each of the multiple result storage units 415.

[0050] In step S111, the control unit 401 stores the output feature image stored in the result storage unit 408 in the storage unit 402. This output feature image is a feature image in the next hierarchical level to be processed.

[0051] In step S113, the control unit 401 determines whether the layer loop has ended. If the processing of all layers has ended, the convolution processing ends. If not, the processing returns to step S103, the processing target layer is changed, and processing of the next layer is started. In this case, the output feature image stored in the holding unit 402 in step S111 is used as the input feature image in the processing of the next layer.

[0052] In step S114, the control unit 401 initializes each of the result storage units 415 in the same manner as in step S105.

[0053] In step S115, the control unit 401 reads out some of the input feature images from the storage unit 402 and transfers them to the feature data storage unit 403. In addition, the control unit 401 reads out some of the weighting coefficients from the storage unit 402 and transfers them to the coefficient storage unit 404. When performing depth-wise convolution processing, in step S116, multiple input feature images and multiple sets of weighting coefficients corresponding to each input feature image are used. Therefore, the control unit 401 can transfer multiple input feature images to be processed in step S116 to the feature data storage unit 403. In addition, the control unit 401 can transfer multiple sets of weighting coefficients to the coefficient storage unit 404. As in step S107, the weighting coefficients and the feature images may be transferred in parallel or sequentially.

[0054] In step S116, the convolution processing unit 406 performs convolution processing using the input feature image stored in the feature data storage unit 403 and the weighting coefficients stored in the coefficient storage unit 404 in accordance with the control signal from the control unit 401. In this embodiment, data of a plurality of channels in the processing target layer stored in the feature data storage unit 403 is input in parallel to a plurality of convolution subunits 410 so that data of one channel is input to a corresponding one of the convolution subunits 410. In this manner, corresponding input feature images different from each other are input to each of the plurality of convolution subunits 410. Also, different weighting coefficients can be input to each of the plurality of convolution subunits 410. In this manner, the respective convolution subunits 410 can perform convolution processing using different weighting coefficients on different input feature images in parallel. The readout unit 405 can control the readout order of the feature data and weighting coefficients from the feature data storage unit 403 and the coefficient storage unit 404 in the same manner as in step S108, except that the data used for the convolution processing is different. The arithmetic core 416 can also perform a multiply-and-accumulate operation in the same manner as in step S108. In this manner, each of the multiple convolution subunits 410 can use data of one channel in the processing target layer to output a calculation result corresponding to one channel in the next layer of the processing target layer.

[0055] Steps S117 and S118 can be performed similarly to steps S110 and S111, resulting in the generation and storage of an output feature image.

[0056] As described above, when performing the convolution process in the depth direction, the loop of steps S106 to S109 can be omitted, thereby improving the processing efficiency.

[0057] (Example of convolution processing) According to the configuration of this embodiment, a plurality of feature images can be processed in parallel. The process according to the flowchart of FIG. 1 will be further described below with reference to the example shown in FIG. 5. FIG. 5 shows how the two layers (layer 2 and layer 3) shown in FIG. 2 are processed. Here, an example of processing four feature images in parallel will be described. In FIG. 5, the solid line blocks indicate the processing targets in each step, and the solid line arrows indicate the convolution processes performed in each step.

[0058] 5A shows the processing of layer 2. In step S103, it is determined that depthwise convolution processing is to be performed, and the processing proceeds to step S114. In step S116, each of the four input feature images is input in parallel to a corresponding one of the multiple convolution subunits 410. Each of the multiple convolution subunits 410 performs depthwise convolution processing on the corresponding input feature image, thus generating four output feature images in parallel. The convolution processing of layer 2 is completed in one step.

[0059] 5B to 5E show the processing of layer 3. In step S103, it is determined that the convolution processing in the depth direction is not performed, and the processing proceeds to step S105. After that, the four input feature images are input one by one as a common input feature image to the multiple convolution subunits 410 in parallel. Each of the convolution subunits 410 repeatedly performs convolution processing on the four input feature images input in sequence and accumulates the processing results, thus generating four output feature images in parallel. That is, one convolution subunit 410 sequentially performs convolution processing on feature image (3,1), feature image (3,2), feature image (3,3), and feature image (3,4), and accumulates the processing results to generate feature image (4,1). The convolution processing of layer 3 is completed in four steps, and the required processing time and calculation amount are four times that of layer 2.

[0060] In the above, a case has been described in which the processing parallelism (the number of convolution subunits 410) and the number of output feature images are the same, but the processing parallelism and the number of output feature images may be different. As an example, when the number of output feature images is greater than the processing parallelism, the processing device may num The convolution subunits 410 are arranged in a parallel processing order (PE num First, we will explain the case where the number of output feature images is greater than the number of output feature images. num (For example, 1st sheet to PE num The process of steps S105 to S111 (or steps S114 to S118) can be performed to generate an output feature image (the 1st image). num (e.g. PE num + 1st to 2xPE num The processing of steps S105 to S111 (or steps S114 to S118) can be repeated to generate an output feature image (1st output feature image). By repeating the processing in this manner, it is possible to generate output feature images in a number greater than the processing parallelism. Note that although the case where the number of feature images is a multiple of the processing parallelism will be described in the third and subsequent embodiments, the number of feature images does not have to be a multiple of the processing parallelism.

[0061] As described above, by determining whether or not to perform depthwise convolution processing in step S103, both normal convolution processing and depthwise convolution processing can be efficiently performed. In particular, when performing depthwise convolution processing, multiple arithmetic units can be used in parallel, making it possible to complete the processing in a shorter time than in the prior art.

[0062] [Modification of the first embodiment] The configuration of the CNN processing unit 305 is not limited to the configuration shown in FIG. 4. Another configuration example of the CNN processing unit 305 will be described with reference to FIG. 10. In this configuration example, the CNN processing unit 305 includes a plurality of feature data storage units 411, each of which corresponds to one of the plurality of arithmetic cores 416. As described above, each of the plurality of arithmetic cores 416 can perform a multiply-and-accumulate operation in parallel using data of one channel of the processing target layer and a coefficient corresponding to the processing target layer. The CNN processing unit 305 according to this configuration example includes a multiplexer (MUX) that can select an input to the arithmetic core 416 from a plurality of sources. Such a MUX can switch between a first process and a second process. The MUX may be provided on the same chip as the arithmetic core 416.

[0063] Figure 10 shows the num 10, convolution subunits 410 are shown. As shown in FIG. 10, one convolution subunit 1 has a corresponding feature data storage unit 1, a coefficient storage unit 1, an arithmetic core 1, and a result storage unit 1. Here, input data to the arithmetic core 1 is transferred from the corresponding feature data storage unit 1. Also, the arithmetic cores 2 to PE num The input data to the feature data storage unit 1 or the corresponding feature data storage unit 2 to PE num The input source is switched by the MUX 1001. In this modification, the MUX 1001 can select the input of feature data to some of the arithmetic cores 416 in accordance with the network structure.

[0064] For example, in the second process that does not perform depthwise convolution processing, one of the multiple feature data storage units 411 supplies data of one common channel in the processing target hierarchy to each of the multiple arithmetic cores 416. That is, the reading unit 405 transfers one piece of feature data to the feature data storage unit 1 and num The weight coefficients of the set are stored in coefficient storage units 1 to PE num As a result of input selection using MUX1001, num The input to each of the arithmetic cores 416 is the same, and common feature data is transferred from the feature data storage unit 1 to the arithmetic cores 1 to PE num Each calculation core 416 performs a multiplication and accumulation operation using the same feature data and a different weighting coefficient, and num The results of the convolution processes are calculated.

[0065] On the other hand, in the first process of performing the convolution process in the depth direction, each of the plurality of feature data storage units 411 supplies data of one channel in the processing target hierarchy to the corresponding calculation core 416. That is, the reading unit 405 reads different feature data from the feature data units 1 to PE num Transfer to PE num The weight coefficients of the set are stored in coefficient storage units 1 to PE num As a result of input selection using MUX1001, num The inputs to the individual arithmetic cores 416 are different, and different feature data are stored in the feature data storage units 1 to PE num From calculation core 1 to PE num Each calculation core 416 performs a multiplication and accumulation operation using different feature data and different weighting coefficients, and num The results of the convolution processes are calculated.

[0066] [Embodiment 2] In the first embodiment, when the depthwise convolution process is not performed, a common input feature image is input in parallel to the multiple convolution subunits 410. In the second embodiment, when the depthwise convolution process is not performed, different input feature images are input in parallel to the multiple convolution subunits 410 so as to generate one output feature image by parallel processing. The following describes configurations different from the first embodiment. Configurations similar to those in the first embodiment are given the same reference symbols, and descriptions thereof will be omitted.

[0067] The lower part of Fig. 7 shows the configuration of the CNN processing unit 305, which is a processing device according to the second embodiment. The processing device according to this embodiment further includes a sum calculation unit 701 that accumulates the calculation results of each of the multiple convolution subunits 410. The sum calculation unit 701 can calculate the sum of the results of the convolution processing for multiple input feature images. Since the sum calculation unit 701 has a configuration that accumulates the results of the convolution processing generated in parallel, it is possible to generate output feature images in parallel.

[0068] As in the first embodiment, the reading unit 405 selects whether to perform convolution processing according to equation (1) or depth-wise convolution processing according to equation (2) depending on the network structure. However, the processing order is different from that in the first embodiment, as shown in box 702.

[0069] The summation calculation unit 701 accumulates the calculation results of each of the multiple convolution subunits 410. The summation calculation unit 701 selects whether to perform the first processing or the second processing according to information specifying the processing in the processing target layer. In the first processing, the calculation results of each of the multiple convolution subunits 410 are output as calculation results corresponding to each channel in the next layer of the processing target layer. When performing the depthwise convolution processing according to the formula (2), the first processing is selected, and the summation calculation unit 701 outputs the calculation results of each of the multiple convolution subunits 410 as data corresponding to each feature image in the next layer of the processing target layer. In the second processing, the calculation result obtained by the summation calculation unit 701 is output as a calculation result corresponding to one channel in the next layer. When performing the convolution processing according to the formula (1), the second processing is selected, and the summation calculation unit 701 accumulates the calculation results of each of the multiple convolution subunits 410, and outputs the accumulated result as data corresponding to one feature image in the next layer.

[0070] The flow of processing performed by the CNN processing unit 305 in this embodiment will be described with reference to the flowchart in Fig. 6. Steps S101 to S103 are performed in the same manner as in the first embodiment.

[0071] Step S105 is performed in the same manner as in the first embodiment. In step S602, the control unit 401 starts a loop of output feature images. In the first loop, a process is performed using the first output feature image as the processing target. In the subsequent loops, a process is performed using the next output feature image (second or later).

[0072] In step S107, the control unit 401 reads out some of the input feature images and some of the weighting coefficients from the storage unit 402 and transfers them to the feature data storage unit 403 and the coefficient storage unit 404. When depth-wise convolution processing is not performed in this embodiment, multiple input feature images and multiple sets of weighting coefficients corresponding to each input feature image are used in step S108. Therefore, the control unit 401 can transfer multiple input feature images to be processed in step S108 to the feature data storage unit 403. In addition, the control unit 401 can transfer multiple sets of weighting coefficients to the coefficient storage unit 404. Unlike the first embodiment, the number of input feature images to be transferred does not change depending on the selection result in step S103.

[0073] In step S108, the convolution processing unit 406 performs convolution processing using the input feature image stored in the feature data storage unit 403 and the weighting coefficients stored in the coefficient storage unit 404, in the same manner as in the first embodiment, except that the data used in the convolution processing is different. In this embodiment, data of a plurality of channels in the processing target layer stored in the feature data storage unit 403 is input in parallel to a plurality of convolution subunits 410 so that data of one channel is input to a corresponding one of the convolution subunits 410. In this manner, input feature images different from each other are input to each of the plurality of convolution subunits 410. Also, weighting coefficients different from each other can be input to each of the plurality of convolution subunits 410. In this manner, each of the convolution subunits 410 can perform convolution processing using weighting coefficients different from each other on input feature images different from each other in parallel.

[0074] In step S108 of this embodiment, the summation calculation unit 701 further accumulates the results of the convolution process using the input feature image and the weighting coefficient output from each of the multiple convolution subunits 410 while adding them. In this way, the summation calculation unit 701 can generate data in which the results of the convolution process for each of the multiple input feature images are accumulated. The data thus generated corresponds to the result of the convolution process for the multiple input feature images using a filter having a dimension in the channel (depth) direction (for example, a point-wise convolution process, or a three-dimensional convolution process using a 3×3×3 filter). Depending on the selection result in step S103, it is determined whether the summation calculation unit 701 calculates and outputs the sum of the convolution process results.

[0075] In step S605, the control unit 401 determines whether the loop of the output feature images is finished. If the results of the convolution process corresponding to all the output feature images have not been generated, the process returns to step S107, and the convolution process is performed to generate the next output feature image. If the results of the convolution process corresponding to all the output feature images have been generated, the process proceeds to step S110.

[0076] In step S110, the post-processing unit 407 generates an output feature image based on the convolution result generated by the summation calculation unit 701, similarly to the first embodiment. Steps S111 and S113 are performed in the same manner as the first embodiment.

[0077] The processes in steps S114 to S118 performed when performing the depthwise convolution process are the same as those in embodiment 1. As described above, in step S116, the summation calculation unit 701 does not calculate the summation of the results of the convolution process, but outputs the calculation results by each of the multiple convolution subunits 410 as data corresponding to each feature image in the next layer of the processing target layer.

[0078] (Example of convolution processing) The configuration of this embodiment also allows multiple feature images to be processed in parallel. The process according to the flowchart of Fig. 6 will be further described below with reference to the example shown in Fig. 9. Fig. 8 shows how the two layers (layer 2 and layer 3) shown in Fig. 2 are processed, similar to Fig. 5.

[0079] 8A shows the processing of layer 2. Here, convolution processing in the depth direction is performed as in the first embodiment.

[0080] 8(B) to (E) show the processing of layer 3. In step S103, it is determined that the depthwise convolution processing is not performed, and the processing proceeds to step S105. In FIG. 8(B), four input feature images are input one by one to the plurality of convolution subunits 410 in parallel. Each of the convolution subunits 410 performs a convolution process on the input feature image, and the sum calculation unit 701 calculates the sum of the convolution processing results, thereby generating one output feature image by parallel processing. That is, each of the convolution subunits 410 performs a convolution process on the feature image (3,1), feature image (3,2), feature image (3,3), or feature image (3,4) in parallel, and the feature image (4,1) is generated by accumulating the processing results. By performing similar processing using different weighting coefficients, feature image (4,2), feature image (4,3), and feature image (4,4) are generated in sequence, as shown in Figures 8(C) to (E). The convolution processing of layer 3 is completed in four steps, and the required processing time and amount of calculations are four times that of layer 2. Note that when there are four or more input feature images, the four or more input feature images can be input in parallel to multiple convolution subunits 410.

[0081] In the above, the case where the processing parallelism (the number of convolution subunits 410) and the number of input feature images are the same has been described, but the processing parallelism and the number of input feature images may be different. As an example, when the number of input feature images is greater than the processing parallelism, the processing device may num The convolution subunits 410 are arranged in a parallel processing order (PE num) is larger than the number of output feature images. num (For example, 1st sheet to PE num The processing of steps S602 to S605 can be performed so as to generate the results of the convolution processing corresponding to each output feature image using the input feature image (page 11). num (e.g. PE num + 1st to 2xPE num The processing in steps S602 to S605 can be repeated to generate the results of the convolution processing corresponding to each output feature image using the input feature image (1st to PE num The result of the convolution process using the input feature image and PE num + 1st to 2xPE num The result of the convolution process using the input feature image of the 1st image can be accumulated. In this way, the 1st image to 2×PE num A result of the convolution process using the 1st input feature image can be obtained. By such a repetitive process, an output feature image based on the result of the convolution process using all the input feature images can be obtained. When performing the convolution process according to the formula (2), it can be performed in the same manner as in the first embodiment. Note that the number of feature images may or may not be a multiple of the processing parallelism.

[0082] As described above, by determining whether or not to perform depthwise convolution processing in step S103, both normal convolution processing and depthwise convolution processing can be efficiently performed. In particular, when performing depthwise convolution processing, multiple arithmetic units can be used in parallel, making it possible to complete the processing in a shorter time than in the prior art.

[0083] [Modification of the second embodiment] The configuration of the CNN processing unit 305 is not limited to the configuration shown in FIG. 6. Another configuration example of the CNN processing unit 305 will be described with reference to FIG. 11. In this configuration example, the CNN processing unit 305 includes a plurality of result storage units 415, each corresponding to one of the plurality of arithmetic cores 416. As described above, each of the plurality of arithmetic cores 416 can perform a multiply-and-accumulate operation in parallel using data of one channel of the processing target layer and a coefficient corresponding to the processing target layer. The CNN processing unit 305 according to this configuration example includes a multiplexer (MUX) that can select one of a plurality of outputs obtained using the plurality of arithmetic cores 416. Such a MUX can switch between the first processing and the second processing. The MUX may be provided on the same chip as the arithmetic cores 416.

[0084] Figure 11 shows the num Convolutional subunits 410 are shown. num The convolution sub-units each have a corresponding feature data storage unit, coefficient storage unit, calculation core, and result storage unit. The readout unit 405 reads different feature data from the feature data storage units 1 to PE. num , and the different weighting coefficients are transferred to the coefficient storage units 1 to PE num Calculation core 1~PE num The input data to the corresponding feature data storage units 1 to PE num In this case, the PE num Each of the calculation cores 416 calculates the result of the convolution process using the input feature data and the corresponding weighting coefficients, thus forming a PE num The summation calculation unit 701 obtains the results of convolution processing. num The results of the convolution processes are added together.

[0085] In this modification, the MUX 1101 can select an input to the result storage unit 1 according to the structure of the network. Specifically, an output selected from the output from the arithmetic core 1 and the output from the summation calculation unit 701 based on the outputs from the multiple arithmetic cores 416 is input to the result storage unit 1.

[0086] For example, in the second process that does not perform the depthwise convolution process, each of the multiple arithmetic cores 416 outputs the calculation result to the summation calculation unit 701, and the summation calculation unit 701 outputs the result of the summation to one of the multiple result storage units 415. That is, as a result of the input selection using the MUX 1101, the summation calculation unit 701 outputs the summation result to one of the multiple result storage units 415. num The results of the convolution processing are input to the processing result storage unit 1. In this case, the result storage units 2 to PE num is not used.

[0087] On the other hand, in the first process of performing the depthwise convolution process, each of the multiple arithmetic cores 416 outputs the arithmetic result to the corresponding result storage unit 415. That is, as a result of the input selection using the MUX 1101, num The results of the convolution processes are stored in the result storage units 1 to PE num is entered into

[0088] [Embodiment 3] In the above-described embodiment, each step shown in FIG. 1 or FIG. 6 is processed in order. However, by using hardware capable of performing pipeline processing, the processing according to each embodiment may be performed by pipeline processing. In the following, a case where the processing according to the first embodiment is performed by pipeline processing will be described with reference to FIG. 12. In the following, the PE num The following describes a case where hardware capable of generating output feature images in parallel is used. For simplicity, the number of input feature images is assumed to be OC=2PE. num However, the number of input feature images is not particularly limited.

[0089] FIG. 12(A) shows an example in which depthwise convolution processing is not performed. In this example, there are IC input feature images and OC output feature images. During the time from 0 cycle to T cycle, the first input feature image and the weighting coefficient used in the convolution processing are transferred. During the time from T cycle to 2T cycle, the second input feature image and the weighting coefficient used in the convolution processing are transferred, and the first input feature image is multiplied by the weighting coefficient. During the time from 2T cycle to 3T cycle, the third input feature image and the weighting coefficient used in the convolution processing are transferred, the second input feature image is multiplied by the weighting coefficient, and the multiplication result for the first input feature image is accumulated. During the time from 3T cycle to (IC+2)×T cycle, a similar process is repeated, and PE num The results of convolution processing are obtained. During the time from (IC+2) × T cycles to (IC+3) × T cycles, the PE obtained by convolution processing for IC input feature images is num The activation process is then performed using each result. num Output feature images (1st to (OC / 2)th) are generated. By performing the same process once again, output feature images (1st to (OC / 2)+1st to OCth) can be obtained.

[0090] FIG. 12B shows an example of performing convolution processing in the depth direction. In this example, the number of input feature images is IC, the number of output feature images is OC, and IC=OC=2PE num From cycle 0 to cycle T, the first sheet to PE num The input feature image and the weight coefficients used for the convolution process are transferred. During the time from cycle T to cycle 2T, num + The input feature images from the 1st to ICth images and the weighting coefficients used for convolution processing are transferred, and the 1st to PE num The weighting coefficient for the input feature image is multiplied. During the time from 2T cycle to 3T cycle, num+The weighting coefficients are multiplied for the 1st to ICth input feature images, and the 1st to PE num The multiplication results for the input feature image are accumulated. During the time from 3T cycle to 4T cycle, num The multiplication results for the +1st to +ICth input feature images are accumulated. num PE obtained by convolution processing for the input feature image num The activation process is performed using each result, and PE num Output feature images (1st to IC / 2) are generated. During the time from 4T cycle to 5T cycle, PE num +PE obtained by convolution processing of the 1st to ICth input feature images num The activation process is then performed using each result. num output feature images (IC / 2+1th image to ICth image) are generated.

[0091] As in the example of FIG. 5, the processing time when depthwise convolution processing is performed is shorter than when depthwise convolution processing is not performed.

[0092] [Embodiment 4] In the above-described embodiment, the convolution process is performed on a feature image basis. For example, in step S107 in the first embodiment, the feature image is transferred to the feature data storage unit 403. However, it is not necessary to perform data transfer and convolution process for each feature image. For example, the feature image may be transferred in blocks in multiple batches, or the convolution process may be performed on a block basis.

[0093] When processing is performed in units of blocks, the data transfer time corresponding to the access time to the holding unit 402 in step S107 is set to T m and the time for the convolution process in step S108 is T c If depthwise convolution is not performed, T m and T ccan be expressed by the following equations (4) and (5).

number

[0094] In addition, when performing depthwise convolution processing, IC and OC become the same value, and T m and T c can be expressed by the following equations (6) and (7).

number

[0095] In formulas (4) to (7), the meaning of each parameter is as follows. The size of the weighting coefficient of the filter used in the convolution process and the size of the pixel data of the feature image may vary depending on the hierarchical information shown in the network structure information. For this reason, the block size, filter size, etc. are expressed using bits instead of the number. IC: Number of input feature images OC: Number of output feature images IN×IM: Input block size (bits) ON×OM: Output block size (bits) X×Y: Filter size (bits) BW FM : Transmission bandwidth of characteristic images (bits / time) BW W : Weighting coefficient transmission bandwidth (bits / time) MAC PE : Parallelism of multiplication and accumulation processing of the convolution processing unit PE num : Number of arithmetic units for convolution processing

[0096] T c is proportional to the filter size. When pipeline processing is performed, the processing time for one cycle is T m and T c This is the maximum value of.

[0097] Fig. 13 shows pseudo code for block-based processing. The loop configuration is the same as the flowchart in Fig. 1, but a loop for block processing has been added. Note that when the number of output feature images is greater than the processing parallelism, first, the PE num Pieces (1st piece ~ PE num Then, by further block-wise processing, the PE num Pieces (PE num As an alternative method, after generating feature data for the first block of all the output feature images, feature data for the second and subsequent blocks of all the output feature images may be generated.

[0098] FIG. 14 shows an example of the configuration of the CNN processing unit 305 that can be used to perform processing on a block-by-block basis. The feature data storage unit 1401 can store an input feature image and an output feature image. In this configuration, the control unit 401 stores the processing result by the post-processing unit 407 in the feature data storage unit 403, and treats this as a feature image of the next layer. When performing processing of the first layer, the control unit 401 reads out some of the input feature images from the storage unit 402 and transfers them to the feature data storage unit 403. On the other hand, when performing processing of the second or subsequent layers, the data stored in the feature data storage unit 403 is used as the input feature image. The CNN processing unit 305 according to this example configuration is the same as the CNN processing unit 305 according to the first embodiment except for the feature data storage unit 1401, but the feature data and the results of the convolution processing are transferred on a block-by-block basis.

[0099] In such a configuration, when it is determined that depthwise convolution is not to be performed, an output feature image of one block is generated using input feature images of multiple blocks, whereas when depthwise convolution is to be performed, an output feature image of one block is generated using an input feature image of one block.

[0100] This block-based processing configuration can process feature data partially and is highly flexible, allowing input images of various sizes to be processed, and even in such cases, the depthwise convolution process can be performed with high efficiency.

[0101] [Other embodiments] In the above embodiment, an example has been described in which the type of convolution processing and the filter size are different for each layer. On the other hand, the filters or feature data used in one layer may be classified into a plurality of groups, and the type of convolution processing and the filter size may be different for each group. In such a case, the processing device according to the above embodiment can also switch processing for each group. That is, in the above embodiment, the feature data and the weighting coefficients of the filters used in the product-sum operation in the processing target layer may be classified into a plurality of groups. In this case, in the processing of the processing target layer, it is possible to select whether to perform the first processing or the second processing for each group to be processed.

[0102] In the above embodiment, an example of performing convolution processing in the depth direction using a filter having a size of 3×3 has been described. However, the processing device according to the above embodiment can perform convolution processing using filters having different sizes using hardware of the same configuration. Therefore, the size of the filter used is not limited to 3×3, and may be 5×5 or 7×7. The filter used may also be a filter containing zero coefficients for dilated convolution processing. That is, the convolution processing unit 406 can perform a product-sum operation using the weighting coefficient of the filter used in the dilated convolution processing. The convolution subunit 410 has a configuration of performing a product-sum operation on the supplied feature data and coefficient data one by one, and is capable of performing convolution processing using a filter of any size. The above embodiment can also handle a combination of convolution processing using a filter of any size and convolution processing in the depth direction.

[0103] FIG. 15 shows an example of the relationship between the network and the convolution process. In the process of layer 1501 performing the convolution process in the depth direction, the mode on the left side of the frame 420 is selected, and the convolution process in the depth direction is performed in parallel using a filter for the dilated convolution process. The filter used in layer 1501 is an expanded 3×3 filter, and has a size of 5×5. In layer 1502 performing the activation process, the process such as ReLU is performed. In the process of layer 1503 performing the convolution process in the point direction, the convolution process in the depth direction is not performed, so the mode on the right side of the frame 420 is selected, and the convolution process using a 1×1 filter is performed.

[0104] In the first embodiment, an example in which one input feature image is processed simultaneously has been described, but multiple input feature images may be processed simultaneously. For example, while some of the convolution subunits 410 are performing processing using a feature image (3,1), other convolution subunits 410 may be performing processing using a feature image (3,2). In the second embodiment, an example in which one output feature image is generated by parallel processing has been described, but multiple output feature images may be generated simultaneously. For example, while some of the convolution subunits 410 are generating a feature image (4,1), other convolution subunits 410 may be generating a feature image (4,2).

[0105] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0106] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0107] 401: control unit, 402: storage unit, 403: feature data storage unit, 404: coefficient storage unit, 405: readout unit, 406: convolution processing unit, 407: post-processing unit, 408: result storage unit, 410: convolution subunit, 411: feature data storage unit, 412: coefficient storage unit, 413: multiplier, 414: adder, 415: result storage unit, 416: calculation core

Claims

1. A processing device for performing calculations in a convolutional neural network having multiple layers, a control means for sequentially selecting each of the plurality of hierarchical layers as a processing target hierarchical layer; a data storage means for storing at least a portion of data of a plurality of channels in the hierarchical layer to be processed; a plurality of calculation means each performing a multiply-and-accumulate operation in parallel using data of one channel of the hierarchical layer to be processed and a coefficient corresponding to the hierarchical layer to be processed; a selection means for selecting, based on information specifying a process in the hierarchical layer to be processed, whether to perform a first process in which data of each of one channel in the hierarchical layer to be processed stored in the data holding means is input in parallel to a corresponding one of the plurality of calculation means and the plurality of calculation means performs parallel calculations, or a second process in which data of the plurality of channels in the hierarchical layer to be processed stored in the data holding means is read out sequentially and data of each of one channel in the hierarchical layer to be processed is input in parallel to each of the plurality of calculation means; In the first process, a value corresponding to data of one channel of a layer next to the layer to be processed is calculated by one of the plurality of calculation means, the data holding means is a memory, the calculation means comprises an arithmetic core including a multiplier and an adder; the processing device comprises a chip on which the memory and the arithmetic core are provided, The selection means is A processing device comprising: an address issuing unit provided on said chip for specifying an address in said memory of data to be input to said arithmetic core.

2. 2. The processing device according to claim 1, wherein the second processing comprises repeating inputting data of one of the plurality of channels in the processing target hierarchical layer in parallel from the data holding means to the plurality of calculation means.

3. 3. The processing device according to claim 2, wherein in the second processing, each of the plurality of calculation means uses data of each channel in the processing target hierarchy inputted in sequence to output a calculation result corresponding to one channel in a hierarchy next to the processing target hierarchy.

4. Further comprising a plurality of data storage means; in the first processing, each of the plurality of data storage means supplies data of a corresponding one channel in the processing target hierarchy to a corresponding one of the plurality of calculation means, each of the plurality of data storage means corresponding to a different one of the plurality of calculation means, In the second process, one of the plurality of data storage means supplies data of the one common channel in the processing target hierarchy to each of the plurality of calculation means.

4. The processing device according to claim 1, wherein the processing device is a processing apparatus.

5. A processing device for performing calculations in a convolutional neural network having multiple layers, a control means for sequentially selecting each of the plurality of hierarchical layers as a processing target hierarchical layer; a data storage means for storing at least a portion of data of a plurality of channels in the hierarchical layer to be processed; a plurality of calculation means each performing a multiply-and-accumulate operation in parallel using data of one channel of the hierarchical layer to be processed and a coefficient corresponding to the hierarchical layer to be processed; an accumulation means for accumulating the calculation results of each of the plurality of calculation means; a selection means for selecting, based on information specifying processing in the processing target hierarchical layer, whether to perform a first processing in which data of one channel in the processing target hierarchical layer stored in the data holding means is input in parallel to a corresponding one of the plurality of calculation means, the plurality of calculation means performs parallel calculations, and the calculation result by each of the plurality of calculation means is output as input data for a corresponding one channel in a next hierarchical layer of the processing target hierarchical layer, or a second processing in which an accumulated calculation result obtained by the accumulation means is output as input data for a channel in the next hierarchical layer; In the first process, a value corresponding to data of one channel of a layer next to the layer to be processed is calculated by one of the plurality of calculation means, the data holding means is a memory, the calculation means comprises an arithmetic core including a multiplier and an adder; the processing device comprises a chip on which the memory and the arithmetic core are provided, The selection means is A processing device comprising: an address issuing unit provided on said chip for specifying an address in said memory of data to be input to said arithmetic core.

6. 6. The processing device according to claim 5, wherein in the second processing, data of a plurality of channels in the processing target hierarchy stored in the data holding means is input in parallel to the plurality of calculation means so that data of one channel is input to a corresponding one of the calculation means.

7. a plurality of result storage means, each corresponding to one of the plurality of computing means; In the first process, each of the plurality of calculation means outputs the calculation result to the corresponding result storage means; In the second process, each of the plurality of calculation means outputs the calculation result to the accumulation means, and the accumulation means outputs the accumulation result to one of the plurality of result storage means.

7. The processing device according to claim 5 or 6.

8. 8. The processing device according to claim 1, wherein in the first processing, data of a plurality of channels in the processing target hierarchy stored in the data holding means is input in parallel to the plurality of calculation means so that data of one channel is input to a corresponding one of the calculation means.

9. 9. The processing device according to claim 1, wherein in the first processing, each of the plurality of calculation means uses data of one channel in the processing target hierarchy to output a calculation result corresponding to one channel in a hierarchy next to the processing target hierarchy.

10. a coefficient holding means for holding at least a part of coefficients used in the multiplication and accumulation operation in the processing target layer; a supply control means for controlling the supply of data from the data holding means and the coefficient holding means to the plurality of calculation means, Each of the plurality of calculation means calculates a product of one of the input data and one of the coefficients, and accumulates the calculated products to perform the product-sum calculation.

10. The processing device according to claim 1 ,

11. the data and the coefficients used in the multiply-and-accumulate operation in the processing target layer are classified into a plurality of groups; 11. The processing device according to claim 1, wherein the selection means selects whether to perform the first processing or the second processing based on the group of the data and the coefficients used in a product-sum operation.

12. 12. The processing device according to claim 1, wherein the coefficients are weighting coefficients of a filter used in convolution processing, and the size of the filter can be set for each of the layers.

13. 12. The processing device according to claim 1, wherein the coefficients are weighting coefficients of a filter used in dilation convolution processing.

14. The selection means is A multiplexer is provided on the chip to select an input to the arithmetic core from a plurality of sources, or to select one output from a plurality of the arithmetic cores.

14. The processing device according to claim 1,

15. The operation according to the convolutional neural network is performed on the processing target image, The data in the processing target layer is a feature image obtained in the process of calculation according to the convolutional neural network.

15. The processing device according to claim 1 ,

16. A control means for controlling the plurality of calculation means so as to perform a calculation according to the convolutional neural network on the processing target image; an image processing means for performing image processing on the processing target image based on a processing result obtained by performing an operation on the processing target image according to the convolutional neural network; The processing device of claim 15 further comprising:

Images