A computing device, information processing method, and program

Abstract

To provide an arithmetic unit capable of improving convolution operation processing on a data array with zero and capable of reducing consumed power and processing time required for the processing.SOLUTION: An arithmetic unit holds data array in a processing target block and a filter weighting factor used for filter processing. The arithmetic unit determines whether or not the data that is set corresponding to the processing target block and within a reference range in the processing target block are zero-valued, generates the convolution operation result between the data and the weighting factor for each of multiple positions in the processing target block. and controls whether or not at least a part of product-sum operation between the data and the weighting factor is performed according to the determination result when generating the convolution operation result.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an arithmetic unit, an information processing method, and a program, and particularly to arithmetic processing using a convolutional neural network.

Background Art

[0002] Convolutional Neural Networks (CNNs) are used in deep learning. In each layer of a convolutional neural network, convolution operations and activation processes are often performed. When the result of a convolution operation is a negative value, the result of the activation process becomes zero, so the feature maps obtained in each layer contain many zero values. Patent Document 1 proposes reducing power consumption by skipping the process when the ratio of zero values in a partial region is large when performing a convolution operation between a filter of size n×n and a partial region of size n×n. Also, Patent Document 2 discloses omitting the product operation when the weight coefficient or data value of a filter is zero in a convolution operation. Patent Document 3 also discloses omitting the product operation when the pixel data of a feature image is zero in a convolution operation.

Prior Art Documents

Patent Documents

[0005] The present invention aims to improve the efficiency of convolution operations on data arrays containing zero values, thereby reducing the power consumption or processing time required for the operation. [Means for solving the problem]

[0006] To achieve the object of the present invention, the arithmetic device according to one embodiment of the present invention has the following configuration. That is, an arithmetic device that performs filter processing on a data array in a processing block of a predetermined size, comprising: data holding means for holding the data array in the processing block; and coefficient holding means for holding the weight coefficients of the filter used in the filter processing. before Within the block to be processed , smaller than the block to be processed Data within the reference range all A determination means for determining whether or not the value is zero, and a processing means for generating the result of a convolution operation between the data and weight coefficients at each of the multiple positions within the processing target block, Equipped with, The determination means If it is determined that all data within the aforementioned reference range are zero values, Recording processing means teeth, The aforementioned convolution operation In the past The data and the above Filter At least part of the sum-of-products operation with weight coefficients The rows or columns of the filter from which the sum-of-products operation is omitted and the rows or columns of the filter from which the sum-of-products operation is not omitted are determined based on the reference range. It is characterized by the following. [Effects of the Invention]

[0007] The present invention can improve the efficiency of convolution operations on data arrays containing zero values, thereby reducing the power consumption or processing time required for the operation. [Brief explanation of the drawing]

[0008] [Figure 1] A flowchart of the process according to one embodiment. [Figure 2] A diagram showing an example of the structure of a neural network according to an embodiment. [Figure 3] A block diagram showing an example of the configuration of a processing device according to an embodiment. [Figure 4] A block diagram showing an example of the configuration of a CNN processing unit according to an embodiment. [Figure 5] A diagram showing an example of convolution processing and a reference range. [Figure 6] A diagram showing an example of the processing of a multiply-accumulate operation according to an embodiment. [Figure 7] A diagram showing an example of the processing of a multiply-accumulate operation according to an embodiment. [Figure 8] A diagram showing an example of convolution processing and a reference range. [Figure 9] A diagram showing an example of convolution processing and a reference range. [Figure 10] A flowchart of processing according to an embodiment. [Figure 11] A diagram showing an example of a reference range. [Figure 12] A diagram showing the relationship between a neural network and a convolution operation. [Figure 13] A diagram showing an example of a reference range.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] A computing device according to one embodiment of the present invention can perform filtering on a data array within a processing block of a predetermined size. Figure 3 is a block diagram showing an example configuration of a convolutional neural network processing device, which is a computing device according to one embodiment of the present invention. Such a processing device can, for example, perform processing on an image using a neural network. Specific examples of processing that such a processing device can perform include object detection processing in an image, object identification processing in an image, region segmentation processing for an image, and image resolution enhancement processing. Such processing can be performed, for example, using a feature map obtained by inputting an image into a neural network.

[0011] The input unit 301 is a device that receives instructions or data from the user. The input unit 301 may be, for example, a keyboard, a pointing device, or a button.

[0012] The data storage unit 302 can store data such as image data. The data storage unit 302 may be, for example, a hard disk, flexible disk, CD-ROM, CD-R, DVD, memory card, CF card, SmartMedia, SD card, Memory Stick, xD-Picture Card, or USB memory. The data storage unit 302 may store programs or other data. In addition, a part of the RAM 308, which will be described later, may be used as the data storage unit 302.

[0013] The communication unit 303 is an interface (I / F) for communication between devices. The processing unit 300 can exchange data with other devices via the communication unit 303. The processing unit 300 may also use the storage device connected via the communication unit 303 as a virtual data storage unit, i.e., as the data storage unit 302.

[0014] The display unit 304 is a device that displays information to the user or others. The display unit 304 can, for example, display an image before or after image processing, or display other images such as a GUI. The display unit 304 may be, for example, a CRT or a liquid crystal display. The display unit 304 may be a device located outside the processing unit 300 and connected by a cable or the like. The input unit 301 and the display unit 304 may be the same device; for example, the input unit 301 and the display unit 304 may be a touchscreen device. In this case, input on the touchscreen corresponds to input to the input unit 301.

[0015] The CNN processing unit 305 can perform neural network processing (S101-S117) on an image according to the flowchart in Figure 1, which will be described later. The processing performed by the CNN processing unit 305 includes filtering using convolution operations. The CNN processing unit 305 may also perform neural network processing on the results of image processing stored in RAM 308 by the image processing unit 309. The CNN processing unit 305 can output the processing results to the data storage unit 302 (or RAM 308). These processing results can be used for various processes such as image processing or image recognition processing by the CPU 306. However, such a CNN processing unit 305 may be used for purposes other than image processing; that is, configurations other than the CNN processing unit 305 shown in Figure 3 are not essential to the present invention. The CNN processing unit 305 can perform filtering on still images or moving images. For example, the CNN processing unit 305 can perform filtering on each of the multiple frames contained in a moving image. In this case, the CPU 306 can perform image processing or image recognition on the moving image.

[0016] The CPU 306 controls the operation of the entire processing unit 300. The CPU 306 can also perform various processes, such as image processing or image recognition, based on the processing results generated by the CNN processing unit 305 and stored in the data storage unit 302 or RAM 308. The CPU 306 can store these processing results in RAM 308.

[0017] ROM 307 and RAM 308 provide the CPU 306 with the programs, data, and work areas necessary for processing by the CPU 306. The programs necessary for processing by the CPU 306 may be stored in the data storage unit 302 or ROM 307, or they may be read from the data storage unit 302 or ROM 307 into RAM 308. The data processing device 300 may also receive programs via the communication unit 303. In this case, the programs may be recorded in the data storage unit 302 first and then read into RAM 308, or they may be read directly from the communication unit 303 into RAM 308. In either case, the CPU 306 can execute the programs read into RAM 308.

[0018] The image processing unit 309 can perform image processing on image data. For example, the image processing unit 309 can read image data written to the data storage unit 302 according to instructions from the CPU 306, adjust the range of pixel values, and write the processing result to the RAM 308.

[0019] The processing device 300 shown in Figure 3 has the above-mentioned parts inside. These parts are connected to each other so that they can send and receive data. However, each part, including, for example, the input unit 301, the data storage unit 302, and the display unit 304, may be connected to each other by a communication path according to a known communication method. In other words, a data processing device according to one embodiment may be composed of multiple physically separate devices.

[0020] Furthermore, although the processing unit 300 shown in Figure 3 has one CPU 306, it may have multiple CPUs. In addition, at least some of the functions of each part of the processing unit 300 (for example, the CNN processing unit 305 and the image processing unit 309) may be realized by the CPU 306 operating according to a program.

[0021] The processing unit 300 may have various components not shown in Figure 3, but their description will be omitted.

[0022] (Example of a neural network structure) As described above, the CNN processing unit 305 can perform filtering on a data array using filters. Furthermore, the CNN processing unit 305 can perform processing according to a neural network including multiple layers, and can perform such filtering at at least one layer. Filtering includes convolution operations, and convolution operations include multiple multiply-accumulate operations. In the following, a single multiply-accumulate operation refers to a set of operations: the product of one data point and one filter coefficient, and the accumulation of these products. Also, a single convolution operation refers to an operation that obtains one output data by convolving a filter onto a specific data array (for example, a local region of a feature image), and includes multiple multiply-accumulate operations. The following describes the case where filtering is performed on a feature image. A feature image has pixel data for each pixel as a data array.

[0023] The following describes an example of a neural network used by the CNN processing unit 305. A CNN, a type of neural network, has a structure in which multiple layers are connected hierarchically. Each layer may contain multiple feature images. In the following, the feature image obtained by performing the corresponding processing on the feature image of the previous layer is called the feature image of the next layer. In the following, we will describe the case where the feature image is two-dimensional, but the feature image may be one-dimensional, or it may be a higher-order feature image of three dimensions or more.

[0024] For example, the feature image of the next level may be calculated using a filtering process applied to the feature image of the previous level. This filtering process can use a filter composed of filter coefficients corresponding to the previous level. Each of the multiple feature images of the next level can be generated by filtering using the corresponding filter. Alternatively, multiple feature images of the previous level may be used to calculate a single feature image of the next level. For example, each of the multiple feature images of the previous level can be filtered using the corresponding filter, and a single feature image of the next level can be obtained based on the multiple processing results obtained.

[0025] For example, the feature image after filtering (O i,j (n)) is the feature image of the previous layer (I i,j (m) and filter coefficients (W 0,0 (m,n)~W X-1,Y-1 The filter coefficients can be calculated using (m,n) and according to equation (1). Here, i and j represent the coordinates of the feature images. Also, x and y represent the coordinates of the filter. n is the number of the next level feature image. Also, m is the number of the previous level feature image, and there are IC feature images in the previous level. The filter coefficients are different for each previous level feature image and for each next level feature image, and there are X × Y of them for one combination of feature images.

number

[0026] As described above, the number of multiply-accumulate operations performed in the convolution operation to calculate the pixel data of one feature image in the next layer is M × X × Y. Thus, the filter has multiple filter coefficients, and the pixel value of each pixel in the filtered feature image is obtained by a convolution operation of the pixel values ​​of the group of pixels surrounding the corresponding pixel in the previous layer's feature image and the filter coefficients of the filter.

[0027] Feature image O obtained through this filtering process i,jBy further processing such as activation or pooling on (n), the next-level feature image can be calculated. The activation process can be performed according to equation (2). In equation (2), f(·) is a function of ReLU (Rectified Linear Unit), and the variable x is the input data.

number

[0028] When performing activation according to equation (2), if the convolution result is negative, the activation result will be zero. In this case, the pixel data at the corresponding position in the feature image of the previous layer will be zero, and zero values ​​will be input to the sum-of-products operation in the next layer. If the pixel data of the feature image is zero, this data does not contribute to the result of the convolution, so the result will not be affected even if the sum-of-products operation using this data is omitted.

[0029] Figure 2 shows a concrete example of a neural network structure. In the neural network shown in Figure 2, there are 4 layers, and each layer contains 4 feature images. The feature images in each layer are obtained based on the filtering results obtained by applying a filter defined for each feature image to the pixel data of the feature image. Here, the filter coefficients are obtained in advance according to known learning techniques. Furthermore, the filtering process to which the filter is applied includes convolution operations, that is, multiple multiplications and cumulative additions. In Figure 2, the arrows indicate convolution operations.

[0030] In Layer 1, based on equations (1) and (2), multiple feature images 202 for Layer 2 are generated by filtering using multiple feature images 201 and filter coefficients. Similarly, in Layer 2, multiple feature images 203 for Layer 3 are generated by filtering using multiple feature images 202 and filter coefficients. Similarly, in Layer 3, multiple feature images 204 for Layer 4 are generated by filtering using multiple feature images 203 and filter coefficients. Thus, the filtering is performed in hierarchical order. As shown in Figure 12, multiple pixel data are extracted from the same position in the four feature images 1201 in Layer 1, and filtering and activation processing are performed. The processing result becomes some of the pixel data of the feature image 1202 in Layer 2.

[0031] Figure 2 further shows the type of filtering and filter size at each layer. Layers 1, 2, and 3 use filters of sizes 3x3, 5x5, and 7x7, respectively. Thus, the size of the filters used for filtering may differ at each layer.

[0032] Network structure information, which indicates the structure of the convolutional neural network, may be stored in RAM308. This network structure information may include, for example, the number of layers, the number of feature images in each layer, the type of filtering performed in each layer, and the types of activation and pooling processes performed in each layer.

[0033] Figure 4 shows an example of the functional configuration of the CNN processing unit 305. In this embodiment, the CNN processing unit 305 can perform filtering on a data array within a processing block of a predetermined size. The CNN processing unit 305 includes a coefficient holding unit 403, a feature data holding unit 405, a data selection unit 408, a zero determination unit 409, and a convolution processing unit 407. Furthermore, as will be described later, the CNN processing unit 305 may also include a control unit 401, a data holding unit 402, a readout unit 403, a reference range setting unit 406, an activation processing unit 410, and a result holding unit 411.

[0034] The data storage unit 402 stores some of the data stored in the data storage unit 302. The coefficient storage unit 403 stores the weight coefficients (filter coefficients) of the filter used for filtering. For example, the coefficient storage unit 405 stores the filter coefficient W obtained from the data storage unit 402. x,y (m,n) can be stored. The feature data storage unit 405 stores the data array within the block to be processed. For example, the feature data storage unit 405 can store some pixel data of the feature image I(m). These filter coefficients are part of the weight coefficients of the convolutional neural network.

[0035] The reference range setting unit 406 sets a reference range corresponding to the block to be processed. The reference range may be predetermined according to the size of the block to be processed and the size of the filter. The reference range setting unit 406 can set one or more reference ranges, and at least one reference range is smaller than the block to be processed, as shown by reference ranges 504 and 505 in Figures 5(B) and 5(C). Details of the reference range will be described later.

[0036] The convolution processing unit 407 generates a convolution operation result between the data and the filter weight coefficients (filter coefficients) at each of multiple positions within the processing target block. In this embodiment, the convolution processing unit 407 can perform a convolution operation by convolving a filter onto the processing target block in the input feature image, and can generate a filter processing result for the input feature image. For example, the convolution processing unit 407 can obtain a convolution operation result using the filter coefficients and pixel data according to equation (1). In this embodiment, the convolution operation unit 407 calculates the convolution operation result using the pixel data held in the feature data holding unit 405 and the filter coefficients held in the coefficient holding unit 403, according to a control signal from the control unit 401.

[0037] The convolution processing unit 407 has multiple convolution processing units 412. Each convolution processing unit 412 can perform a sum-of-accumulate operation by accumulating the product of pixel data and filter coefficients. Furthermore, multiple convolution processing units 412 can perform sum-of-accumulate operations in parallel for multiple positions within the processing block. The convolution processing unit 412 has an arithmetic core 418, a feature data storage unit 413, a coefficient storage unit 414, and a result storage unit 417. The multiplier 415 and adder 416 used for the sum-of-accumulate operation are included in the arithmetic core 418. However, in one embodiment, a single convolution processing unit 412 may be used to sequentially perform sum-of-accumulate operations for each of the multiple positions within the processing block. Details of the processing by the convolution processing unit 407 will be described later.

[0038] The data selection unit 408 transfers pixel data to be used for processing by the convolution processing unit 407 to the convolution processing unit 407. The data selection unit 408 also controls whether to perform at least a portion of the sum-of-products operation at multiple locations within the processing target block, according to the determination result by the zero determination unit 409. The zero determination unit 409 determines whether the data within the reference range within the processing target block, which is set by the reference range setting unit 406 corresponding to the processing target block, is zero. In accordance with the control of the data selection unit 408 and the zero determination unit 409, the convolution processing unit 407 can omit a portion of the sum-of-products operation for generating the filter processing result for the processing target block. In other words, the convolution processing unit 407 can generate the filter processing result for the processing target block by partially performing the convolution operation. In this embodiment, if the data selection unit 408 determines that the data within the reference range is zero, the convolution processing unit 407 can omit at least a portion of the sum-of-products operation between the data and the weight coefficient at each of the multiple locations within the processing target block. Details of the processing performed by the data selection unit 408 and the zero determination unit 409 will be described later.

[0039] In this specification, a zero value is zero itself in the following embodiments, but it may also be a number with an absolute value less than or equal to a predetermined value (e.g., 1 or less) that has little influence on the convolution operation result. Furthermore, in the following embodiments, data within the reference range being a zero value means that all data within the reference range is zero. On the other hand, as in Patent Document 1, data within the reference range being a zero value may mean that the number of zeros is above a predetermined percentage (e.g., 85% or more) and has little influence on the convolution operation result.

[0040] The activation processing unit 410 performs further activation processing on the results of the filtering process. The activation processing unit 410 can calculate the activation processing result according to, for example, equation (2). However, the type of activation processing is not particularly limited, and the activation processing unit 410 may perform activation processing using other nonlinear functions or quantization functions. In addition, the activation processing unit 410 may adjust the size of the output feature image by performing pooling processing based on the results of the activation processing according to the network structure information. Depending on the case, both or one of the activation processing and pooling processing may be omitted.

[0041] The result holding unit 411 holds the processing results obtained by the activation processing unit 410. The reading unit 403 transfers addresses for accessing the feature data holding unit 405 and the coefficient holding unit 403 to the zero determination unit 409 and the convolution processing unit 407. The zero determination unit 409 and the convolution processing unit 407 can read pixel data from the feature data holding unit 405 or read filter coefficients from the coefficient holding unit 403 according to these addresses.

[0042] These processes do not necessarily need to be performed by a dedicated processing unit. For example, the activation process and the pooling process may be performed by the CPU 306. Also, one or more of the processes among the reference range setting unit 406, the zero determination unit 408, and the data selection unit 408 may be performed by the CPU 306.

[0043] Figure 1 is an example of a flowchart of the information processing method performed by the CNN processing unit 305. The control processing shown in S101 to S117 can be performed by the control unit 401 (for example, a CPU or sequencer in the control unit 401). The following describes each step of the processing using the convolutional neural network in this embodiment with reference to Figure 1.

[0044] In step S101, the control unit 401 reads the input feature image, filter coefficients used for filtering, and network structure information from the RAM 308 and stores them in the data storage unit 402. In the example in Figure 2, the input feature image may be an image of each of the RGB+D (depth) planes, or it may be a feature image obtained by filtering the image.

[0045] In S102, a loop for each hierarchical level begins. In S102, the control unit 401 can select the first hierarchical level. In the following description, the hierarchical level selected in S102 will be referred to as the previous hierarchical level, and the hierarchical level following the previous hierarchical level will be referred to as the next hierarchical level. By performing the processes S103 to S116 in order for each hierarchical level, the results of processing using a convolutional neural network can be obtained.

[0046] In S103, a loop for each block begins. In this embodiment, each output feature image of the next level is divided into multiple feature image blocks. Furthermore, the pixel data of a feature image block of a single output feature image is calculated using the pixel data of the corresponding feature image block of the input feature image of the previous level. For example, in the example in Figure 5(A), the pixel data of feature image block 512 of output feature image 502 is obtained by filtering each feature image block 503 of multiple input feature images 501. In this example, the feature image blocks of the output feature image are adjacent but do not overlap, while the feature image blocks of the input feature image are aligned so that they overlap each other. Each feature image block corresponds to a block to be processed.

[0047] In S103, one feature image block of the output feature image (for example, feature image block 512) is selected. Also selected is the corresponding feature image block of the input feature image (for example, feature image block 503) used to calculate the pixel data for this feature image block of the output feature image. In S104-S115, one feature image block common to multiple output feature images is selected, and the pixel data for each output feature image in the selected feature image block is calculated. During this process, the pixel data for each input feature image in the selected feature image block is referenced. By performing the processes in S104-S115 sequentially for each feature image block, the output feature images of the next hierarchical level can be obtained.

[0048] In S104, a loop for each output feature image of the next tier begins. In S105-S114, pixel data for one output feature image in the feature image block selected in S103 is calculated. In this way, pixel data is calculated sequentially for each of the multiple output feature images of the next tier.

[0049] In S105, the control unit 401 initializes the convolution operation result held in the result storage unit 417 of the convolution processing unit 407. For example, the control unit 401 can set the convolution operation result to zero.

[0050] In S106, a loop for each input feature image from the previous layer begins. In S107-S111, filtering is performed on the feature image block selected in S103 from a single input feature image. By performing the processes in S107-S111 in order, filtering is performed on each input feature image. The loop from S107-S111 can be performed for each input feature image referenced to obtain the output feature image selected in S104.

[0051] In S107, the reference range setting unit 406 sets the reference range for zero values ​​of the pixel data. The method for setting the reference range by the reference range setting unit 406 will be described later.

[0052] In S108, the control unit 401 reads a portion of the input feature image from the data storage unit 402 and transfers it to the feature data storage unit 405. The control unit 401 can transfer pixel data from the feature image block selected in S103, from the input feature image selected in S106, to the feature data storage unit 405. The control unit 401 also reads a portion of the filter coefficients from the data storage unit 402 and transfers them to the coefficient storage unit 403. The control unit 401 can transfer the filter coefficients for the filtering process performed on the input feature image selected in S106 to obtain the output feature image selected in S104 to the coefficient storage unit 403. In this way, in S108, the control unit 401 can read pixel data and filter coefficients referenced in the convolution operation performed in S109 to S111 from the data storage unit 402.

[0053] In S109, the zero determination unit 409 determines whether all pixel data within the reference range of the input feature image is zero. If all pixel data within the reference range is zero, the process proceeds to S110. Otherwise, the process proceeds to S111.

[0054] In S110, the convolution processing unit 407 generates a convolution result for the feature image block selected in S103 from the input feature image selected in S106. This convolution result is composed of the sum-of-products calculation result of pixel data and filter coefficients for each of the multiple locations within the feature image block. In S110, the convolution processing unit 407 obtains the convolution result for the location within the feature image block in the first process. Specifically, in S110, the convolution calculation unit 407 can omit part of the sum-of-products calculation of pixel data and filter coefficients for each of the multiple locations within the feature image block. The specific processing in S110 will be described later.

[0055] In S111, the convolution processing unit 407 generates a convolution result for the feature image block selected in S103 from the input feature image selected in S106. In S111, the convolution processing unit 407 obtains the convolution result in a second process different from S110. For example, the convolution processing unit 407 can perform a sum-of-products operation between pixel data and filter coefficients for each of multiple positions within the feature image block without omitting any steps.

[0056] In S112, the control unit 401 determines the end of the loop for the input feature images. If processing for all input feature images is complete, the process proceeds to S113. Otherwise, the process returns to S107, and processing for the next input feature image begins.

[0057] When proceeding from S112 to S113, the result storage unit 417 accumulates the filtering results for the input feature image selected in S106, pixel by pixel. For example, the result storage unit 417 stores pixel data O according to equation (1) for each pixel of the feature image block selected in S103 from the output feature image selected in S104. i,j (n) may be stored.

[0058] In S113, the activation processing unit 410 performs activation processing based on the filter processing results held in the result storage unit 417, in accordance with the control signal from the control unit 401.

[0059] In S114, the control unit 401 stores the processing result from the activation processing unit 410 in the data storage unit 402. The processing result stored in the data storage unit 402 corresponds to the pixel data of the feature image block selected in S103 from the output feature image selected in S104. The pixel data of the output feature image stored in this way is used as the pixel data of the input feature image when processing at the next level.

[0060] In S115, the control unit 401 determines the end of the loop for the output feature image. If processing for all output feature images is complete, the process proceeds to S116. Otherwise, the process returns to S105, and processing for the next output feature image begins.

[0061] In S116, the control unit 401 determines the end of the loop for the feature image block. If processing for all feature image blocks is complete, the process proceeds to S117. Otherwise, the process returns to S104, and processing for the next feature image block begins.

[0062] In S117, the control unit 401 determines the end of the loop for each hierarchy. If processing for all hierarchical levels is complete, the process shown in Figure 1 ends. Otherwise, the process returns to S103 and processing for the next hierarchy begins.

[0063] (Example of processing according to this embodiment) According to this embodiment, the computational complexity in filtering can be reduced, thereby improving the processing efficiency of filtering. The improvement in processing efficiency according to this embodiment will be explained with reference to Figures 5 to 7. The processing device according to this embodiment is capable of processing multiple pixel data contained in multiple feature images in parallel, but below we will explain the case where processing is performed using a four-layer convolutional neural network as shown in Figure 2. In Figure 2, solid lines indicate the processing target, and solid lines indicate the convolution operations related to the processing target.

[0064] Figure 5(A) shows an example of a process that filters the feature image of Layer 1 (input feature image) to generate a feature image of Layer 2 (output feature image). This filtering process uses a 3x3 size filter, and therefore the kernel size is 3x3. The input feature image 501 is divided into multiple feature image blocks 503, which are processed sequentially. Each feature image block 503 of the input feature image 501 contains 5x5 pixel data. Further filtering (and activation processing, etc.) using the feature image blocks 503 yields the feature image block 512 of the output feature image 502. Each feature image block 512 contains 3x3 pixel data. Figures 5(B) and 5(C) also show, as examples of zero-value reference ranges, a 5x4 region enclosed by a dotted line, reference range 504, and a 5x3 region, reference range 505.

[0065] In the processing example shown in Figures 6(A) to (C), block 601 is a feature image block of the input feature image, with a size of 5x5 and 25 pixel data points. Each pixel is assigned a different number from 1 to 25. In this processing example, a 3x3 filter with a vertical stride of 1 and a horizontal stride of 1 is applied to block 601, using nine 3x3 kernels 602 to 610 (kernels 1 to 9). Kernels 602 to 610 each overlap with adjacent kernels and share some pixel data. In Figures 6(A) and 6(C), white pixels indicate that the pixel data is zero, while hatched pixels indicate that the pixel data is not zero.

[0066] In this embodiment, the nine kernels 602-610 are processed in parallel by the nine convolution units 412. The pixel data within each kernel 602-610 is processed sequentially. The same filter is applied to kernels 602-610. Therefore, the convolution units 412 can perform multiply-accumulate operations in parallel between the data input to each (for example, the upper-left pixel data of kernels 602-610) and a common weight coefficient (for example, the upper-left weight coefficient of the filter).

[0067] As shown in the example in Figures 6(A) and 6(C), if all pixel data within the reference range 504 is zero, then the pixel data in the lower two lines of kernels 602 to 604 and the three lines of kernels 605 to 610 will all be zero. In other words, for kernels 602 to 610, only the upper line contains non-zero values. Also, for kernels 602 to 610, the lower two lines have zero values, and multiply-accumulate operations using this pixel data do not affect the filtering result. Therefore, in this embodiment, the nine convolution units 412 process only the pixel data of the upper line of the corresponding kernel, as indicated by arrow 611 in Figure 6(B). For example, instead of multiplying each of pixels 1 to 9 by the corresponding filter coefficient and accumulating the resulting product, the convolution unit 412 multiplies only each of pixels 1 to 3 by the corresponding filter coefficient and accumulates the resulting product. In this way, the convolution unit 412 can obtain the convolution result for all of the multiple positions within the block to be processed using a common first process. With this configuration, each convolution unit 412 can perform three multiply-accumulate operations while omitting six multiply-accumulate operations.

[0068] In this embodiment, processing using the pixel data of the upper line is also performed for kernels 605 to 610, but since the calculation result is zero, this processing result does not affect the filtering result. However, processing using the pixel data of the upper line for kernels 605 to 610 may be omitted. In this case, the convolution processing unit 412 that processes kernels 605 to 610 can be controlled so that it does not perform calculation processing on block 601.

[0069] The processing in S110 can be performed as described above. That is, since all pixel data in the reference region is determined to be zero in S109, the convolution operation is performed only partially. In other words, each convolution unit 412, which operates in parallel, performs a sum-of-products operation on the three pixel data and three filter coefficients of each kernel, but does not perform a sum-of-products operation on the remaining six pixel data and six filter coefficients of each kernel. As described above, since the remaining six pixel data of each kernel are zero, not performing calculations using this pixel data does not affect the filtering result. In S110, the convolution result is obtained by performing sum-of-products operations fewer times (3 times in this example) than the number of filter coefficients (9 in this example). Furthermore, the convolution unit 412 performs sum-of-products operations on sequential data and filter coefficients using only a portion of the multiple filter coefficients. Furthermore, the convolution unit 412 performs sum-of-products operations on sequential data and weight coefficients using each of the multiple weight coefficients.

[0070] In this process, the coefficient data used in the sum-of-products operation is supplied from the filter coefficient holding unit 403 to the convolution processing unit 407. The pixel data used in the sum-of-products operation is supplied from the zero determination unit 409 to the convolution processing unit 407 via the data selection unit 408. The data selection unit 408 can control the supply of pixel data from the filter coefficient holding unit 403 to the convolution processing unit 407 according to the determination result of the zero determination unit 409. That is, the data selection unit 408 can control the convolution processing unit 407 to perform sum-of-products operations using specific pixel data and to omit sum-of-products operations using specific pixel data. For example, the data selection unit 408 may control the sum-of-products operation by the convolution processing unit 407 by supplying only the pixel data used in the sum-of-products operation to the convolution processing unit 407. Furthermore, the transfer of filter coefficients may be controlled so that only the filter coefficients used in the sum-of-products operation are transferred from the filter coefficient holding unit 403 to the convolution processing unit 407 according to the determination result of the zero determination unit 409.

[0071] Furthermore, the size of the reference range may be 5 × 3, as shown in the reference range 505 in Figure 5(C). If all pixel data within the reference range 505 is zero, then the pixel data in the bottom line of kernels 602 to 604, the bottom two lines of kernels 605 to 607, and the three lines of kernels 608 to 610 will all be zero. Therefore, in this embodiment, the nine convolution processing units 412 can process only the pixel data of the top two lines of the corresponding kernels. In this embodiment, processing using the pixel data of the central line is also performed for kernels 605 to 607, and processing using the pixel data of the top two lines is also performed for kernels 608 to 610. However, since the calculation results using this pixel data will be zero, these processing results will not affect the filtering result.

[0072] In this way, various reference ranges of different sizes can be set. On the other hand, from the explanation above, it should be clear that the larger the reference range, the greater the effect of reducing computational complexity.

[0073] On the other hand, as shown in the example in Figure 7(A), if it is not determined that all pixel data within the reference range 504 are zero values, the nine convolution processing units 412 will perform the operation indicated by the arrow in Figure 7(B). 7 As shown in 11, the corresponding kernel Pixel data of all lines The processing is performed. That is, the calculation processing using all pixel data is carried out to the end for kernels 702 to 710 shown in Figure 7(C). The processing in S111 can be performed in this way. In other words, the convolution processing unit 412 can obtain the convolution result for all of the multiple positions in the feature image block using a common second process, which is different from the first process that omits some of the multiply-accumulate operations. Furthermore, the convolution result is obtained by multiply-accumulate operations a different number of times in the first process in S110 and the second process in S111. In S111, the convolution result is obtained by multiply-accumulate operations a number of times equal to the number of filter coefficients (9 in this example) (9 times in this example).

[0074] As described above, according to this embodiment, it is determined whether all data within the reference range are zero values, and a portion of the sum-of-products operation required to obtain the convolution result is omitted according to the determination result. Therefore, the amount of computation can be reduced, the efficiency of the convolution operation is improved, and the power consumption and processing time required for processing can be reduced. In this embodiment, it is determined whether the data is zero values ​​for a reference range larger than the filter size. In other words, each of the multiple convolution operations can be controlled based on the determination result for the reference range. Therefore, it is easy to reduce the amount of computation.

[0075] In particular, in this embodiment, a reference range having a different size from the block size of the input feature image was used. Therefore, even if not all pixel data within a block is zero, the computational complexity can be reduced. Furthermore, according to the configuration of this embodiment, even if not all data in the kernel is zero, the computational complexity can be reduced and processing can be sped up by omitting part of the multiply-accumulate operation as shown by arrow 611. Such a configuration is particularly effective when multiple convolutional processing units operate in parallel and process different kernels, as shown in Figure 4. That is, when each convolutional unit decides to omit multiplication of pixel data with zero values, or to omit the convolution operation for kernels with many zero values, the processing time of each convolutional processing unit will vary. Therefore, additional circuitry may be required for multiple convolutional processing units to operate synchronously. On the other hand, according to the configuration of this embodiment, as explained with reference to Figures 6(A) to (C), it is possible to reduce the computational complexity while operating multiple convolutional processing units synchronously.

[0076] Furthermore, according to the method of Patent Document 1, the convolution operation is skipped even if a small number of non-zero values ​​exist in the kernel, which may reduce the accuracy of the calculation. On the other hand, as in the embodiment described above, if a configuration is used in which it is determined whether all data within the reference range are zero values ​​and a part of the sum-of-accumulate operation to obtain the convolution result is omitted according to the determination result, the accuracy of the filtering process can be maintained.

[0077] [Differentiation] In the embodiment described above, it was determined whether the pixel data within one reference range was zero, but two or more reference ranges may be used. For example, the zero determination unit 409 may determine whether the data within each of the multiple reference ranges is zero. If the zero determination unit 409 determines that the data within a particular reference range is zero, the convolution processing unit 407 can obtain the convolution result by performing a sum-of-products operation on the position within the feature image block a number of times corresponding to the particular reference range.

[0078] Furthermore, in the above-described embodiment, when the pixel data within the reference range is zero, the convolution operation was partially performed when obtaining the convolution result for the feature image block of the input feature image. On the other hand, the convolution operation may be omitted when obtaining the convolution result for the feature image block of the input feature image. Such cases will be described below.

[0079] In this case, instead of processing S109 in Figure 1, processing S1001 to S1002 can be performed as shown in Figure 10. Below, we will describe the case in which two reference ranges are used as shown in Figures 11(A) and 11(B). Similar to the reference ranges in the embodiment described above, the first reference range 1121 has a size smaller than the feature image block 1101 of the input feature image. The second reference range 1122 has the same size as the feature image block 1101 of the input feature image. The first reference range 1121 is included in the second reference range 1122. In this example as well, filtering processing using a 3x3 size filter is performed, similar to Figures 6(A) to 6(C).

[0080] In S1001, which follows S108, the zero determination unit 409 determines whether all pixel data within the first reference range of the input feature image is zero. If all pixel data within the first reference range is zero, the process proceeds to S1002. Otherwise, the process proceeds to S111, where the convolution operation is performed.

[0081] In S1002, the zero determination unit 409 determines whether all pixel data within the second reference range of the input feature image is zero. If all pixel data within the second reference range is zero, the convolution operation result is zero, and the process proceeds to S112. In this case, the convolution operation is omitted. Thus, when it is determined that the data within a reference range of the same size as the feature image block is zero, the convolution processing unit 407 obtains zero as the convolution operation result without performing a sum-of-products operation. In this case, the number of sum-of-products operations corresponding to the second reference range is 0.

[0082] If it is not determined that all pixel data within the second reference range of the input feature image is zero, the process proceeds to S110. In this case, all pixel data within the first reference range is determined to be zero, but not all pixel data within the second reference range is determined to be zero. In S110, the convolution result is calculated by partially performing the convolution operation, similar to the embodiment described above. In this case, the convolution result can be obtained in the same way as in Figures 6(A) to (C). In the case of Figure 6(A), the number of multiply-accumulate operations corresponding to the first reference range is 3.

[0083] When processing according to the flowchart in Figure 1, even if all pixel data in feature image block 1101 are zero values, the convolution operation is partially performed to obtain the convolution result. On the other hand, when processing according to the flowchart in Figure 10, if all pixel data in feature image block 1101 are zero values, the process proceeds from S1001 to S1002. Also, in S1002, it is determined that all data within the second reference range is zero, so the convolution operation is omitted when obtaining the convolution result. On the other hand, when processing the feature image block shown in Figure 6(A) or Figure 7(A), a complete or partial convolution operation is performed in S110 or S111. In this way, by determining whether the pixel data within the second reference range, i.e., within the feature image block of the input feature image, is zero value, it is possible to omit all convolution operations in the filtering process and reduce the computational load. On the other hand, with this configuration, it is also possible to perform a partial convolution operation, which further reduces the computational load.

[0084] Furthermore, multiple reference ranges smaller than the feature image block of the input feature image may be used. For example, Figures 13(A) to 13(D) show an example of filtering a 5x5 size feature image block using a 3x3 size filter, similar to Figure 5(A). Here, as shown in Figure 13(A), if all the pixel data within the 5x5 size reference range 1312 in the feature image block 1301 is zero, the convolution operation can be omitted in the filtering process. On the other hand, if it is not determined that all the pixel data within the reference range 1311 is zero, a smaller reference range 1312 can be used, as shown in Figure 13(B). That is, if all the pixel data within the 5x4 size reference range 1312 in the feature image block 1302 is zero, the sum-of-products operation can be performed using only the pixel data of the top line of each kernel. Similarly, if it is not determined that all the pixel data within the reference range 1312 is zero, a smaller reference range 1313 can be used, as shown in Figure 13(B). In other words, if all pixel data within a 5x3 reference range 1313 in feature image block 1303 are zero, the sum-of-products operation can be performed using only the pixel data of the top two lines of each kernel. If it is determined that not all pixel data within the reference range 1313 are zero, the sum-of-products operation is performed using all pixel data of each kernel.

[0085] As described above, the number of sum-of-products operations to obtain the convolution result can be changed depending on whether all pixel data within the reference range are zero values ​​or not. Furthermore, the number of sum-of-products operations to obtain the convolution result can be changed depending on the reference range in which all pixel data is determined to be zero values. In addition, the order of sum-of-products operations using each pixel data may be changed depending on the reference range in which all pixel data is determined to be zero values.

[0086] The processing time for the convolution operation is shortest in the case of Figure 13(A), longer in the case of Figure 13(B), even longer in the case of Figure 13(C), and longest in the case of Figure 13(D). The control unit 401 can determine which of the above cases the feature image block falls into and, based on the determination result, can determine the processing time for the convolution operation by each convolution processing unit 412. In this case, the convolution processing unit 407 can obtain the necessary pixel data and filter coefficients at a timing corresponding to the determined processing time, based on the control signal from the control unit 401, and obtain the result of the convolution operation. By using multiple reference patterns, the probability of being able to omit the sum-of-products operation for feature images with various distributions can be increased.

[0087] The above processing is also applicable when the size of the feature image block and the filter size are different. Figure 8(A) shows an example where the previous layer is Layer 2 (filter size 5×5). In this example, the size of the feature image block in the input feature image is adjusted so that the size of the feature image block in the output feature image is 3×3. Specifically, if the filter size is M×M and the size of the feature image block in the output feature image is Y×Y, the size of the feature image block in the input feature image can be set to (Y+M-1)×(Y+M-1). Note that the size of the feature image block in the output feature image can be set to be less than or equal to the number of convolutional processing units 412. In this case, the pixel data of each pixel in the feature image block of the output feature image can be calculated in parallel by multiple convolutional processing units 412.

[0088] In the filtering process shown in Figure 8(A), a 5x5 size filter is used, and therefore the kernel size is 5x5. The input feature image 501 is divided into multiple feature image blocks 801, which are processed sequentially. Each feature image block 801 contains 7x7 pixel data. Filtering (and further processing such as activation) using the feature image blocks 801 yields the feature image block 812 of the output feature image 502. Each feature image block 812 contains 3x3 pixel data. In Figure 8(B), a 7x6 size reference range 802 is shown as an area enclosed by a dotted line. In this example, if it is determined in S109 that all pixel data within the reference range 802 are zero values, the filtering process performs a sum-of-products operation using the pixel data of the top line of each kernel. The size of the reference range can also be set to 7x7, 7x5, 7x4, or 7x3.

[0089] Figure 9(A) shows an example where the previous layer is Layer 3 (filter size 7x7). The input feature image 501 is divided into multiple feature image blocks 901, which are processed sequentially. Each feature image block 901 contains 9x9 pixel data. Filtering using feature image block 801 (and further processing such as activation) yields the feature image block 912 of the output feature image 502. Each feature image block 912 contains 3x3 pixel data. Figure 9(B) shows a 9x8 size reference range 902 as an area enclosed by a dotted line. In this example, if S109 determines that all pixel data within the reference range 902 are zero values, the filtering process performs a sum-of-accumulate operation using the pixel data of the top line of each kernel. The size of the reference range can also be set to 9x9, 9x7, 9x6, 9x5, 9x4, or 9x3.

[0090] Based on the above explanation, regardless of the filter size, determining whether the data within the reference range is zero or not makes it possible to omit some sum-of-products operations in the filtering process.

[0091] In this embodiment, the reference range corresponding to the processing target block may be determined in advance, or the reference range setting unit 406 may determine the reference range. For example, the reference range setting unit 406 can determine the reference range based on the size of the processing target block and the filter size. As an example, the reference range setting unit 406 can determine, as the reference range, the area excluding the top row among the feature image blocks of the input feature image. When the data within this reference range is a zero value, in the filter process, similar to FIGS. 6(A) to (C), the sum-of-products operation using the top row of each kernel can be performed, and the sum-of-products operation using the remaining rows of each kernel can be omitted. Further, the reference range setting unit 406 can determine, as the reference range, the area excluding the top N rows (where N < M in the case of a filter size of M×M) among the feature image blocks of the input feature image. When the data within this reference range is a zero value, in the filter process, the sum-of-products operation using the top N rows of each kernel can be performed, and the sum-of-products operation using the remaining rows of each kernel can be omitted.

[0092] In this specification, the reference range is a rectangular area set below the feature image block, but the method of setting the reference range is not limited to this. For example, the reference range may be the area excluding the leftmost column among the feature image blocks of the input feature image. When the data within this reference range is a zero value, in the filter process, the sum-of-products operation using the leftmost column of each kernel can be performed, and the sum-of-products operation using the remaining columns of each kernel can be omitted.

[0093] Furthermore, as explained with reference to Figures 5(B) and 5(C), a larger reference range can significantly reduce the amount of computation, but a smaller reference range increases the likelihood that data within the reference range will be zero. Therefore, the reference range setting unit 406 may select a reference range according to the input data array (feature image). For example, the reference range setting unit 406 can set a larger reference range when the frequency of zero values ​​in the feature image is higher, and a smaller reference range when it is lower. In this way, the reference range setting unit 406 may select a reference range according to the input data array in a way that minimizes processing time or maximizes the amount of computation that can be reduced.

[0094] In the above embodiment, the zero determination unit 409 determined whether the data within the reference range of the block to be processed is zero. Alternatively, when calculating the feature image of the previous level, it may record whether the data after activation processing is zero. In this case, when calculating the feature image of the next level, the zero determination unit 409 may determine whether the data within the reference range is zero based on this record. Furthermore, the zero determination unit 409 may determine whether the data within the reference range is zero in response to the data array within the block to be processed being stored in the feature data storage unit 405 or the data storage unit 402. For example, the zero determination unit 409 can determine whether the data within the reference range becomes zero when a feature image is stored in the feature data storage unit 405 or the data storage unit 402. This determination result can be referenced when performing filtering on the block to be processed. For example, the zero determination unit 409 may calculate a region where the pixel data is zero based on the position of the zero value in the feature image. Then, the zero determination unit 409 may determine whether the data within the reference range is zero by comparing this region with the reference range.

[0095] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0096] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0097] 403: Coefficient holding unit, 405: Feature data holding unit, 406: Reference range setting unit, 407: Convolution processing unit, 408: Data selection unit, 409: Zero determination unit

Claims

1. A processing unit that performs filtering on a data array within a processing block of a predetermined size, A data holding means for holding the data array within the block to be processed, A coefficient holding means for holding the weight coefficients of the filter used in the filtering process, A determination means for determining whether all data within a reference range smaller than the processing target block within the processing target block are zero values, The system includes a processing means for generating a convolution result of data and weight coefficients at each of a plurality of locations within the processing target block, If the determination means determines that all data within the reference range are zero, the processing means omits at least a portion of the sum-of-products operation between the data and the weight coefficients of the filter in the convolution operation, and the rows or columns of the filter from which the sum-of-products operation is omitted entirely, and the rows or columns of the filter from which the sum-of-products operation is not omitted, are determined based on the reference range.

2. If the determination means determines that the data within the reference range is zero, the processing means omits at least a portion of the sum-of-products operation between the data and the weight coefficient at each of the multiple locations within the block to be processed. The computing device according to claim 1, characterized in that

3. The arithmetic device according to claim 1 or 2, characterized in that one or more reference ranges are set, and at least one of the reference ranges is smaller than the processing target block.

4. If the determination means determines that the data within the reference range is zero, the processing means obtains the convolution result in the first processing for the position within the block to be processed. If the determination means determines that the data within the reference range is not zero, the processing means obtains the convolution result in the second process for the position within the block to be processed. The computing device according to any one of claims 1 to 3, characterized in that

5. If the determination means determines that the data within the reference range is zero, the processing means obtains the convolution result for all of the multiple positions within the processing target block using a common first process. If the determination means determines that the data within the reference range is not zero, the processing means obtains the convolution result for all of the multiple positions within the processing target block using a common second process. The computing device according to any one of claims 1 to 4, characterized in that

6. The arithmetic apparatus according to claim 4 or 5, characterized in that the first process and the second process are processes that obtain the convolution operation result by performing sum-of-products operations a different number of times.

7. The first process is a process of obtaining the convolution result by performing sum-of-products operations fewer times than the number of weight coefficients, The second process described above is a process of obtaining the convolution result by performing sum-of-products operations a number of times equal to the number of weight coefficients. The arithmetic device according to any one of claims 4 to 6, characterized in that

8. The determination means determines, for each of the multiple reference ranges, whether the data within the reference range is zero or not. If the determination means determines that the data within a specific reference range is zero, the processing means obtains the convolution result by performing a sum-of-products operation for the position within the processing target block a number of times corresponding to the specific reference range. The arithmetic device according to any one of claims 1 to 7, characterized in that

9. The determination means further determines whether the data within a reference range of the same size as the block to be processed is zero, If the determination means determines that the data within the reference range of the same size as the block to be processed is zero, the processing means obtains the zero value as the convolution result without performing a sum-of-products operation. The computing device according to claim 8, characterized in that

10. The computing device according to any one of claims 1 to 9, further comprising control means for controlling the supply of the data from the data holding means to the processing means in accordance with the determination result by the determination means.

11. The processing means includes a plurality of sum-of-accumulate means, The arithmetic device according to any one of claims 1 to 10, characterized in that the plurality of sum-of-products arithmetic means are configured to perform sum-of-products operations in parallel for a plurality of positions within the processing target block.

12. The arithmetic device according to claim 11, characterized in that the plurality of sum-of-products arithmetic means each perform a sum-of-products operation with a common weight coefficient and the data input to each in parallel.

13. If the determination means determines that the data within the reference range is not zero, the plurality of sum-of-products calculation means sequentially perform sum-of-products calculations between the data and the weight coefficients using each of the plurality of weight coefficients. The arithmetic device according to claim 12, characterized in that, if the determination means determines that the data within the reference range is zero, the plurality of sum-of-products calculation means sequentially perform sum-of-products calculations between the data and the weight coefficients using only a portion of the plurality of weight coefficients.

14. The arithmetic device according to any one of claims 1 to 13, further comprising setting means for setting the reference range based on the size of the block to be processed and the size of the filter.

15. The arithmetic device according to any one of claims 1 to 14, further comprising activation processing means for performing activation processing on the results of the filtering process.

16. The arithmetic device according to any one of claims 1 to 15, characterized in that the determination means determines whether or not the data within the reference range is zero in response to the data array in the processing target block being stored in the data holding means.

17. The aforementioned computing device is configured to perform processing using a convolutional neural network, The aforementioned data array is a feature image processed using a convolutional neural network. The computing device according to any one of claims 1 to 16, characterized in that the weight coefficient is a part of the weight coefficient of a convolutional neural network.

18. An information processing method performed by an arithmetic unit that performs filtering on a data array within a processing block of a predetermined size, A step of determining whether all data within a reference range smaller than the block to be processed within the block to be processed are zero values, The process includes generating a convolution result between the data and the filter weight coefficients at each of the multiple locations within the processing target block, An information processing method characterized in that, if it is determined that all data within the reference range are zero values, at least a portion of the sum-of-products operation between the data and the weight coefficients of the filter is omitted in the convolution operation, and the rows or columns of the filter from which the sum-of-products operation is omitted entirely and the rows or columns of the filter from which the sum-of-products operation is not omitted are determined based on the reference range.

19. A computer that performs filtering on a data array within a block of a predetermined size to be processed, A step of determining whether all data within a reference range smaller than the block to be processed within the block to be processed are zero values, A program characterized in that, when it is determined that all data within the reference range are zero values, the processing means that generates the convolution result of the data and weight coefficients at each of the multiple positions within the processing target block is controlled to omit at least a part of the sum-of-products operation between the data and the filter's weight coefficients in the convolution operation, and the rows or columns of the filter from which the sum-of-products operation is omitted and the rows or columns of the filter from which the sum-of-products operation is not omitted are determined based on the reference range.

Images