Arithmetic unit, information processing method, and program
Patent Information
- Application Number
- JP2021186520
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-11-16
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-16
AI Technical Summary
Existing convolutional neural network (CNN) processing methods incur increased processing load and power consumption due to repeated determinations of zero values in data arrays, particularly when performing convolution operations.
An arithmetic device with a configuration that includes a data holding unit, coefficient holding unit, and a control unit to determine zero values within a reference range, allowing for the selective performance of sum-of-products operations, thereby reducing unnecessary calculations.
This configuration improves the efficiency of convolution operation processing, reduces power consumption, and decreases processing time by selectively performing operations based on zero value determinations within reference ranges.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a computing device, an information processing method, and a program, and more particularly to computational processing using a convolutional neural network. [Background technology]
[0002] Convolutional neural networks (CNNs) are used in deep learning. Convolutional neural networks often perform convolution and activation processing at each layer. When the result of the convolution is a negative value, the result of the activation processing is zero, resulting in a feature map obtained at each layer containing many zero values. Patent Document 1 proposes reducing power consumption by skipping the convolution of an n×n filter with an n×n subregion if the proportion of zero values in the subregion is high. Patent Document 2 also discloses omitting a multiplication operation in a convolution operation when the filter weight coefficient or data value is zero. Patent Document 3 also discloses omitting a multiplication operation in a convolution operation when the pixel data of a feature image is zero. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] US Patent Application Publication No. 2019 / 0114532 [Patent Document 2] US Patent Application Publication No. 2019 / 0147324 [Patent Document 3] U.S. Patent No. 9,818,059 Summary of the Invention [Problem to be solved by the invention]
[0004] In the method of Patent Document 1, the determination of the proportion of zero values is repeated while sliding the partial region, which may increase the processing load required for the determination. Also, in the methods of Patent Documents 2 and 3, the determination of the zero value is repeated for each pixel of the feature map, which may increase the processing load required for the determination.
[0005] An object of the present invention is to improve the efficiency of convolution processing for a data array having zero values, and to reduce the power consumption or processing time required for the processing. [Means for solving the problem]
[0006] In order to achieve the object of the present invention, an arithmetic device according to one embodiment of the present invention has the following configuration: That is, an arithmetic device that performs a filter process on a data array within a block to be processed of a predetermined size, comprising: data storage means for storing the data array within the block to be processed; coefficient storage means for storing weighting coefficients of a filter used in the filter process; determination means for determining whether data within a reference range within the block to be processed that is set corresponding to the block to be processed is zero; processing means for generating a result of a convolution operation between data at each of a plurality of positions within the block to be processed and a weighting coefficient; and control means for controlling, depending on the determination result of the determination means, whether the processing means performs at least a part of a product-sum operation between the data and the weighting coefficient when generating the result of the convolution operation. [Effects of the Invention]
[0007] The present invention can improve the efficiency of convolution processing for a data array having zero values, and reduce the power consumption or processing time required for the processing. [Brief explanation of the drawings]
[0008] [Figure 1] 1 is a flowchart of a process according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the structure of a neural network according to an embodiment. [Figure 3] FIG. 1 is a block diagram showing an example of the arrangement of a processing device according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a CNN processing unit according to an embodiment. [Figure 5] FIG. 10 is a diagram showing an example of convolution processing and a reference range. [Figure 6] FIG. 10 is a diagram showing an example of a multiply-and-accumulate operation according to an embodiment; [Figure 7] FIG. 10 is a diagram showing an example of a multiply-and-accumulate operation according to an embodiment; [Figure 8] FIG. 10 is a diagram showing an example of convolution processing and a reference range. [Figure 9] FIG. 10 is a diagram showing an example of convolution processing and a reference range. [Figure 10] 1 is a flowchart of a process according to an embodiment. [Figure 11] FIG. 10 is a diagram showing an example of a reference range. [Figure 12] A diagram showing the relationship between neural networks and convolution operations. [Figure 13] FIG. 10 is a diagram showing an example of a reference range. DETAILED DESCRIPTION OF THE INVENTION
[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.
[0010] A computing device according to an embodiment of the present invention is capable of performing filter processing on a data array within a processing target block of a predetermined size. FIG. 3 is a block diagram showing an example configuration of a convolutional neural network processing device, which is a computing device according to an embodiment of the present invention. Such a processing device can, for example, perform processing on an image using a neural network. Specific examples of processing that can be performed by such a processing device include detection processing of an object in an image, identification processing of an object in an image, region segmentation processing of an image, and high-resolution processing of an image. Such processing can be performed, for example, using a feature map obtained by inputting an image into a neural network.
[0011] The input unit 301 is a device that accepts instructions or data from a user, and may be, for example, a keyboard, a pointing device, or a button.
[0012] The data storage unit 302 can store data such as image data. The data storage unit 302 may be, for example, a hard disk, a flexible disk, a CD-ROM, a CD-R, a DVD, a memory card, a CF card, a SmartMedia card, an SD card, a memory stick, an xD picture card, or a USB memory. The data storage unit 302 may also store programs or other data. Note that a portion of the RAM 308, which will be described later, may be used as the data storage unit 302.
[0013] The communication unit 303 is an interface (I / F) for communication between devices. The processing device 300 can exchange data with other devices via the communication unit 303. The processing device 300 may use a storage device connected via the communication unit 303 as a virtual data storage unit, i.e., as the data storage unit 302.
[0014] The display unit 304 is a device that displays information to a user or the like. The display unit 304 can display, for example, an image before or after image processing, or other images such as a GUI. The display unit 304 may be, for example, a CRT or a liquid crystal display. The display unit 304 may be a device external to the processing device 300, connected by a cable or the like. The input unit 301 and the display unit 304 may be the same device, and for example, the input unit 301 and the display unit 304 may be a touchscreen device. In this case, input on the touchscreen corresponds to input to the input unit 301.
[0015] The CNN processing unit 305 can perform neural network processing (S101 to S117) on an image according to the flowchart of FIG. 1, which will be described later. The processing performed by the CNN processing unit 305 includes filtering using a convolution operation. The CNN processing unit 305 may also perform neural network processing on the image processing results stored in the RAM 308 by the image processing unit 309. The CNN processing unit 305 can output the processing results to the data storage unit 302 (or RAM 308). The processing results can be used for various processes, such as image processing or image recognition processing, by the CPU 306. However, such a CNN processing unit 305 may be used for purposes other than image processing; that is, components other than the CNN processing unit 305 shown in FIG. 3 are not essential to the present invention. The CNN processing unit 305 can perform filtering on still images or moving images. For example, the CNN processing unit 305 can perform filtering on each of multiple frames included in a moving image. In this case, the CPU 306 can perform image processing or image recognition on the moving image.
[0016] The CPU 306 controls the overall operation of the processing device 300. The CPU 306 can also perform various types of processing, such as image processing or image recognition processing, based on processing results generated by the CNN processing unit 305 and stored in the data storage unit 302 or the RAM 308. The CPU 306 can store these processing results in the RAM 308.
[0017] ROM 307 and RAM 308 provide CPU 306 with programs, data, work areas, and the like required for processing by CPU 306. Programs required for processing by CPU 306 may be stored in data storage unit 302 or ROM 307, or may be loaded from data storage unit 302 or ROM 307 into RAM 308. Data processing device 300 may also receive programs via communication unit 303. In this case, the programs may be recorded in data storage unit 302 and then loaded into RAM 308, or may be directly loaded from communication unit 303 into RAM 308. In either case, CPU 306 can execute the programs loaded into RAM 308.
[0018] The image processing unit 309 can perform image processing on image data. For example, the image processing unit 309 can read image data written in the data storage unit 302, adjust the range of pixel values, and write the processing results to the RAM 308 in accordance with instructions from the CPU 306.
[0019] The processing device 300 shown in Fig. 3 includes the above-described units. The above-described units are connected to each other so as to be able to transmit and receive data to and from each other. However, for example, the units including the input unit 301, data storage unit 302, and display unit 304 may be connected to each other via a communication path conforming to a known communication method. In other words, the data processing device according to one embodiment may be composed of multiple physically separate devices.
[0020] 3 has one CPU 306, it may have multiple CPUs. Furthermore, at least some of the functions of the units (e.g., the CNN processing unit 305 and the image processing unit 309) of the processing unit 300 may be realized by the CPU 306 operating in accordance with a program.
[0021] The processing device 300 may include various components not shown in FIG. 3, but the description thereof will be omitted.
[0022] (Example of neural network structure) As described above, the CNN processing unit 305 can perform filtering using a filter on a data array. The CNN processing unit 305 can also perform processing according to a neural network including multiple layers, and can perform such filtering in at least one layer. The filtering includes a convolution operation, which includes multiple product-sum operations. Hereinafter, one product-sum operation refers to a set of operations: a multiplication operation between one piece of data and one filter coefficient, and an operation to accumulate this product. A single convolution operation refers to an operation to obtain one piece of output data by convolving a filter with a specific data array (e.g., a local region of a feature image), and includes multiple product-sum operations. The following describes the case where filtering is performed on a feature image. A feature image has pixel data for each pixel as a data array.
[0023] An example of a neural network used by the CNN processing unit 305 will be described below. CNN, a type of neural network, has a structure in which multiple layers are hierarchically connected. Each layer may include multiple feature images. In the following, a feature image obtained by performing corresponding processing on a feature image in a previous layer will be referred to as a feature image in the next layer. Note that, although a case in which the feature image is two-dimensional will be described below, the feature image may be one-dimensional or may be a higher-order feature image of three or more dimensions.
[0024] For example, a feature image in the next layer may be calculated by filtering the feature image in the previous layer. This filtering can use a filter configured with filter coefficients corresponding to the previous layer. Each of the multiple feature images in the next layer can be generated by filtering using a corresponding filter. Furthermore, multiple feature images in the previous layer may be used to calculate one feature image in the next layer. For example, filtering can be performed on each of the multiple feature images in the previous layer using a corresponding filter, and one feature image in the next layer can be obtained based on the multiple processing results obtained.
[0025] For example, the feature image after filtering (O i,j (n)) is the feature image of the previous layer (I i,j (m)) and the filter coefficients (W 0,0 (m,n)~W X-1,Y-1 (m, n)) according to formula (1). Here, i and j indicate the coordinates of the feature image. Also, x and y indicate the coordinates of the filter. n is the number of the feature image in the next layer. Also, m is the number of the feature image in the previous layer, and there are IC feature images in the previous layer. The filter coefficients are different for each feature image in the previous layer and for each feature image in the next layer, and there are X × Y filter coefficients for one combination of feature images.
number
[0026] As described above, the number of product-sum operations performed in the convolution operation to calculate one pixel data item of one feature image in the next layer is M × X × Y. In this way, the filter has multiple filter coefficients, and the pixel value of each pixel in the feature image after filter processing is obtained by convolution of the pixel values of the pixel group around the corresponding pixel in the feature image in the previous layer with the filter coefficients of the filter.
[0027] The feature image O obtained by such filtering i,jBy further processing (n) such as activation processing or pooling processing, the next-level feature image can be calculated. Activation processing can be performed according to equation (2). In equation (2), f(·) is a function of ReLu (Rectified Linear Unit), and the variable x is the input data.
number
[0028] When performing activation processing according to equation (2), if the result of the convolution operation is a negative value, the result of the activation processing will be zero. In this case, the pixel data at the corresponding position in the feature image of the previous layer will be zero, and a zero value will also be input to the product-sum operation in the next layer. If the pixel data of the feature image is zero, this data does not contribute to the result of the convolution operation, so the result will not be affected even if the product-sum operation using this data is omitted.
[0029] FIG. 2 shows a specific example of the structure of a neural network. The neural network shown in FIG. 2 has four layers, each containing four feature images. The feature images of each layer are obtained based on the filter processing results obtained by applying a filter, specific to each feature image, to the pixel data of the feature image. Here, the filter coefficients of the filters are obtained in advance using a known learning technique. The filter processing for applying the filters includes a convolution operation, i.e., multiple multiplications and cumulative additions. In FIG. 2, arrows indicate convolution operations.
[0030] In layer 1, a plurality of feature images 202 in layer 2 are generated by filtering using a plurality of feature images 201 and filter coefficients based on equations (1) and (2). In layer 2, a plurality of feature images 203 in layer 3 are generated by filtering using a plurality of feature images 202 and filter coefficients. In layer 3, a plurality of feature images 203 in layer 4 are generated by filtering using a plurality of feature images 203 and filter coefficients. In this way, filtering is performed in hierarchical order. As shown in FIG. 12 , a plurality of pixel data are extracted from the same position in four feature images 1201 in layer 1, and filtering and activation processing are performed. The processing result becomes a portion of the pixel data of the feature image 1202 in layer 2.
[0031] 2 further shows the type of filtering and the filter size for each layer. Filtering is performed using filters of size 3x3, size 5x5, and size 7x7 for Layer 1, Layer 2, and Layer 3, respectively. In this way, the size of the filter used for filtering may differ for each layer.
[0032] Network structure information indicating the structure of the convolutional neural network may be stored in the RAM 308. This network structure information may include, for example, the number of layers, the number of feature images in each layer, the type of filtering performed in each layer, the types of activation processing and pooling processing performed in each layer, and the like.
[0033] 4 shows an example of the functional configuration of the CNN processing unit 305. In this embodiment, the CNN processing unit 305 can perform filter processing on a data array in a processing target block of a predetermined size. The CNN processing unit 305 has a coefficient storage unit 403, a feature data storage unit 405, a data selection unit 408, a zero determination unit 409, and a convolution processing unit 407. Furthermore, as will be described later, the CNN processing unit 305 may have a control unit 401, a data storage unit 402, a read unit 403, a reference range setting unit 406, an activation processing unit 410, and a result storage unit 411.
[0034] The data holding unit 402 holds some of the data in the data saving unit 302. The coefficient holding unit 403 holds the weighting coefficients (filter coefficients) of the filter used in the filtering process. For example, the coefficient holding unit 405 holds the filter coefficients W obtained from the data holding unit 402. x,y The feature data storage unit 405 can store pixel data (m, n) in the feature image I(m). The feature data storage unit 405 stores the data array in the target block. For example, the feature data storage unit 405 can store pixel data for a portion of the feature image I(m). This filter coefficient is part of the weighting coefficient of the convolutional neural network.
[0035] The reference range setting unit 406 sets a reference range corresponding to the block to be processed. The reference range may be determined in advance according to the size of the block to be processed and the size of the filter. The reference range setting unit 406 can set one or more reference ranges, and at least one of the reference ranges is smaller than the block to be processed, such as reference ranges 504 and 505 shown in FIGS. 5(B) and 5(C). Details of the reference ranges will be described later.
[0036] The convolution processing unit 407 generates a convolution operation result between data at each of multiple positions in the processing target block and filter weighting coefficients (filter coefficients). In this embodiment, the convolution processing unit 407 can perform a convolution operation to convolve the processing target block in the input feature image with a filter, and can generate a filter processing result for the input feature image. For example, the convolution processing unit 407 can obtain the convolution operation result using the filter coefficients and pixel data according to equation (1). In this embodiment, the convolution operation unit 407 calculates the convolution operation result using the pixel data stored in the feature data storage unit 405 and the filter coefficients stored in the coefficient storage unit 403 in accordance with a control signal from the control unit 401.
[0037] The convolution processing unit 407 has multiple convolution processing units 412. Each convolution processing unit 412 can perform a multiply-and-accumulate operation to accumulate the products of pixel data and filter coefficients. Furthermore, the multiple convolution processing units 412 can perform the multiply-and-accumulate operation in parallel for multiple positions within the block to be processed. The convolution processing unit 412 has an arithmetic core 418, a feature data storage unit 413, a coefficient storage unit 414, and a result storage unit 417. The multiplier 415 and adder 416 used in the multiply-and-accumulate operation are included in the arithmetic core 418. However, in one embodiment, a single convolution processing unit 412 may be used to perform the multiply-and-accumulate operation sequentially for each of multiple positions within the block to be processed. Details of the processing by the convolution processing unit 407 will be described later.
[0038] The data selection unit 408 transfers pixel data used in processing by the convolution processing unit 407 to the convolution processing unit 407. The data selection unit 408 also controls whether to perform at least some of the product-sum operations at multiple positions within the block to be processed, depending on the determination result by the zero determination unit 409. The zero determination unit 409 determines whether data within a reference range within the block to be processed, which is set by the reference range setting unit 406 corresponding to the block to be processed, is zero. Under the control of the data selection unit 408 and the zero determination unit 409, the convolution processing unit 407 can omit some of the product-sum operations used to generate the filter processing result for the block to be processed. In other words, the convolution processing unit 407 can generate the filter processing result for the block to be processed by partially performing the convolution operation. In this embodiment, if the data selection unit 408 determines that the data within the reference range is zero, the convolution processing unit 407 can omit at least some of the product-sum operations between the data and the weighting coefficients at each of multiple positions within the block to be processed. The processing by the data selection unit 408 and the zero determination unit 409 will be described in detail later.
[0039] In the following embodiments, the zero value in this specification refers to zero itself, but it may also refer to a number whose absolute value is equal to or less than a predetermined value (for example, equal to or less than 1) and which has little effect on the convolution calculation result. Furthermore, in the following embodiments, the zero value of data within the reference range means that all data within the reference range is zero. On the other hand, as in Patent Document 1, the zero value of data within the reference range may also mean that the number of zeros is equal to or greater than a predetermined percentage (for example, 85% or more) and has little effect on the convolution calculation result.
[0040] The activation processor 410 further performs activation processing on the result of the filtering processing. The activation processor 410 can calculate the activation processing result, for example, according to equation (2). However, the type of activation processing is not particularly limited, and the activation processor 410 may perform activation processing using other nonlinear functions or quantization functions. Furthermore, the activation processor 410 may adjust the size of the output feature image by performing pooling processing based on the result of the activation processing according to the network structure information. In some cases, both or either of the activation processing and the pooling processing may be omitted.
[0041] The result storage unit 411 stores the processing results obtained by the activation processing unit 410. The reading unit 403 transfers addresses for accessing the feature data storage unit 405 and the coefficient storage unit 403 to the zero determination unit 409 and the convolution processing unit 407. The zero determination unit 409 and the convolution processing unit 407 can read pixel data from the feature data storage unit 405 or filter coefficients from the coefficient storage unit 403 according to these addresses.
[0042] These processes do not need to be performed by dedicated processing units. For example, the activation process and the pooling process may be performed by the CPU 306. Furthermore, one or more of the processes of the reference range setting unit 406, the zero determination unit 408, and the data selection unit 408 may be performed by the CPU 306.
[0043] 1 is an example of a flowchart of an information processing method performed by the CNN processing unit 305. The control processes shown in S101 to S117 can be performed by the control unit 401 (for example, a CPU or sequencer included in the control unit 401). Each step of the process using the convolutional neural network in this embodiment will be described below with reference to FIG. 1.
[0044] In S101, the control unit 401 reads out an input feature image, filter coefficients used in the filter processing, and network structure information from the RAM 308, and stores them in the data storage unit 402. In the example of Fig. 2, the input feature image may be an image of each of the RGB+D (depth) planes, or may be a feature image obtained by performing filter processing on an image.
[0045] In S102, a loop for each layer begins. In S102, the control unit 401 can select the first layer. In the following description, the layer selected in S102 is referred to as the previous layer, and the layer after the previous layer is referred to as the next layer. By performing the processes of S103 to S116 for each layer in order, the results of processing using the convolutional neural network can be obtained.
[0046] In S103, a loop for each block begins. In this embodiment, each output feature image in the next layer is divided into multiple feature image blocks. Furthermore, pixel data for a feature image block in one output feature image is calculated using pixel data for the corresponding feature image block in the input feature image in the previous layer. For example, in the example of FIG. 5(A), pixel data for feature image block 512 in output feature image 502 is obtained by filtering feature image blocks 503 in each of multiple input feature images 501. In this example, the feature image blocks in the output feature image are adjacent and do not overlap, while the feature image blocks in the input feature images are aligned so as to overlap each other. Each feature image block corresponds to a target block for processing.
[0047] In S103, one feature image block (e.g., feature image block 512) of the output feature image is selected. Also, a corresponding feature image block (e.g., feature image block 503) of the input feature image is selected, which is used to calculate pixel data for this feature image block in the output feature image. In S104 to S115, one feature image block common to multiple output feature images is selected, and pixel data for each output feature image in the selected feature image block is calculated. In this case, pixel data for each input feature image in the selected feature image block is referenced. By sequentially performing the processes of S104 to S115 for each feature image block, each output feature image in the next layer can be obtained.
[0048] In S104, a loop is started for each output feature image in the next layer. In S105 to S114, pixel data of one output feature image in the feature image block selected in S103 is calculated. In this way, pixel data is calculated in order for each of the multiple output feature images in the next layer.
[0049] In S105, the control unit 401 initializes the convolution operation result stored in the result storage unit 417 of the convolution processing unit 407. For example, the control unit 401 can set the convolution operation result to zero.
[0050] In S106, a loop is started for each input feature image in the previous layer. In S107 to S111, a filter process is performed on the feature image block selected in S103 from one input feature image. By performing the processes of S107 to S111 in order, the filter process is performed on each input feature image. The loop of S107 to S111 can be performed for each input feature image referenced to obtain the output feature image selected in S104.
[0051] In S107, the reference range setting unit 406 sets a reference range for pixel data with a zero value. The method for setting the reference range by the reference range setting unit 406 will be described later.
[0052] In S108, the control unit 401 reads out a portion of the input feature image from the data storage unit 402 and transfers it to the feature data storage unit 405. The control unit 401 can transfer pixel data in the feature image block selected in S103, of the input feature image selected in S106, to the feature data storage unit 405. The control unit 401 also reads out a portion of the filter coefficients from the data storage unit 402 and transfers them to the coefficient storage unit 403. The control unit 401 can transfer filter coefficients for filtering the input feature image selected in S106 to the coefficient storage unit 403 in order to obtain the output feature image selected in S104. In this way, in S108, the control unit 401 can read out, from the data storage unit 402, pixel data and filter coefficients to be referenced in the convolution operations performed in S109 to S111.
[0053] In S109, the zero determination unit 409 determines whether all pixel data within the reference range of the input feature image is zero. If all pixel data within the reference range is zero, the process proceeds to S110. If not, the process proceeds to S111.
[0054] In S110, the convolution processing unit 407 generates a convolution operation result for the feature image block selected in S103 from the input feature image selected in S106. This convolution operation result is composed of the results of product-sum operations between pixel data and filter coefficients for each of multiple positions in the feature image block. In S110, the convolution processing unit 407 acquires the convolution operation result for the position in the feature image block through the first process. Specifically, in S110, the convolution operation unit 407 can omit some of the product-sum operations between pixel data and filter coefficients for each of multiple positions in the feature image block. Specific processing in S110 will be described later.
[0055] In S111, the convolution processing unit 407 generates a convolution operation result for the feature image block selected in S103 from the input feature image selected in S106. In S111, the convolution processing unit 407 obtains the convolution operation result by a second process different from S110. For example, the convolution processing unit 407 can perform a product-sum operation between pixel data and filter coefficients for each of a plurality of positions in the feature image block without omitting any operations.
[0056] In S112, the control unit 401 determines whether the loop of input feature images has ended. If the processing for all input feature images has ended, the process proceeds to S113. If not, the process returns to S107, and processing for the next input feature image is started.
[0057] When proceeding from S112 to S113, the result storage unit 417 has accumulated the filter processing results for the input feature image selected in S106 for each pixel. For example, the result storage unit 417 stores pixel data O according to equation (1) for each pixel of the feature image block selected in S103 in the output feature image selected in S104. i,j (n) may be stored.
[0058] In S 113 , the activation processing unit 410 performs activation processing based on the filter processing results stored in the result storage unit 417 in accordance with a control signal from the control unit 401 .
[0059] In S114, the control unit 401 stores the processing result by the activation processing unit 410 in the data storage unit 402. The processing result stored in the data storage unit 402 corresponds to the pixel data of the feature image block selected in S103 from the output feature image selected in S104. The pixel data of the output feature image stored in this manner is used as pixel data of the input feature image when processing the next layer.
[0060] In S115, the control unit 401 determines whether the loop for the output feature image has ended. If the processing for all output feature images has ended, the process proceeds to S116. If not, the process returns to S105, and the processing for the next output feature image is started.
[0061] In S116, the control unit 401 determines whether the loop for the feature image blocks has ended. If the processing for all feature image blocks has ended, the process proceeds to S117. If not, the process returns to S104, and processing for the next feature image block begins.
[0062] In S117, the control unit 401 determines whether the loop for the hierarchical layers has ended. If the processing for all hierarchical layers has ended, the processing in Fig. 1 ends. If not, the process returns to S103, and the processing for the next hierarchical layer begins.
[0063] (Processing example according to this embodiment) According to this embodiment, the amount of calculation in the filter processing can be reduced, thereby improving the processing efficiency of the filter processing. The improvement in processing efficiency according to this embodiment will be described with reference to Figs. 5 to 7. The processing device according to this embodiment is capable of processing multiple pixel data contained in multiple feature images in parallel, but the following description will be given of a case where processing is performed using a four-layer convolutional neural network shown in Fig. 2. In Fig. 2, solid line blocks indicate processing targets, and solid line arrows indicate convolution operations related to the processing targets.
[0064] FIG. 5(A) shows an example of a process for generating a feature image (output feature image) of layer 2 by filtering a feature image (input feature image) of layer 1. This filtering process uses a 3×3 filter, so the kernel size is 3×3. The input feature image 501 is divided into multiple feature image blocks 503, which are processed sequentially. Each feature image block 503 of the input feature image 501 contains 5×5 pixel data. Furthermore, filtering (and further processing such as activation processing) using the feature image block 503 results in a feature image block 512 of the output feature image 502. Each feature image block 512 contains 3×3 pixel data. Also, FIGS. 5(B) and 5(C) show examples of zero-value reference ranges: reference range 504, a 5×4 area surrounded by dotted lines, and reference range 505, a 5×3 area.
[0065] In the processing example shown in Figures 6(A) to (C), block 601 is a feature image block of the input feature image, and has a size of 5x5 and 25 pixel data. Each pixel is assigned a different number from 1 to 25. In this processing example, block 601 is subjected to filtering processing using a 3x3 filter with a vertical stride of 1 and a horizontal stride of 1. Nine 3x3 kernels 602 to 610 (kernel 1 to kernel 9) are used. Each of the kernels 602 to 610 overlaps with an adjacent kernel, and they share some pixel data. In Figures 6(A) and 6(C), white pixels indicate pixel data with a value of zero, and hatched pixels indicate pixel data with a value other than zero.
[0066] In this embodiment, the nine kernels 602 to 610 are processed in parallel by the nine convolution processing units 412. The pixel data in each of the kernels 602 to 610 is processed in sequence. The same filter is applied to the kernels 602 to 610. Therefore, the convolution processing units 412 can perform multiply-and-accumulate operations in parallel between the data input to each of them (for example, the pixel data at the upper left of the kernels 602 to 610) and a common weighting coefficient (for example, the weighting coefficient at the upper left of the filter).
[0067] As shown in the examples of FIGS. 6A and 6C, when all pixel data in the reference range 504 is zero, the pixel data in the bottom two lines of kernels 602 to 604 and the three lines of kernels 605 to 610 all have zero values. In other words, only the top line of kernels 602 to 610 contains non-zero values. Furthermore, the bottom two lines of kernels 602 to 610 have zero values, and product-sum operations using these pixel data do not affect the filter processing results. Therefore, in this embodiment, the nine convolution processing units 412 process only the pixel data in the top line of the corresponding kernel, as indicated by arrows 611 in FIG. 6B. For example, instead of multiplying each of pixels 1 to 9 by a corresponding filter coefficient and accumulating the resulting products, the convolution processing units 412 multiply only each of pixels 1 to 3 by a corresponding filter coefficient and accumulating the resulting products. In this way, the convolution processing unit 412 can obtain convolution operation results for all of the multiple positions in the processing target block by performing the same first process. With this configuration, each convolution processing unit 412 performs three multiply-accumulate operations, while eliminating six multiply-accumulate operations.
[0068] In this embodiment, processing using pixel data from the top one line for kernels 605 to 610 is also performed, but since the calculation result is zero, this processing result does not affect the filter processing result. However, processing using pixel data from the top one line for kernels 605 to 610 may be omitted. In this case, of the nine convolution processing units 412, the convolution processing unit 412 that processes kernels 605 to 610 can be controlled so as not to perform calculation processing on block 601.
[0069] The processing in S110 can be performed as described above. That is, since it is determined in S109 that all pixel data in the reference region are zero, the convolution calculation is performed partially. That is, each of the convolution processing units 412 operating in parallel performs a multiply-and-accumulate operation between three pixel data of each kernel and three filter coefficients, and does not perform a multiply-and-accumulate operation between the remaining six pixel data of each kernel and six filter coefficients. As described above, since the remaining six pixel data of each kernel are zero, not performing a calculation using this pixel data does not affect the filter processing result. In S110, the convolution calculation result is obtained by performing a number of multiply-and-accumulate operations (three in this example) fewer than the number of filter coefficients (nine in this example). Furthermore, the convolution processing unit 412 performs a multiply-and-accumulate operation between the sequential data and the filter coefficients using only a portion of the multiple filter coefficients. Furthermore, the convolution processing unit 412 performs a multiply-and-accumulate operation between the sequential data and each weighting coefficient using each of the multiple weighting coefficients.
[0070] At this time, coefficient data used in the product-sum operation is supplied from the filter coefficient holding unit 403 to the convolution processing unit 407. Furthermore, pixel data used in the product-sum operation is supplied from the zero determination unit 409 to the convolution processing unit 407 via the data selection unit 408. Here, the data selection unit 408 can control the supply of pixel data from the filter coefficient holding unit 403 to the convolution processing unit 407 according to the determination result by the zero determination unit 409. That is, the data selection unit 408 can control the convolution processing unit 407 to perform product-sum operation using specific pixel data and omit product-sum operation using specific pixel data. For example, the data selection unit 408 may control the product-sum operation by the convolution processing unit 407 by supplying only pixel data used in the product-sum operation to the convolution processing unit 407. Furthermore, the transfer of filter coefficients may be controlled according to the determination result by the zero determination unit 409 so that only the filter coefficients used in the product-sum operation are transferred from the filter coefficient holding unit 403 to the convolution processing unit 407.
[0071] Alternatively, the size of the reference range may be 5×3, as in the reference range 505 shown in FIG. 5(C). When all pixel data within the reference range 505 are zero, the pixel data in the bottom line of kernels 602-604, the bottom two lines of kernels 605-607, and the three lines of kernels 608-610 all have zero values. Therefore, in this embodiment, the nine convolution processing units 412 can process only the pixel data in the top two lines of the corresponding kernel. Note that in this embodiment, processing using pixel data in the center line for kernels 605-607 and processing using pixel data in the top two lines for kernels 608-610 are also performed. However, because the calculation results using these pixel data are zero, the processing results do not affect the filter processing results.
[0072] In this way, it is possible to set reference ranges of various sizes. On the other hand, as can be seen from the above explanation, the larger the reference range, the greater the effect of reducing the amount of calculation.
[0073] On the other hand, when it is not determined that all pixel data within the reference range 504 are zero, as in the example shown in FIG. 7A, the nine convolution processing units 412 process only the pixel data of the top line of the corresponding kernel, as indicated by the arrow 611 in FIG. 7B. That is, the calculation process using all pixel data for the kernels 702 to 710 shown in FIG. 7C is performed to the end. The process in S111 can be performed in this manner. That is, the convolution processing units 412 can obtain convolution calculation results for all of the multiple positions within the feature image block by a common second process, which is different from the first process in which some product-sum calculations are omitted. Furthermore, the first process in S110 and the second process in S111 obtain convolution calculation results by different numbers of product-sum calculations. In S111, the convolution calculation results are obtained by performing product-sum calculations a number of times (9 in this example) equal to the number of filter coefficients (9 in this example).
[0074] As described above, according to this embodiment, it is determined whether all data within the reference range is zero, and some of the product-sum operations for obtaining the convolution operation result are omitted according to the determination result. This reduces the amount of calculation, improves the efficiency of the convolution operation process, and reduces the power consumption and processing time required for the process. In this embodiment, it is determined whether data in a reference range larger than the filter size is zero. In other words, it is possible to control each of the multiple convolution operations based on the determination result for the reference range. This makes it easy to reduce the amount of calculation.
[0075] In particular, in this embodiment, a reference range having a size different from the block size of the input feature image is used. Therefore, the amount of calculation can be reduced even if all pixel data in a block is not zero. Furthermore, according to the configuration of this embodiment, even if all data in a kernel is not zero, the amount of calculation can be reduced and processing can be accelerated by omitting some of the multiply-and-accumulate operations as indicated by arrow 611. This configuration is particularly effective when multiple convolution processing units perform processing on different kernels in parallel, as shown in FIG. 4. That is, when each convolution processing unit determines to omit multiplication of pixel data with zero values or to omit convolution operations on a kernel with many zero values, the processing time of each convolution processing unit varies. Therefore, additional circuitry may be required to synchronize the operation of multiple convolution processing units. On the other hand, according to the configuration of this embodiment, as described with reference to FIGS. 6(A) to 6(C), it is possible to reduce the amount of calculation while simultaneously operating multiple convolution processing units in synchronization.
[0076] Furthermore, according to the method of Patent Document 1, the convolution operation is skipped even when a small number of non-zero values exist in the kernel, which may result in a decrease in operation accuracy. On the other hand, according to the above-described embodiment, the configuration in which it is determined whether all data in the reference range is zero and a part of the product-sum operation for obtaining the convolution operation result is omitted in accordance with the determination result can maintain the accuracy of the filter processing.
[0077] [Variations] In the above embodiment, it is determined whether pixel data within one reference range is zero, but two or more reference ranges may be used. For example, the zero determination unit 409 may determine whether data within each of multiple reference ranges is zero. If the zero determination unit 409 determines that data within a specific reference range is zero, the convolution processing unit 407 can obtain a convolution operation result for a position within the feature image block by performing a number of product-sum operations corresponding to the specific reference range.
[0078] In the above-described embodiment, when pixel data in the reference range is zero, the convolution operation is partially performed when obtaining the convolution operation result for the feature image block of the input feature image. On the other hand, the convolution operation may be omitted when obtaining the convolution operation result for the feature image block of the input feature image. Such a case will be described below.
[0079] In this case, instead of the process of S109 in Fig. 1, the processes of S1001 and S1002 can be performed as shown in Fig. 10. Below, a case where two reference ranges are used as shown in Figs. 11(A) and (B) will be described. As with the reference ranges in the above-described embodiment, the first reference range 1121 has a size smaller than the feature image block 1101 of the input feature image. Furthermore, the second reference range 1122 has the same size as the feature image block 1101 of the input feature image. The first reference range 1121 is included in the second reference range 1122. Also, in this example, it is assumed that filtering is performed using a 3 × 3 filter, as in Figs. 6(A) to (C).
[0080] In S1001, which is performed after S108, the zero determination unit 409 determines whether all pixel data within the first reference range of the input feature image is zero. If all pixel data within the first reference range is zero, the process proceeds to S1002. If not, the process proceeds to S111, where a convolution operation is performed.
[0081] In S1002, the zero determination unit 409 determines whether all pixel data within the second reference range of the input feature image is zero. If all pixel data within the second reference range is zero, the convolution operation result is zero, and processing proceeds to S112. In this case, the convolution operation is omitted. In this way, for a reference range of the same size as the feature image block, if it is determined that the data within the reference range is zero, the convolution processing unit 407 acquires a zero value as the convolution operation result without performing a product-sum operation. In this case, the number of product-sum operations corresponding to the second reference range is 0.
[0082] If it is not determined that all pixel data within the second reference range of the input feature image is zero, the process proceeds to S110. In this case, all pixel data within the first reference range is determined to be zero, but not all pixel data within the second reference range is determined to be zero. In S110, as in the above-described embodiment, the convolution operation result is calculated by performing the convolution operation partially. In this case, the convolution operation result can be obtained as in Figures 6(A) to 6(C). In the case of Figure 6(A), the number of product-sum operations corresponding to the first reference range is three.
[0083] When processing according to the flowchart of FIG. 1, even if all pixel data in the feature image block 1101 are zero, a partial convolution operation is performed to obtain a convolution operation result. On the other hand, when processing according to the flowchart of FIG. 10, if all pixel data in the feature image block 1101 are zero, processing proceeds from S1001 to S1002. Furthermore, because it is determined in S1002 that all data in the second reference range are zero, the convolution operation is omitted when obtaining the convolution operation result. On the other hand, when processing the feature image block shown in FIG. 6(A) or FIG. 7(A), a full or partial convolution operation is performed in S110 or S111. In this way, by determining whether pixel data in the second reference range, i.e., within the feature image block of the input feature image, are zero, all convolution operations can be omitted in the filter processing, thereby reducing the amount of calculation. On the other hand, this configuration also allows partial convolution operations to be performed, further reducing the amount of calculation.
[0084] Furthermore, multiple reference ranges smaller than the feature image block of the input feature image may be used. For example, FIGS. 13A to 13D show an example of performing filter processing using a 3×3 filter on a 5×5 feature image block, similar to FIG. 5A. Here, as shown in FIG. 13A, if all pixel data within a 5×5 reference range 1312 in a feature image block 1301 are zero, the convolution operation can be omitted in the filter processing. On the other hand, if it is not determined that all pixel data within the reference range 1311 are zero, a smaller reference range 1312 can be used, as shown in FIG. 13B. That is, if all pixel data within a 5×4 reference range 1312 in a feature image block 1302 are zero, a product-sum operation can be performed using only the pixel data in the top line of each kernel. Similarly, if it is not determined that all pixel data within the reference range 1312 are zero, a smaller reference range 1313 can be used, as shown in FIG. 13B. That is, if all pixel data within a 5x3 reference range 1313 in the feature image block 1303 are zero, a product-sum operation can be performed using only the pixel data in the top two lines of each kernel. Also, if it is not determined that all pixel data within the reference range 1313 are zero, a product-sum operation is performed using all pixel data in each kernel.
[0085] As described above, the number of product-sum operations to obtain the convolution operation result can be changed depending on whether or not all pixel data in the reference range is zero. Also, the number of product-sum operations to obtain the convolution operation result can be changed depending on the reference range in which it is determined that all pixel data is zero. Furthermore, the order of product-sum operations using each pixel data may be changed depending on the reference range in which it is determined that all pixel data is zero.
[0086] The processing time for the convolution operation is shortest in the case of FIG. 13(A), longer in the case of FIG. 13(B), even longer in the case of FIG. 13(C), and longest in the case of FIG. 13(D). The control unit 401 determines which of the above cases the feature image block falls into and can determine the processing time for the convolution operation by each convolution processing unit 412 based on the determination result. In this case, the convolution processing unit 407 can obtain the necessary pixel data and filter coefficients at a timing corresponding to the determined processing time based on the control signal from the control unit 401, and obtain the result of the convolution operation. By using multiple reference patterns, the probability of omitting a product-sum operation can be increased for feature images with various distributions.
[0087] The above processing can also be applied when the size and filter size of the feature image blocks are different. FIG. 8(A) shows an example in which the previous layer is Layer 2 (filter size 5×5). In this example, the size of the feature image blocks of the input feature image is adjusted so that the size of the feature image blocks of the output feature image is 3×3. Specifically, if the filter size is M×M and the size of the feature image blocks of the output feature image is Y×Y, the size of the feature image blocks of the input feature image can be set to (Y+M−1)×(Y+M−1). Note that the size of the feature image blocks of the output feature image can be set to be equal to or less than the number of convolution processing units 412. In this case, pixel data for each pixel of the feature image blocks of the output feature image can be calculated in parallel by multiple convolution processing units 412.
[0088] In the filter processing shown in FIG. 8(A), a 5×5 filter is used, and therefore the kernel size is 5×5. The input feature image 501 is divided into multiple feature image blocks 801, which are processed sequentially. Each feature image block 801 contains 7×7 pixel data. Furthermore, a feature image block 812 of the output feature image 502 is obtained by filter processing using the feature image block 801 (and further processing such as activation processing). Each feature image block 812 contains 3×3 pixel data. In FIG. 8(B), a 7×6 reference range 802 is shown as an area surrounded by a dotted line. In this example, if it is determined in S109 that all pixel data within the reference range 802 are zero, a product-sum operation is performed using pixel data from the top line of each kernel in the filter processing. The size of the reference range can also be set to 7×7, 7×5, 7×4, or 7×3.
[0089] FIG. 9A shows an example in which the previous layer is Layer 3 (filter size 7×7). The input feature image 501 is divided into multiple feature image blocks 901, which are processed sequentially. Each feature image block 901 contains 9×9 pixel data. Furthermore, a feature image block 912 of the output feature image 502 is obtained by filtering using the feature image block 801 (and further processing such as activation processing). Each feature image block 912 contains 3×3 pixel data. Also, in FIG. 9B, a 9×8 reference range 902 is shown as an area surrounded by a dotted line. In this example, if it is determined in S109 that all pixel data within the reference range 902 are zero, a product-sum operation is performed using pixel data from the top line of each kernel in the filtering process. The size of the reference range can also be set to 9×9, 9×7, 9×6, 9×5, 9×4, or 9×3.
[0090] As explained above, by determining whether or not data within the reference range is a zero value, it becomes possible to omit some product-sum operations in the filter processing, regardless of the size of the filter.
[0091] In this embodiment, the reference range corresponding to the processing target block may be determined in advance, or the reference range setting unit 406 may determine the reference range. For example, the reference range setting unit 406 can determine the reference range based on the size of the processing target block and the filter size. As an example, the reference range setting unit 406 can determine, as the reference range, the area excluding the top row among the feature image blocks of the input feature image. When the data within this reference range is a zero value, in the filter process, similar to FIGS. 6(A) to (C), the sum-of-products operation using the top row of each kernel can be performed, and the sum-of-products operation using the remaining rows of each kernel can be omitted. Further, the reference range setting unit 406 can determine, as the reference range, the area excluding the top N rows (where N < M in the case of a filter size of M × M) among the feature image blocks of the input feature image. When the data within this reference range is a zero value, in the filter process, the sum-of-products operation using the top N rows of each kernel can be performed, and the sum-of-products operation using the remaining rows of each kernel can be omitted.
[0092] In this specification, the reference range is a rectangular area set below the feature image block, but the method of setting the reference range is not limited to this. For example, the reference range may be the area excluding the leftmost column among the feature image blocks of the input feature image. When the data within this reference range is a zero value, in the filter process, the sum-of-products operation using the leftmost column of each kernel can be performed, and the sum-of-products operation using the remaining columns of each kernel can be omitted.
[0093] Furthermore, as described with reference to Figures 5(B) and 5(C), the larger the reference range, the greater the reduction in the amount of calculations. However, the smaller the reference range, the greater the likelihood that data within the reference range will be zero. Therefore, the reference range setting unit 406 may select a reference range according to the input data array (feature image). For example, the reference range setting unit 406 may set a larger reference range if the frequency of zero values in the feature image is high, or a smaller reference range if the frequency of zero values is low. In this way, the reference range setting unit 406 may select a reference range according to the input data array so as to minimize the processing time or maximize the amount of calculations that can be reduced.
[0094] In the above embodiment, the zero determination unit 409 determines whether the data within the reference range of the target block is zero. Alternatively, whether the data after the activation process is zero may be recorded when calculating the feature image of the previous layer. In this case, the zero determination unit 409 may determine whether the data within the reference range is zero based on this record when calculating the feature image of the next layer. The zero determination unit 409 may also determine whether the data within the reference range is zero when the data array within the target block is stored in the feature data storage unit 405 or the data storage unit 402. For example, the zero determination unit 409 can determine whether the data within the reference range is zero when the feature image is stored in the feature data storage unit 405 or the data storage unit 402. This determination result can be referenced when performing filtering on the target block. For example, the zero determination unit 409 may calculate an area where pixel data is zero based on the position of zero values in the feature image. The zero determination unit 409 may then compare this area with the reference range to determine whether the data within the reference range is zero.
[0095] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.
[0096] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]
[0097] 403: Coefficient storage unit, 405: Feature data storage unit, 406: Reference range setting unit, 407: Convolution processing unit, 408: Data selection unit, 409: Zero determination unit
Claims
1. 1. A computing device that performs a filter process on a data array in a processing target block of a predetermined size, data storage means for storing a data array in the processing target block; a coefficient holding means for holding a weighting coefficient of a filter used in the filtering process; a determination means for determining whether data within a reference range in the target block to be processed, which is set corresponding to the target block, is zero; a processing means for generating a result of a convolution operation between data at each of a plurality of positions in the processing target block and a weighting factor; a control means for controlling whether or not the processing means performs at least a part of a product-sum operation between the data and the weighting coefficient when generating the convolution operation result, according to a result of the determination by the determination means; A computing device comprising:
2. When the determining means determines that the data in the reference range is a zero value, the processing means omits at least a part of the product-sum calculations of the data and the weighting coefficients at each of a plurality of positions in the processing target block.
2. The computing device according to claim 1, wherein:
3. 3. The arithmetic device according to claim 1, wherein one or more reference ranges are set, and at least one of the reference ranges is smaller than the block to be processed.
4. When the determining means determines that the data in the reference range is a zero value, the processing means obtains a convolution operation result in a first process for a position in the processing target block; If the determining means determines that the data in the reference range is not a zero value, the processing means obtains a convolution operation result in a second process for the position in the processing target block.
4. The computing device according to claim 1, wherein the computing device comprises:
5. When the determination means determines that the data in the reference range is a zero value, the processing means acquires a convolution operation result by a common first process for all of a plurality of positions in the processing target block; If the determining means determines that the data in the reference range is not a zero value, the processing means obtains a convolution operation result by a common second process for all of the multiple positions in the processing target block.
5. The computing device according to claim 1, wherein the computing device comprises:
6. 6. The arithmetic device according to claim 4, wherein the first process and the second process are processes for obtaining the convolution operation result by performing a different number of product-sum operations.
7. the first processing is processing for obtaining the convolution operation result by performing a number of product-sum operations that is less than the number of the weighting coefficients, The second process is a process of obtaining the convolution operation result by performing a number of product-sum operations equal to the number of the weighting coefficients.
7. A computing device according to claim 4, wherein the computing device comprises:
8. The determining means determines whether or not the data within each of a plurality of reference ranges is a zero value, If the determining means determines that the data within the specific reference range is a zero value, the processing means obtains the convolution operation result by performing a number of product-sum operations corresponding to the specific reference range for the position within the processing target block.
8. A computing device according to claim 1, characterized in that it comprises:
9. the determining means determines whether or not data in a reference range having the same size as the processing target block is zero; If the determining means determines that the data in the reference range having the same size as the processing target block is a zero value, the processing means does not perform a product-sum operation and acquires a zero value as the result of the convolution operation.
9. The computing device according to claim 8, wherein:
10. 10. The arithmetic device according to claim 1, wherein the control means controls the supply of the data from the data holding means to the processing means in accordance with the result of the determination by the determination means.
11. the processing means has a plurality of product-sum calculation means, 11. The arithmetic device according to claim 1, wherein the plurality of product-sum calculation means are configured to perform product-sum calculations in parallel for a plurality of positions within the processing target block.
12. 12. The arithmetic device according to claim 11, wherein said plurality of product-sum calculation means perform product-sum calculations in parallel between data input thereto and a common weighting coefficient.
13. when the determining means determines that the data in the reference range is not a zero value, the plurality of product-sum calculation means sequentially perform product-sum calculations of the data and the weighting coefficients using each of the plurality of weighting coefficients; 13. The arithmetic device according to claim 12, wherein, when the determination means determines that the data within the reference range is a zero value, the plurality of product-sum operation means perform product-sum operations on the sequential data and weighting coefficients using only some of the plurality of weighting coefficients.
14. 14. The arithmetic device according to claim 1, further comprising: a setting unit that sets the reference range based on a size of the block to be processed and a size of the filter.
15. 15. The arithmetic device according to claim 1, further comprising activation processing means for performing an activation process on the result of the filtering process.
16. 16. The arithmetic device according to claim 1, wherein the determination means determines whether the data in the reference range is a zero value in response to the data array in the processing target block being stored in the data holding means.
17. the computing device is configured to perform processing using a convolutional neural network; the data array is a feature image processed using a convolutional neural network; 17. The computing device according to claim 1, wherein the weighting factors are part of the weighting factors of a convolutional neural network.
18. An information processing method performed by a computing device that performs a filter process on a data array within a processing target block of a predetermined size, comprising: determining whether data within a reference range in the target block to be processed, which is set corresponding to the target block, is zero; a step of controlling whether or not to perform at least a part of a multiply-and-accumulate operation between the data and a weighting coefficient when generating a convolution operation result according to the determination result; generating a convolution operation result of the data and the weighting coefficient at each of a plurality of positions in the target block according to the control; An information processing method comprising:
19. A computer that performs a filter process on a data array in a processing target block of a predetermined size, determining whether data within a reference range in the target block to be processed, which is set corresponding to the target block, is zero; a step of controlling, according to the determination result, whether or not a processing means for generating a convolution operation result of the data and the weighting coefficient at each of a plurality of positions within the processing target block performs at least a part of a multiply-and-accumulate operation of the data and the weighting coefficient when generating the convolution operation result; A program to perform the following.
