Image processing device, image processing method, and image processing program

Abstract

In the present invention, when convolutional processing is carried out, an input feature map serving as an input for the convolutional processing is divided into small regions, and when a feature constituting a small region corresponds to a predetermined feature or a feature of a small region that has been processed in the past, convolutional processing is not carried out for the small region, and the result of the processing for the predetermined feature or the result of the processing carried out in the past is output as the processing result for the small region.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] When performing inference using a Convolutional Neural Network (CNN), the network is composed of multiple layers, and the convolutional layer performs convolutional processing, which involves product-sum operations and activation processing.

[0003] In inference using CNN, the above convolution operations account for the majority of the overall processing load.

[0004] Even when implementing an inference engine using CNN as hardware, the performance of the convolution operation is directly linked to the performance of the entire engine.

[0005] Examples of convolution operations when the kernel size is 3x3 are shown in Figures 25 and 26. Figure 25 shows an example in which a convolution operation is performed on a 3x3 input feature map using a 3x3 kernel. In this example, nine product-sum operations are performed to output a 1x1 output feature map.

[0006] Also, Figure 26 shows an example in which a 3x3 kernel is used to perform a convolution operation on a (W+2)x(H+2) input feature map. In this example, nine product-sum operations are performed repeatedly while moving the kernel over the input feature map, and a WxH output feature map is output.

[0007] To increase throughput, hardware that performs convolutional operations in CNN often divides the input feature map into small regions of a fixed size and provides circuits that can perform multiply-and-accumulate operations on one small region at a time (see Figure 27). Figure 27 shows an example in which a 26x14 input feature map is divided into nine 10x6 small regions, and the calculation circuit performs convolution processing on 32 points (8x4 points) simultaneously using a 3x3 kernel, outputting an 8x4 output feature map. In this example, the dotted portion of the input feature map is one small region, and the calculation circuit performs convolution processing on 32 points simultaneously for each of the nine small regions, outputting a 24x12 output feature map.

[0008] Also, as one method for speeding up calculations, a method is known in which calculations are skipped when the values ​​of a small region in an input feature map are all zero, as shown in FIG. 28 (for example, Non-Patent Document 1). FIG. 28 shows an example in which the size of the output small region is 4×2, the kernel size is 3×3, and 4-bit data representing 0 to 15 is used. In this example, the size of the small region in the input feature map is 6×4, and the values ​​of the small regions represented by dotted lines are all 0. Since the result of a product-sum operation on 0 is 0, there is no need to perform convolution processing in the calculation circuit, and it is possible to skip the convolution processing of that small region. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W. Keckler, William J. Dally, "SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks", arXiv:1708.04485, 23 May 2017 Summary of the Invention [Problem to be solved by the invention]

[0010] Here, if the size of the small region is increased in order to increase throughput, the values ​​of the small region in the input feature map will rarely be all zero, and sufficient speed-up of the calculation cannot be expected. For example, as shown in FIG. 29A, when the size of the small region in the output feature map is 4×2 (when the size of the small region in the input feature map is 6×4), the values ​​of the small region represented by the dotted line in the input feature map will all be zero. On the other hand, as shown in FIG. 29B, when the size of the small region in the output feature map is 8×4 (when the size of the small region in the input feature map is 10×6), the small region represented by the dotted line in the input feature map will contain non-zero values.

[0011] Furthermore, since the size of the small region is directly linked to the computational throughput, it is often difficult to change it.

[0012] The disclosed technology has been made in consideration of the above points, and aims to provide an image processing device, an image processing method, and an image processing program that can speed up processing using a neural network including convolution processing. [Means for solving the problem]

[0013] A first aspect of the present disclosure is an image processing device including a neural network that performs convolution processing on an image, the image processing device including an acquisition unit that acquires a target image to be processed, and a processing unit that processes the target image using the neural network that includes the convolution processing. When performing the convolution processing, the processing unit performs the convolution processing for each small region obtained by dividing an input feature map that serves as the input to the convolution processing. When performing the convolution processing for each small region, if a feature that constitutes the small region corresponds to a predetermined feature or a feature of a small region that has been processed in the past, the processing unit does not perform the convolution processing on the small region, and outputs the result of processing the predetermined feature or the result of processing the small region in the past as the result of processing the small region. The small region that corresponds to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, and the small region that corresponds to the feature of the small region that has been processed in the past is the small region whose difference from the feature of the small region that has been processed in the past is equal to or less than a threshold.

[0014] A second aspect of the present disclosure is an image processing method in an image processing device including a neural network including convolution processing for an image, wherein an acquisition unit acquires a target image to be processed, and a processing unit processes the target image using the neural network including the convolution processing, wherein when the processing unit performs the convolution processing, the processing unit performs the convolution processing for each small region obtained by dividing an input feature map that serves as the input to the convolution processing, and when performing the convolution processing for each small region, if a feature constituting the small region corresponds to a predetermined feature or a feature of a small region processed in the past, the processing unit does not perform the convolution processing for the small region, and outputs a result of processing the predetermined feature or a result of processing the small region in the past as a result of processing the small region, and the small region corresponding to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, and the small region corresponding to the feature of the small region processed in the past is the small region whose difference from the feature of the small region processed in the past is equal to or less than a threshold.

[0015] A third aspect of the present disclosure is an image processing program for causing a computer including a neural network including convolution processing for an image to acquire a target image to be processed and process the target image using the neural network including the convolution processing, wherein when performing the convolution processing, the convolution processing is performed for each small region obtained by dividing an input feature map that is the input to the convolution processing, and when performing the convolution processing for each small region, if a feature constituting the small region corresponds to a predetermined feature or a feature of a small region processed in the past, the convolution processing is not performed for the small region, and the result of processing the predetermined feature or the result of processing the small region in the past is output as the result of processing the small region, and the small region corresponding to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, and the small region corresponding to the feature of the small region processed in the past is the small region whose difference from the feature of the small region processed in the past is equal to or less than a threshold. [Effects of the Invention]

[0016] According to the disclosed technology, it is possible to speed up processing using a neural network including convolution processing. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 10 is a diagram for explaining a small region of an input feature map. [Figure 2] FIG. 10 is a diagram illustrating a method for skipping convolution processing for a small region of an input feature map. [Figure 3] FIG. 10 is a diagram for explaining a small region of an input feature map. [Figure 4A] FIG. 10 is a diagram for explaining a method for determining small regions of the same value. [Figure 4B] FIG. 10 is a diagram for explaining a method for determining that consecutive small regions are the same; [Figure 5] FIG. 10 is a diagram for explaining a method for determining small regions of the same value. [Figure 6]FIG. 10 is a diagram for explaining a method for determining that consecutive small regions are the same; [Figure 7] FIG. 1 is a schematic block diagram of an example of a computer that functions as an image processing apparatus according to the first, second, and third embodiments. [Figure 8] FIG. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment, a second embodiment, and a third embodiment. [Figure 9] FIG. 2 is a block diagram showing the functional configuration of a learning unit of the image processing device according to the first, second, and third embodiments. [Figure 10] FIG. 2 is a block diagram showing the functional configuration of an inference unit of the image processing device according to the first, second, and third embodiments. [Figure 11] 4 is a flowchart showing the flow of a learning process according to the first, second, and third embodiments. [Figure 12] 10 is a flowchart showing the flow of convolution processing in learning processing and image processing according to the first, second, and third embodiments. [Figure 13] 10 is a flowchart showing the flow of a process for determining whether small regions have the same value according to the first embodiment. [Figure 14] 10 is a flowchart showing the flow of a process for determining whether the small regions are consecutive and identical in the first embodiment. [Figure 15] 10 is a flowchart showing the flow of a process for determining whether small regions have the same value according to the first embodiment. [Figure 16] 10 is a flowchart showing the flow of a process for determining whether the small regions are consecutive and identical in the first embodiment. [Figure 17] 1 is a flowchart showing the flow of image processing in the first, second, and third embodiments. [Figure 18] 10 is a flowchart showing the flow of a process for determining whether small regions have the same value according to the second embodiment. [Figure 19] 10 is a flowchart showing the flow of a process for determining whether the small regions are consecutive and identical according to the second embodiment. [Figure 20]10A and 10B are diagrams for explaining a method of performing convolution processing in accordance with a determination result for a small region. [Figure 21] FIG. 10 is a diagram for explaining a method of performing convolution processing on a plurality of small regions in parallel. [Figure 22] FIG. 10 is a diagram for explaining a method for skipping convolution processing for a small region of an input feature map in a modified example. [Figure 23] FIG. 10 is a diagram for explaining a method for skipping convolution processing for a small region of an input feature map in a modified example. [Figure 24] FIG. 10 is a diagram for explaining a method for skipping convolution processing for a small region of an input feature map in a modified example. [Figure 25] FIG. 10 is a diagram illustrating an example of convolution processing. [Figure 26] FIG. 10 is a diagram illustrating an example of convolution processing. [Figure 27] FIG. 10 is a diagram illustrating an example of convolution processing for each small region. [Figure 28] FIG. 10 is a diagram illustrating an example of a small region of an input feature map. [Figure 29A] FIG. 10 is a diagram illustrating an example of a small region of an input feature map. [Figure 29B] FIG. 10 is a diagram illustrating an example of a small region of an input feature map. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0019] <Summary of the Disclosed Technology> In the disclosed technology, after reading input feature map data for a convolutional layer from a RAM (Random Access Memory) or the like, it is determined for each small region whether all features within the small region are the same or whether they are consecutive, identical small regions. Hereinafter, small regions in the input feature map whose interiors all have the same features are referred to as "same small regions." Also, small regions whose features are identical to those of the previous small region are referred to as "consecutive, identical small regions." In FIG. 1, as an example of "same small regions," an example is shown in which all features within the small region surrounded by dotted lines are 4. Also, as an example of "consecutive, identical small regions," an example is shown in which the features within the small region surrounded by thin dashed lines are identical to the features within the small region surrounded by thick dashed lines. Here, whether the small regions are identical is determined because the input image often also has the same value. Also, whether the small regions are consecutive, identical small regions is determined because the input image often has a series of similar regions. For example, suppose the value of the 10 points in the top row of a 10x6 region of the input image is 10, and the value of the 50 points in the bottom three rows is 12. In both cases of regions with the same value and consecutive identical small regions, this applies when the input image is not a complex picture region but a flat, uniform region.

[0020] As shown in Figure 2, an equivalence flag indicating the result of the judgment for each small region as to whether or not the small regions are of the same value, a continuity flag indicating the result of the judgment for each small region as to whether or not the small regions are consecutive and identical, and small region data, which is the data for each small region of the input feature map, are input to an arithmetic circuit that performs convolution processing.

[0021] In the arithmetic circuit, if the subregions to be processed are subregions of the same value or are consecutive, identical subregions, a process skip is performed without performing convolution processing. If the subregions are consecutive and identical, the processing result will be the same as that of the subregion immediately preceding the subregion processed, so it is sufficient to simply output the processing result of the subregion immediately preceding the subregion processed, thereby speeding up processing. Figure 2 shows an example in which the features in subregion 3 and subregion 4 are identical, so a process skip is performed on subregion 4.

[0022] In addition, for small regions with the same value, the processing results are limited. If the feature value is 4 bits, the processing results are 16 patterns, and if the feature value is 8 bits, the processing results are 256 patterns. The processing results for all patterns are pre-calculated and the pre-calculated result table is stored in RAM, and for each layer processing, the pre-calculated result table is read from RAM into the internal memory of the arithmetic circuit. This speeds up processing because processing results can be obtained by simply referencing internal memory without performing convolution processing for small regions with the same value. Figure 2 shows an example in which processing is skipped for small region 2 because it is a small region with the same value, and the processing results stored in the pre-calculated result table when all features are 4 are referenced and output.

[0023] As shown in Figure 2, the output feature map, which is the result of the convolution process by the arithmetic circuit, is written to RAM, etc. The written output feature map is used as the input feature map in the next layer and onwards, and the above-mentioned determination process and convolution process using processing skip are repeated again.

[0024] Here, the size of the small regions processed at one time is proportional to the scale of the calculation circuit. Furthermore, the larger the size of the small regions processed at one time, the higher the throughput of the inference process. However, when the size of the small regions is increased in order to increase the throughput, the number of small regions in the input feature map that have the same value or that are consecutively the same small regions tends to decrease. For this reason, it is often not possible to expect a sufficient increase in calculation speed in terms of throughput in response to an increase in the scale of the calculation circuit.

[0025] For example, as shown in Figure 3, if the size of the subregions in the output feature map is 4x4 instead of 4x2 (if the size of the subregions in the input feature map is 6x6 instead of 6x4), the feature map will be divided into three subregions, but none of the subregions will have the same values ​​inside. Also, none of the subregions will have two consecutive subregions that are the same. Therefore, no processing skip occurs, and three convolution processes are required.

[0026] If the size of the subregions in the output feature map is 4x2, one of the six subregions is the same subregion and one is the same subregion in succession (see Figure 2 above), so two processing skips occur and only four convolution processes are required.

[0027] Increasing the size of the small regions in the output feature map from 4x2 to 4x4 requires doubling the size of the calculation circuit in hardware, but on the other hand, because processing skips no longer occur, the actual throughput only increases by 4 / 3 times. The throughput per calculation circuit decreases by (4 / 3) / 2 = 2 / 3 times. In this way, the skip rate tends to decrease as the size of the small regions increases, which can reduce the effectiveness of speeding up the calculation circuit by processing skips.

[0028] Therefore, in this embodiment, in the convolution processing using a neural network, the processing skip effect is increased to the extent that the effect on the processing accuracy is small, thereby speeding up the calculation.

[0029] Specifically, even in the case of small regions where not all the values ​​inside the small region are the same, or where not all the values ​​inside the small region are continuously the same, if there is little impact on processing accuracy, skipping the convolution process is applied, thereby speeding up calculations even when the size of the small region or the bit depth is large.

[0030] To minimize the impact on processing accuracy, a process is skipped when the difference between the features of the same small area or the features of consecutive small areas is less than a threshold. More specifically, a process is skipped when (1) the difference between each pixel in the small area is greater than the threshold, or (2) the number of pixels with different features in the small area is less than a threshold.

[0031] The case (1) where the difference between pixels in a small region is equal to or less than a threshold value will be described in detail below.

[0032] Regarding the features within a small region, if the accuracy is reduced, the small regions can be regarded as having the same value or as consecutively the same small regions, and processing is skipped.

[0033] In the small area judgment, it is judged whether the small areas have the same value and whether they are consecutive and identical small areas, while allowing for a difference in the lower bits. At this time, the lower bits of the features in the small area to be processed are masked, and then it is judged whether they are all the same value and whether they are consecutive and identical small areas.

[0034] In this case, the setting parameters are set to the number of bits α1 for masking the lower bits when determining whether the small areas have the same value, and the number of bits α2 for masking the lower bits when determining whether the small areas are consecutive and the same.

[0035] In this case, the setting parameters are set to the number of bits α1 for masking the lower bits when determining whether the small areas have the same value, and the number of bits α2 for masking the lower bits when determining whether the small areas are consecutive and the same.

[0036] For example, as shown in FIG. 4A, when α=2, the two least significant bits are ignored, so the two least significant bits "01" in the binary number "1101" (i.e., decimal number "13") are masked and regarded as the binary number "11" (i.e., decimal number "3"). This is equivalent to shifting right by α bits, as in ">>α". Also, 2 α The quotient when divided by is equal to (13%(2 2 )=13%4=3).

[0037] FIG. 4A shows an example in which small regions are determined to have the same value when the difference between each pixel in the small region is equal to or less than a threshold. In this example, α1=2, the kernel size is 3×3, and the output small region size is 4×2 (input 6×4). If α1=0, that is, if the determination process is performed using the values ​​inside the small region as is, the 6×4=24 values ​​are not all the same, so the small regions are not determined to have the same value. On the other hand, if α1=2, the lower two bits are masked, so all 24 values ​​become 3, and the small regions are determined to have the same value, and processing is skipped.

[0038] FIG. 4B shows an example in which consecutive small regions are determined to be the same small region when the difference between each pixel in the small region is equal to or less than a threshold value. In this example, α1=2, the kernel size is 3×3, and the output small region size is 4×2 (the input small region size is 6×4). When α2=0, i.e., when the determination process is performed using the internal values ​​of the small region, the 6×4=24 values ​​do not match at all between the small region to be processed and the immediately preceding small region, and the consecutive small regions are not determined to be the same small region. On the other hand, when α2=2, the lower two bits are masked, so all 24 values ​​match between the small region to be processed and the immediately preceding small region, and the consecutive small regions are determined to be the same small region, and a processing skip is applied.

[0039] The case where the number of pixels different from the feature in the small region is equal to or less than the threshold value will be described in detail below.

[0040] Regarding the features within a small region, if it can be considered that the small regions have the same value except for some pixels or that the small regions are consecutive and identical, processing is skipped.

[0041] In the small region determination, it is determined whether the small regions have the same value or are consecutive and identical small regions, while allowing for deviation of several features within the small regions.

[0042] At this time, the number of pixels that do not have the same value and the number of pixels that do not match the previous small region are counted for the features within the small region. If the count number is less than a threshold, it is determined that the small regions have the same value or are consecutively the same small region, and processing is skipped.

[0043] In addition, as setting parameters, the number of pixels β1 that are not the same value and are allowed when determining whether they are small areas of the same value, and the number of pixels β2 that are not the same as the previous small area and are allowed when determining whether they are consecutive and the same small area, are set.

[0044] Figure 5 shows an example of determining that a small region has the same value when the number of pixels that differ from the feature in the small region is below a threshold. In this example, β1=3, kernel size is 3 × 3, and output small region size is 4 × 2 (input small region size is 6 × 4).

[0045] If β1 = 0, that is, if even one non-equal value is not allowed, then all 6 x 4 = 24 values ​​are not equal, and so the area is not determined to be an equal-value small area. On the other hand, if β1 = 3, up to three non-equal-value pixels are allowed. In Figure 5, there are two non-equal values, "17" and "16," so 2 <= β1, the area is determined to be an equal-value small area, and processing is skipped.

[0046] Figure 6 shows an example of determining that a small region is the same continuous region when the number of pixels that differ from the feature in the small region is equal to or less than a threshold. In this example, β1=3, kernel size is 3 × 3, and output small region size is 4 × 2 (input small region size is 6 × 4).

[0047] If β2=0, that is, if there is no tolerance for even one mismatch between the small area being processed and the immediately preceding small area, then all 6 x 4 = 24 values ​​will not match, and the consecutive small areas will not be determined to be the same.

[0048] On the other hand, when β2=3, the number of mismatched pixels allowed is up to 3. In Figure 6, there are two mismatched pixels between the small area to be processed and the immediately preceding small area: {"25" and "28"} and {"8" and "12"}. This is less than β2, so the small areas are determined to be consecutive and identical, and processing is skipped.

[0049] The number of bits α1, α2 and the number of pixels β1, β2 corresponding to the thresholds are predetermined so that the accuracy of the processing using the neural network is a predetermined accuracy. The number of bits α1, α2 and the number of pixels β1, β2 may be dynamically set during the inference processing.

[0050] [First embodiment] <Configuration of image processing device according to first embodiment> FIG. 7 is a block diagram showing the hardware configuration of the image processing device 10 of the first embodiment.

[0051] 7, the image processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, a communication interface (I / F) 17, and an arithmetic circuit 18. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0052] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores a learning processing program for performing learning processing on a neural network and an image processing program for performing image processing using a neural network. The learning processing program and the image processing program may be a single program, or may be a group of programs consisting of multiple programs or modules.

[0053] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0054] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0055] The input unit 15 receives, as input, training data for training the neural network. For example, the input unit 15 receives, as input, training data including a target image to be processed and a processing result for the target image that has been obtained in advance.

[0056] Furthermore, the input unit 15 receives as input a target image to be processed.

[0057] The display unit 16 is, for example, a liquid crystal display, and displays various information including the processing results. The display unit 16 may be a touch panel type and function as the input unit 15.

[0058] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0059] The arithmetic circuit 18 executes convolution processing in the convolution layer of the neural network. Specifically, the arithmetic circuit 18 receives a subregion of an input feature map and a kernel, and outputs a subregion of an output feature map that is the result of the convolution processing of each point in the subregion.

[0060] Next, a description will be given of the functional configuration of the image processing device 10. Fig. 8 is a block diagram showing an example of the functional configuration of the image processing device 10.

[0061] Functionally, the image processing device 10 includes a learning unit 20 and an inference unit 22, as shown in FIG.

[0062] As shown in FIG. 9, the learning unit 20 includes an acquisition unit 30, a processing unit 32, and an update unit .

[0063] The acquisition unit 30 acquires the target image and processing results of the input learning data.

[0064] The processing unit 32 processes the target image using a neural network that includes convolution processing. When performing convolution processing, the processing unit 32 performs the convolution processing for each small region obtained by dividing an input feature map that serves as the input for the convolution processing. The convolution processing for each small region is performed using the arithmetic circuit 18. At this time, small region data and a kernel of the input feature map are input to the arithmetic circuit 18, and an output feature map that represents the result of the convolution processing for each small region is output from the arithmetic circuit 18.

[0065] Here, the input feature map is divided into small regions as shown in Figure 1. In Figure 1, regions corresponding to small regions in the output feature map are indicated by bold frames. A small region in the input feature map includes the corresponding small region in the output feature map and the overlapping regions where adjacent small regions overlap. In this way, the small regions divided into the input feature map overlap with adjacent small regions, and the overlapping regions correspond to the kernel size of the convolution processing. The division of the small regions is determined at the design stage. For example, it can be determined by balancing hardware constraints (size and power consumption) with the desired throughput (e.g., xx fps). If the kernel size is 3 × 3, the input feature map has 10 × 6 small regions, and the output feature map has 8 × 4 small regions, a throughput of 32 simultaneous output points can be achieved. In this case, 288 multiply-accumulate circuits (= (3 × 3) × 8 × 4) are required. Furthermore, if the input feature map subregions are 6x4 and the output feature map subregions are 4x2, the throughput will be such that only 8 points are output simultaneously. In this case, 72 multiply-accumulate circuits (=(3x3)x4x2) are required. Furthermore, when implementing multiply-accumulate operations in hardware, the division of the subregions is determined at the time of design, so the division of the subregions does not change depending on the layer.

[0066] When performing convolution processing for each small region, if all features constituting the small region correspond to a small region with the same value, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for a predetermined case where all features constituting the small region have the same value as the result of processing the small region.

[0067] Specifically, for each small region, the processing unit 32 determines whether all the features constituting the small region correspond to a small region of the same value. If it is determined that all the features constituting the small region correspond to a small region of the same value, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for the case where all the features constituting the small region are the same, which is stored in the pre-computation result table, as the result of processing the small region.

[0068] More specifically, for each small region, if the difference between the features in the small region of the same value and those in the same small region is equal to or less than a threshold, the processing unit 32 determines that all of the features constituting the small region correspond to the small region of the same value. For example, if the bits other than the lower-order predetermined number α1 of bits representing the features in the small region to be processed are identical, the processing unit 32 determines that all of the features constituting the small region correspond to the small region of the same value.

[0069] Furthermore, for each small region, if the number of pixels that differ from the features in the small region of the same value is equal to or less than a threshold value, the processing unit 32 determines that all features constituting the small region correspond to the small region of the same value. For example, the processing unit 32 counts the number of pixels with different features compared to the small region of the same value, and if the counted number of pixels is equal to or less than a threshold value β1, it determines that all features constituting the small region correspond to the small region of the same value.

[0070] Furthermore, when performing convolution processing for each small region, if the features constituting the small region correspond to the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the results of processing the small region processed immediately before as the results of processing the small region.

[0071] Specifically, for each small region, the processing unit 32 determines whether the features constituting the small region correspond to the features of the small region processed immediately before. If it is determined that the features constituting the small region correspond to the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the results of processing the immediately previous small region as the results of processing the small region.

[0072] More specifically, for each small region, if the difference between the features of the small region processed immediately before and the features of the small region processed immediately before is equal to or less than a threshold, the processing unit 32 determines that the features of the small region correspond to the features of the small region processed immediately before. For example, if bits other than the lower-order predetermined number α2 bits representing the features of the small region processed immediately before are identical to bits other than the lower-order predetermined number α2 bits representing the features of the small region, the processing unit 32 determines that the features of the small region correspond to the features of the small region processed immediately before.

[0073] Furthermore, for each small region, if the number of pixels that differ from the features in the small region processed immediately before is equal to or less than a threshold, the processing unit 32 determines that the features constituting the small region correspond to the features of the small region processed immediately before. For example, the processing unit 32 counts the number of pixels that differ from the features in the small region processed immediately before, and if the counted number of pixels is equal to or less than a threshold β2, it determines that the features constituting the small region correspond to the features of the small region processed immediately before.

[0074] The update unit 34 updates the parameters of the neural network so that the result of processing the target image using the neural network matches the pre-calculated processing result. The update unit 34 also updates the pre-calculation result table of each convolution layer based on the updated neural network parameters.

[0075] The processes of the processing unit 32 and the updating unit 34 are repeated until a predetermined iteration end condition is met, thereby training the neural network.

[0076] As shown in FIG. 10, the inference unit 22 includes an acquisition unit 40 and a processing unit 42.

[0077] The acquisition unit 40 acquires the input target image to be processed.

[0078] The processing unit 42 processes the target image using a neural network that includes convolution processing. When performing convolution processing, the processing unit 42 performs the convolution processing for each small region obtained by dividing an input feature map that serves as the input for the convolution processing. The convolution processing for each small region is performed using the arithmetic circuit 18. At this time, small region data and a kernel of the input feature map are input to the arithmetic circuit 18, and an output feature map representing the result of the convolution processing for each small region is output from the arithmetic circuit 18.

[0079] The processing unit 42 determines, for each small region, whether all the features that make up the small region correspond to small regions with the same value, similar to the processing unit 32. If it is determined that all the features that make up the small region correspond to small regions with the same value, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the results of processing for cases where all the features that make up the small region are the same, which are stored in the pre-computation result table, as the results of processing the small region.

[0080] Furthermore, processing unit 42 determines whether the features constituting the relevant small region correspond to the features of the sub-region processed immediately before, similar to processing unit 32. If it is determined that the features constituting the relevant small region correspond to the features of the sub-region processed immediately before, arithmetic circuit 18 does not perform convolution processing on the relevant small region, and outputs the results of processing the previously processed sub-region as the results of processing the relevant small region.

[0081] The result of processing the target image using the neural network is displayed on the display unit 16.

[0082] <Action of the image processing device according to the first embodiment> Next, the operation of the image processing device 10 according to the first embodiment will be described.

[0083] 11 is a flowchart showing the flow of the learning process by the image processing device 10. The learning process is performed by the CPU 11 reading out a learning process program from the ROM 12 or the storage 14, expanding it into the RAM 13, and executing it. In addition, learning data is input to the image processing device 10.

[0084] In step S100, the CPU 11 functions as the acquisition unit 30 to acquire a target image to be processed from the input learning data and the processing result.

[0085] In step S102, the CPU 11 processes the target image of the learning data using a neural network including convolution processing as the processing unit 32.

[0086] In step S104, the CPU 11, as the update unit 34, updates the parameters of the neural network so that the results of processing the target image of the learning data using the neural network match the processing results obtained in advance, and updates the pre-calculation result table.

[0087] In step S106, the CPU 11 determines whether a predetermined iteration end condition is satisfied. If the iteration end condition is not satisfied, the process returns to step S102, and the processes of the processing unit 32 and the update unit 34 are repeatedly performed. This allows the neural network to learn.

[0088] In step S102, the arithmetic processing of each layer of the neural network is performed. Here, the arithmetic processing of the convolution layer is realized by the processing routine shown in FIG.

[0089] In step S110, the CPU 11 as the processing unit 32 divides the input feature map, which is the input of the convolution layer, into small regions.

[0090] In step S112, the CPU 11, functioning as the processing unit 32, reads out the pre-computation result table of the convolution layer from the RAM 13.

[0091] In step S114, the CPU 11, as the processing unit 32, processes the divided small areas in order and determines whether the features constituting the small area to be processed correspond to those of an equivalent small area or to those of the small area processed immediately before.

[0092] Specifically, if the difference between the feature and the feature in the small area of ​​the same value is equal to or less than a threshold, the CPU 11 determines that the feature constituting the small area to be processed corresponds to the small area of ​​the same value.

[0093] Furthermore, if the number of pixels that differ from the feature in the small area of ​​the same value is equal to or less than the threshold, the CPU 11 determines that the feature that constitutes the small area to be processed corresponds to the small area of ​​the same value.

[0094] Furthermore, if the difference between the features in the small region processed immediately before and the features in the small region processed immediately before is equal to or less than a threshold, the CPU 11 determines that the features constituting the small region to be processed correspond to the features of the small region processed immediately before.

[0095] Furthermore, if the number of pixels in the small region processed immediately before that differ from the features of the small region processed immediately before is equal to or less than a threshold, the CPU 11 determines that the features constituting the small region to be processed correspond to the features of the small region processed immediately before.

[0096] In step S116, the CPU 11, functioning as the processing unit 32, outputs each small region data of the input feature map, the pre-computation result table, and the equivalence flag and continuity flag indicating the determination result of step S114 to the arithmetic circuit 18. The arithmetic circuit 18 then performs convolution processing for each small region. At this time, if the small region to be processed does not correspond to a small region with the same value and does not correspond to consecutive identical small regions, the arithmetic circuit 18 performs convolution processing on the small region to be processed. If the small region to be processed corresponds to a small region with the same value, the arithmetic circuit 18 does not perform convolution processing on the small region to be processed, and outputs the result of processing for cases where the features constituting the small region are equivalent, stored in the pre-computation result table, as the result of processing the small region to be processed.

[0097] Furthermore, if the small area to be processed corresponds to the same consecutive small area, the arithmetic circuit 18 does not perform convolution processing on the small area to be processed, but outputs the processing result for the small area processed immediately before as the processing result for the small area to be processed.

[0098] Then, the processing routine is terminated, and an output feature map consisting of the processing results for each small region is output and used as the input feature map for the next layer.

[0099] In step S114, the divided small regions are processed in order, and it is determined whether the characteristics constituting the small region to be processed correspond to those of a small region of the same value or to those of the small region processed immediately before. Here, the process of determining that a small region corresponds to a small region of the same value when the difference between pixels of the small region is equal to or less than a threshold is realized by the processing routine shown in FIG.

[0100] In step S130, the CPU 11, functioning as the processing unit 32, performs a masking process to mask a predetermined number α1 of lower-order bits for the features in the small region to be processed.

[0101] In step S132, the CPU 11, as the processing unit 32, determines whether the features constituting the small area to be processed correspond to a small area of ​​the same value, based on whether all the features in the small area to be processed after the masking process in step S130 are of the same value.

[0102] Furthermore, when the difference between pixels in a small region is equal to or smaller than a threshold value, the process of determining that the consecutive pixels correspond to the same small region is realized by the processing routine shown in FIG.

[0103] In step S140, the CPU 11, functioning as the processing unit 32, performs a masking process to mask the lower predetermined number α2 of bits for the features in the small region to be processed.

[0104] In step S142, the CPU 11, as the processing unit 32, determines whether the features constituting the small area to be processed correspond to the small area processed immediately before, based on whether the features within the small area to be processed after the masking process in step S140 are identical to the features within the small area processed immediately before, after the masking process.

[0105] Furthermore, when the number of pixels in a small area that differ from the feature is equal to or less than a threshold, the process of determining that the small area corresponds to a small area of ​​the same value is realized by the processing routine shown in FIG.

[0106] In step S150, the CPU 11 as the processing unit 32 compares the small area to be processed with small areas of the same value and counts the number of pixels having different characteristics.

[0107] In step S152, the CPU 11, as the processing unit 32, determines whether the features constituting the small area to be processed correspond to a small area of ​​the same value, based on whether the number of pixels counted in step S150 is less than or equal to the threshold value β1.

[0108] Furthermore, when the number of pixels in a small region that differ from the feature is equal to or less than a threshold, the process of determining that the pixels correspond to the same continuous small region is realized by the processing routine shown in FIG.

[0109] In step S160, the CPU 11 as the processing unit 32 counts the number of pixels in the small area to be processed that have different characteristics from those in the small area processed immediately before.

[0110] In step S162, the CPU 11, as the processing unit 32, determines whether the features constituting the small area to be processed correspond to those of the small area processed immediately before, based on whether the number of pixels counted in step S160 is less than or equal to the threshold value β2.

[0111] 17 is a flowchart showing the flow of image processing by the image processing device 10. Image processing is performed by the CPU 11 reading out an image processing program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. A target image is input to the image processing device 10.

[0112] In step S120, the CPU 11 functions as the acquisition unit 40 to acquire the input target image.

[0113] In step S122, the CPU 11 processes the target image using the neural network learned by the above-described learning process as the processing unit 42. Then, the result of processing the target image using the neural network is displayed on the display unit 16.

[0114] In step S122, the arithmetic processing of each layer of the neural network is performed. Here, the arithmetic processing of the convolution layer is realized by the processing routine shown in FIG.

[0115] As described above, when performing convolution processing for each small region of an input feature map, if the features constituting the small region correspond to a small region of the same value or to the features of the small region processed immediately before, the image processing device according to the first embodiment does not perform convolution processing on the small region, and outputs the results of processing a predetermined small region of the same value or the results of the immediately preceding processing as the results of processing the small region. This enables the speed of processing using a neural network including convolution processing to be increased.

[0116] When increasing the size of the subregions into which an input feature map is divided to improve throughput, or when increasing the bit depth representing the input feature map to improve CNN calculation accuracy, conventional methods often fail to speed up calculations because the interior of the subregion rarely becomes all zero. For example, when the features of the input feature map are represented by 4-bit data, the interior of the subregion becomes all zero, whereas when the features of the input feature map are represented by 6-bit data, the interior of the subregion does not become all zero. On the other hand, in this embodiment, sufficient calculation speedup can be expected compared to conventional methods for subregions of the same value or subregions corresponding to consecutive identical subregions.

[0117] The number of masked bits, α1 and α2, and the upper limit of the number of pixels with different features, β1 and β2, are the setting parameters. The larger the parameter value, the higher the skip rate and the easier it is to speed up, while the smaller the parameter value, the less impact it has on the accuracy of CNN calculations.

[0118] It is possible to determine the values ​​suitable for the neural network through preliminary experiments, etc. Through preliminary experiments, it is possible to determine α1, α2, β1, and β2 that can achieve a certain level of CNN calculation accuracy and often result in the highest skip performance.

[0119] Alternatively, it is possible to feedback changes in CNN calculation accuracy and adaptively change parameter values ​​within a range that has little effect on CNN calculation accuracy. For example, if a threshold for CNN calculation accuracy is set and the accuracy falls below that threshold during inference processing, α1, α2, β1, and β2 can be reduced by one step to adaptively improve CNN calculation accuracy.

[0120] The above processing makes it possible to significantly improve the skip rate while keeping the impact on CNN calculation accuracy below a certain value, which is expected to speed up the entire CNN calculation process.

[0121] [Second embodiment] Next, a second embodiment will be described. Note that, since the configuration is similar to that of the first embodiment, the same reference numerals are used and the description will be omitted.

[0122] The second embodiment differs from the first embodiment in that, in the small area determination, after mask processing, the number of pixels with different characteristics within the small area is counted, and based on whether the counted number of pixels is below a threshold, it is determined whether the pixels correspond to small areas of the same value or consecutive identical small areas.

[0123] <Outline of the second embodiment> In the second embodiment, both of the following cases are considered: (1) when the difference between pixels in a small region is not equal to a threshold value, and (2) when the number of pixels with different characteristics in the small region is equal to or less than a threshold value. Specifically, the setting parameters α1, α2, β1, and β2 are set, and the masking of the lower bits and the counting of the number of pixels with different characteristics in the small region are performed simultaneously.

[0124] As a result, if the difference between pixels is equal to or less than the threshold, any number of pixels is allowed, and the number of pixels where the difference between pixels is equal to or greater than the threshold is allowed to be equal to or less than a certain number β1, β2, and processing is skipped under this condition.

[0125] <Configuration of image processing device according to second embodiment> For each small region, the processing unit 32 determines whether all the features constituting the small region correspond to a small region of the same value. If it is determined that all the features constituting the small region correspond to a small region of the same value, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for the case where all the features constituting the small region are the same, which is stored in the pre-computation result table, as the result of processing the small region.

[0126] Specifically, for each small region, if the difference between the feature and the feature in the small region of the same value is less than a threshold, the processing unit 32 considers the feature to be the same, and if the number of pixels that differ from the feature in the small region of the same value is less than a threshold, determines that all of the features that make up the small region correspond to the small region of the same value. For example, if the bits other than the lower-order predetermined number α1 that represent the feature in the small region to be processed are the same, the processing unit 32 considers the feature to be the same, compares it with the small region of the same value, and counts the number of pixels with different features. If the counted number of pixels is less than a threshold β1, it determines that all of the features that make up the small region correspond to the small region of the same value.

[0127] Furthermore, for each small region, the processing unit 32 determines whether the features constituting the small region correspond to the features of the small region processed immediately before. If it is determined that the features constituting the small region correspond to the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the results of processing the immediately previous small region as the results of processing the small region.

[0128] Specifically, for each small area, if the difference between the features in the small area processed immediately before and the features in the small area processed immediately before is less than a threshold, the processing unit 32 considers the features to be the same, and if the number of pixels that differ from the features in the small area processed immediately before is less than a threshold, it determines that the features that make up the small area correspond to the features of the small area processed immediately before.

[0129] For example, if the bits other than the lower-order predetermined number α2 of bits representing the characteristics within the small region to be processed are identical, they are considered to be the same characteristics, and the number of pixels with different characteristics is counted compared to the small region processed immediately before.If the counted number of pixels is less than or equal to threshold β2, it is determined that all of the features that make up the small region correspond to the features of the small region processed immediately before.

[0130] <Action of the image processing device according to the second embodiment> The computational processing of the convolutional layer is realized by the processing routine shown in Fig. 12. Here, in step S114, the CPU 11, as the processing unit 32, processes the divided small regions in order, and determines whether the features constituting the small region to be processed correspond to those of a small region of the same value, or to those of the small region processed immediately before.

[0131] Specifically, if the difference between the features in a small area of ​​the same value is less than a threshold, the CPU 11 considers the features to be the same, and if the number of pixels that differ from the features in the small area of ​​the same value is less than a threshold, it determines that the features that make up the small area to be processed correspond to the small area of ​​the same value.

[0132] Furthermore, if the difference between the features in the small area processed immediately before and the features in the small area processed immediately before is less than a threshold, the CPU 11 considers them to be the same features, and if the number of pixels that differ from the features in the small area processed immediately before is less than a threshold, it determines that the features that make up the small area to be processed correspond to the features of the small area processed immediately before.

[0133] In step S114, the divided small regions are processed in order, and it is determined whether the features constituting the small region to be processed correspond to those of a small region of the same value or to those of the small region processed immediately before. Here, if the difference between the pixels of the small regions is equal to or less than a threshold, the small regions are considered to have the same features, and if the number of pixels in the small region that differ from the features is equal to or less than a threshold, it is determined that the small region corresponds to a small region of the same value. This process is realized by the processing routine shown in FIG.

[0134] In step S130, the CPU 11, functioning as the processing unit 32, performs a masking process to mask a predetermined number α1 of lower-order bits for the features in the small region to be processed.

[0135] In step S200, the CPU 11 as the processing unit 32 compares the small region to be processed after the mask processing with a small region of the same value, and counts the number of pixels having different characteristics.

[0136] In step S202, the CPU 11, as the processing unit 32, determines whether the features constituting the small area to be processed correspond to a small area of ​​the same value, based on whether the number of pixels counted in step S200 is less than or equal to the threshold value β1.

[0137] In addition, if the difference between pixels in a small region is below a threshold, the feature is considered to be the same, and if the number of pixels in the small region that differ from the feature is below a threshold, the consecutive pixels are determined to correspond to the same small region.This process is realized by the processing routine shown in Figure 19.

[0138] In step S140, the CPU 11, functioning as the processing unit 32, performs a masking process to mask the lower predetermined number α2 of bits for the features in the small region to be processed.

[0139] In step S210, the CPU 11 as the processing unit 32 counts the number of pixels having different characteristics in the small region to be processed after the mask processing compared with the small region processed immediately before after the mask processing.

[0140] In step S212, the CPU 11, as the processing unit 32, determines whether the features constituting the small area to be processed correspond to those of the small area processed immediately before, based on whether the number of pixels counted in step S210 is less than or equal to the threshold value β2.

[0141] Next, an example of the computational processing of the convolution layer will be shown using FIG. 20.

[0142] First, one small region is given as an input feature map (S220). Figure 20 shows the case of 8-bit precision, kernel size 3x3, and output small region size 4x2. In this case, the size of the small region to be processed is 6x4. 24 features in the small region are f t Expressed as (t:0~23).

[0143] Then, a masking process is performed to determine whether the subregions are of the same value (S222).t =f t ≫Feature f' in the small region after mask processing as α1 t Calculate.

[0144] Then, a counting process is performed to determine whether the small regions have the same value (S224). For the small regions after the masking process, the first value f'0 is set to x, and if the value of f' is not equal to x, n Count the number of items and call it y.

[0145] If y≦β1, and bits other than the predetermined number α1 of lower bits representing the feature in the small area to be processed are the same, the small area is regarded as having the same feature, and is compared with a small area of ​​the same value. If the number of pixels with different features is determined to be equal to or less than the threshold β1, the small area is determined to be an equivalent small area (S226). An equivalent flag is set to 1.

[0146] If the equivalence flag is 1, the process is skipped (S228). x (= f'0) is the representative value of the small region, and the result of the convolution process corresponding to x stored in the pre-computation result table can be used as is as the output data.

[0147] For example, if a1=4, f' t is equivalent to 8-4=4 bits, so f' t can only take on 16 levels of values. The results of the convolution process, which indicate what the output will be in each case, are calculated in advance and saved as a pre-calculated result table.

[0148] If the input is a small area of ​​the same value, all internal values ​​will be the same as the result of the convolution process, so it is only necessary to store one value for each of the 16 input values. In this case, the output data for the representative input value x is taken as dx, and all eight values ​​in the 4 x 2 size of the convolution process result are filled with dx to obtain the result of the convolution process, allowing the calculation result to be obtained by simply reading the data without actually performing a multiply-and-accumulate operation.

[0149] In addition, a mask process is performed to determine whether the small regions are consecutive and identical (S230). t =f t ≫Feature f'' in the small region after mask processing as α2 t Calculate.

[0150] In addition, the values ​​after masking (6 x 4 = 24 values) are saved so that they can be used for the next small region (S232). t (t:0~23).

[0151] A counting process is performed to determine whether the small regions are consecutive and identical (S234). t and the feature g'' of the previous small region t For each pixel, the number of pixels that do not have the same value is counted and set to z.

[0152] If z≦β2, and bits other than the predetermined number α2 of lower bits representing the characteristics of the small region to be processed are the same, the small region is regarded as having the same characteristics, and is compared with a small region of the same value. If the number of pixels with different characteristics is determined to be equal to or less than the threshold β2, the small region is determined to be the same continuous small region (S236). A contiguous flag is set to 1.

[0153] If the continuity flag is 1, processing is skipped (S238). The result of the previous convolution process is used as is. The result of the previous convolution process (4 x 2 = 8 values) is saved each time, and when the continuity flag is 1, that data can be read to obtain the result of the convolution process, making it possible to obtain the calculation result just by reading the data without actually performing a product-sum operation.

[0154] If both the equality flag and the consecutive flag are 0, the process cannot be skipped, and the result of the convolution process is obtained by performing a product-sum operation in the arithmetic circuit 18 as usual.

[0155] The other configurations and operations of the image processing device according to the second embodiment are the same as those of the first embodiment, and therefore will not be described.

[0156] As described above, when performing convolution processing for each small region of an input feature map, if the features constituting the small region correspond to a small region of the same value or to the features of the small region processed immediately before, the image processing device according to the second embodiment does not perform convolution processing on that small region, and outputs the results of processing a predetermined small region of the same value or the results of the immediately preceding processing as the results of processing that small region. This enables the speed of processing using a neural network including convolution processing to be increased.

[0157] Furthermore, in small region determination, after masking, the number of pixels in the small region with different features is counted, and based on whether the counted number of pixels is equal to or less than a threshold, it is determined whether the small regions correspond to the same value or consecutive identical small regions, thereby reducing the amount of calculation circuitry for comparison and internal memory required for data storage. For example, in an 8-bit feature map, if the small region determination is also made 8 bits, or if the small region determination is made by masking the lower bits, the number of AND circuits that determine whether each bit matches will be reduced when determining whether the small regions are the same value and whether the small regions are consecutive identical small regions.

[0158] In addition, to determine whether or not a subregion has the same value, it is necessary to store the results for each value as a table. For example, in an 8-bit feature map, if the determination of whether or not a subregion has the same value is also made 8 bits, the input value for the subregion with the same value can take 256 levels from 0 to 255. Furthermore, if the determination of whether or not a subregion has the same value is made 4 bits, the input value for the subregion with the same value can take 16 levels from 0 to 15, which makes it possible to significantly reduce the amount of memory required for the table for determining whether or not a subregion has the same value.

[0159] [Third embodiment] The third embodiment differs from the first embodiment in that the arithmetic circuit performs convolution processing on a plurality of small regions in parallel.

[0160] <Outline of the third embodiment> In the third embodiment, the arithmetic circuit performs convolution processing in parallel on multiple small regions of an input feature map, as shown in Fig. 21. Fig. 21 shows an example in which the arithmetic circuit sets the size of the small regions of the input feature map to 6 × 4, performs convolution processing on four small regions in parallel, and simultaneously outputs the results of the convolution processing on 32 points.

[0161] At this time, it is determined for each small region whether they are small regions of the same value or whether they are consecutive and identical small regions. In the example of Fig. 21, small region 1 is a small region of the same value, and small region 3 is a consecutive and identical small region, so the convolution process for small region 1 and small region 3 is skipped. Furthermore, convolution process is performed in parallel on small region 0, small region 2, small region 4, and small region 5, and as a result, processing results for six small regions are obtained at one time.

[0162] In this way, for each small region, a determination is made as to whether the small regions are of the same value or whether they are consecutive, identical small regions, and by using an arithmetic circuit to perform convolution processing on multiple small regions in parallel, the probability that the small regions are of the same value and the probability that they are consecutive, identical small regions is increased, thereby increasing the probability that the convolution processing will be skipped.

[0163] <Configuration of image processing device according to the third embodiment> An image processing apparatus according to the third embodiment will now be described. Portions having the same configuration as those in the first embodiment will be given the same reference numerals and descriptions thereof will be omitted.

[0164] The hardware configuration of the image processing device 210 of the third embodiment is similar to the hardware configuration of the image processing device 10 shown in FIG.

[0165] The arithmetic circuit 18 of the image processing device 210 receives small region data of the input feature map and a kernel, repeatedly performs convolution processing on a predetermined number of small regions in parallel, and outputs an output feature map, which is the result of the convolution processing of each point in each small region.

[0166] The processing unit 32 of the learning unit 20 processes a target image using a neural network that includes convolution processing. When performing convolution processing, the processing unit 32 performs the convolution processing for each small region obtained by dividing an input feature map that serves as the input for the convolution processing. The convolution processing for each small region is performed using the arithmetic circuit 18. At this time, the processing unit 32 determines, for each small region, whether the small regions have the same value and whether the small regions are consecutive and identical. The small region data and kernel of the input feature map, the determination results for each small region, and the pre-computation result table are input to the arithmetic circuit 18, and an output feature map is output from the arithmetic circuit 18.

[0167] Specifically, when performing convolution processing, the processing unit 32 determines, for each small region, whether all of the features that make up the small region correspond to small regions with the same value. If it is determined that all of the features that make up the small region correspond to small regions with the same value, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for the case where all of the features that make up the small region are the same, which is stored in the pre-computation result table, as the result of processing the small region.

[0168] Furthermore, for each small region, the processing unit 32 determines whether the features constituting the small region correspond to the features of the small region processed immediately before. If it is determined that the features constituting the small region correspond to the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the results of processing the immediately previous small region as the results of processing the small region.

[0169] The arithmetic circuit 18 also performs convolution processing in parallel on a predetermined number of small regions that do not correspond to small regions in which all of the features constituting the small regions have the same value and do not correspond to the features of the small region processed immediately before.

[0170] The output feature map, which is the result of the convolution process performed for each small region as described above, becomes the input for the next layer of the neural network.

[0171] Similar to the processing unit 32, the processing unit 42 of the inference unit 22 processes the target image using a neural network including convolution processing. When performing convolution processing, the processing unit 42 performs the convolution processing for each small region obtained by dividing the input feature map that serves as the input for the convolution processing. The convolution processing for each small region is performed using the arithmetic circuit 18. At this time, the processing unit 42 determines, for each small region, whether or not the small regions correspond to small regions of the same value, and whether or not they correspond to consecutive, identical small regions. The small region data and kernel of the input feature map, the determination results for each small region, and the pre-computation result table are input to the arithmetic circuit 18, and an output feature map is output from the arithmetic circuit 18.

[0172] Specifically, when performing convolution processing, processing unit 42 determines for each small region whether all features constituting the small region correspond to small regions with the same value, similar to processing unit 32. If it is determined that all features constituting the small region correspond to small regions with the same value, arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for the case where all features constituting the small region are the same, which is stored in the pre-computation result table, as the result of processing the small region.

[0173] Similarly to processing unit 32, processing unit 42 determines for each small region whether the features constituting that small region correspond to the features of the small region processed immediately before. If it is determined that the features constituting that small region correspond to the features of the small region processed immediately before, arithmetic circuit 18 does not perform convolution processing on that small region, and outputs the results of processing the immediately previous small region as the results of processing that small region.

[0174] The arithmetic circuit 18 also performs convolution processing in parallel on a predetermined number of small regions that do not correspond to small regions in which all of the features constituting the small regions have the same value and do not correspond to the features of the small region processed immediately before.

[0175] The output feature map, which is the result of the convolution process performed for each small region as described above, becomes the input for the next layer of the neural network.

[0176] <Action of the image processing device according to the third embodiment> Next, the operation of the image processing device 10 according to the third embodiment will be described.

[0177] The CPU 11 of the image processing device 210 reads out the learning processing program from the ROM 12 or the storage 14, expands it in the RAM 13, and executes it, thereby performing processing similar to the learning processing shown in Fig. 11. Furthermore, learning data is input to the image processing device 10.

[0178] In step S102, the arithmetic processing of each layer of the neural network is performed. Here, the arithmetic processing of the convolution layer is realized by the processing routine shown in FIG.

[0179] In step S116, if the features constituting the small region to be processed correspond to small regions of the same value, the arithmetic circuit 18 does not perform convolution processing on the small region to be processed, and outputs the results of processing for the case where the features constituting the small region are of the same value, stored in the pre-computation result table, as the result of processing the small region to be processed.

[0180] Furthermore, if the features constituting the small area to be processed correspond to the features of the small area processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small area to be processed, but outputs the results of processing on the small area processed immediately before as the results of processing the small area to be processed.

[0181] The arithmetic circuit 18 also performs convolution processing in parallel on a predetermined number of small regions whose features do not correspond to small regions of the same value and do not correspond to the features of the small region processed immediately before.

[0182] The CPU 11 of the image processing device 210 reads out an image processing program from the ROM 12 or the storage 14, expands it in the RAM 13, and executes it, thereby performing processing similar to the image processing shown in Fig. 17. Furthermore, a target image is input to the image processing device 210.

[0183] In step S122, the arithmetic processing of each layer of the neural network is performed. Here, the arithmetic processing of the convolution layer is realized by the processing routine shown in FIG.

[0184] The other configurations and operations of the image processing device 210 of the third embodiment are the same as those of the first embodiment, and therefore the description thereof will be omitted.

[0185] As described above, the image processing device according to the third embodiment configures the arithmetic circuit to perform convolution processing on multiple small regions in parallel, and when performing convolution processing on each small region of the input feature map, if the features constituting the small region correspond to those of a small region with the same value or to those of the small region processed immediately before, the image processing device does not perform convolution processing on that small region, and outputs the results of processing on a predetermined small region with the same value or the results of the immediately preceding processing as the results of processing that small region. This makes it possible to expect increased speed by skipping processing without a decrease in calculation accuracy or throughput, even in cases where the bit depth or parallel processing unit is large and skipping convolution processing for small regions with all zeros is not expected.

[0186] The present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

[0187] For example, the above description has been given with reference to an example in which processing results are calculated in advance for all values ​​(e.g., 16 values ​​when the feature is expressed as 4-bit data) for subregions with the same value and stored in memory. However, the present invention is not limited to this. Feature values ​​that are likely to appear as subregions with the same value may be determined in advance through simulation or the like, and processing results for subregions with the same value may be calculated in advance for only the top few feature values ​​with the highest occurrence probability, and stored in a pre-calculated result table (see FIG. 22). For subregions with the same value and processing results that have been calculated in advance, the pre-calculated processing results are output without performing convolution processing. FIG. 22 shows an example in which processing results are calculated in advance only for subregions with the same value whose feature value is 1, subregions with the same feature value is 4, and subregions with the same feature value is 15. Furthermore, for subregions with the same feature value 0, the processing result is 0, so the pre-calculated result is not stored in the pre-calculated result table. This reduces the size of the pre-calculated result table that stores pre-calculated processing results, leading to a reduction in the amount of hardware memory.

[0188] In addition, although the example has been described in which processing results for all values ​​(e.g., 16 values ​​when the feature is expressed as 4-bit data) of subregions with the same value are calculated in advance and stored in memory, the present invention is not limited to this. When subregions with the same feature value appear for the first time, convolution processing may be performed and the processing results may be stored in a calculation result table. When subregions with the same feature value appear for the second or subsequent time, the processing results may be read from the calculation result table without performing convolution processing (see FIG. 23). FIG. 23 shows an example in which subregions with the same feature value, 4, appear for the first time in subregion 2, convolution processing is performed, and the processing results are stored in the calculation result table. In this example, when subregions with the same feature value, 4, appear for the second or subsequent time, the processing results are read from the calculation result table without performing convolution processing. In addition, the calculation result table is reset each time each layer is processed. This eliminates the need for pre-calculation processing. In addition, since there is no need to transfer the pre-calculated results of each layer from RAM etc. to the internal memory of the arithmetic circuit for each layer, it is possible to reduce the amount of data transferred to RAM etc.

[0189] Furthermore, while the example described above deals with determining whether consecutive subregions correspond to the same subregion, the present invention is not limited to this. For example, it may be determined whether a subregion corresponds to a subregion processed several subregions ago, and if so, the processing result of the subregion processed several subregions ago may be output without convolution processing (see FIG. 24). FIG. 24 shows an example in which subregion 4 is determined to correspond to subregion 1 using not only an identity flag indicating whether a subregion corresponds to a subregion processed several subregions ago, but also a subregion interval parameter indicating how many subregions ago the subregion corresponded to, and the same processing result as subregion 1 is output without convolution processing. In this case, it is necessary to set an upper limit for the subregion interval and store the processing results for the subregions in the internal memory of the arithmetic circuit by the amount equal to the upper limit for the subregion interval. This increases the probability of skipping processing, which is expected to result in faster processing.

[0190] Although the case where a small region corresponds to a small region of the same value has been described as an example, the present invention is not limited to this. It may also be possible to determine whether a small region corresponds to a predetermined feature pattern other than a small region of the same value. For example, a processing result for a predetermined frequently occurring feature pattern may be obtained in advance, and it may be determined whether a small region corresponds to the predetermined frequently occurring feature pattern. If the small region corresponds to the predetermined frequently occurring feature pattern, the previously obtained processing result may be output without performing convolution processing.

[0191] Furthermore, although the image processing device has been described as including a learning unit and an inference unit, the present invention is not limited to this. The device including the learning unit and the device including the inference unit may be configured as separate devices. When there are significant hardware constraints, such as power consumption and size, it is preferable to configure the device including the learning unit and the device including the inference unit as separate devices. For example, this is the case when a device including an inference unit is mounted on a drone or used as an IoT or edge device. Furthermore, configuring a single device to include both a learning unit and an inference unit is generally used when performing high-speed learning using hardware located in a data center, such as in cloud computing.

[0192] Furthermore, the various processes executed by the CPU after reading software (programs) in the above embodiments may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and dedicated electrical circuits such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. Furthermore, the learning process and image processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0193] In addition, in each of the above embodiments, the learning processing program and the image processing program are described as being pre-stored (installed) in the storage 14, but this is not limiting. The programs may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network.

[0194] The following additional notes are provided regarding the above-described embodiments.

[0195] (Additional note 1) An image processing device including a neural network that includes convolution processing for an image, Memory and at least one processor coupled to said memory; Including, The processor: Acquire a target image to be processed; An image processing program for executing processing of the target image using a neural network including the convolution processing, When performing the convolution processing, the convolution processing is performed for each small region obtained by dividing an input feature map that is an input of the convolution processing; When performing the convolution process for each small region, if a feature constituting the small region corresponds to a predetermined feature or a feature of a small region processed in the past, the convolution process is not performed for the small region, and a result of processing the predetermined feature or a result of processing the small region in the past is output as a result of processing the small region; the small region corresponding to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, The small region corresponding to the feature of the previously processed small region is the small region whose difference from the feature of the previously processed small region is equal to or less than a threshold. Image processing device.

[0196] (Additional note 2) A non-transitory storage medium storing a program executable by a computer, the program including a neural network including convolution processing for an image, for performing image processing, The image processing Acquire a target image to be processed; An image processing program for executing processing of the target image using a neural network including the convolution processing, When performing the convolution processing, the convolution processing is performed for each small region obtained by dividing an input feature map that is an input of the convolution processing; When performing the convolution process for each small region, if a feature constituting the small region corresponds to a predetermined feature or a feature of a small region processed in the past, the convolution process is not performed for the small region, and a result of processing the predetermined feature or a result of processing the small region in the past is output as a result of processing the small region; the small region corresponding to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, The small region corresponding to the feature of the previously processed small region is the small region whose difference from the feature of the previously processed small region is equal to or less than a threshold. Non-transitory storage medium. [Explanation of symbols]

[0197] 10, 210 Image processing device 11 CPU 13 RAM 18 Arithmetic circuit 20 Learning Department 22 Reasoning part 30, 40 Acquisition Department 32, 42 Processing section 34 Update section

Claims

1. An image processing device including a neural network that includes convolution processing for an image, an acquisition unit that acquires a target image to be processed; a processing unit that processes the target image using a neural network including the convolution processing, When performing the convolution processing, the processing unit performs the convolution processing for each small region obtained by dividing an input feature map that is an input of the convolution processing, and When performing the convolution processing for each of the small regions, if a feature constituting the small region corresponds to a predetermined feature, the convolution processing is not performed for the small region, and a result of processing the predetermined feature is output as a result of processing the small region; If the features constituting the small region correspond to the features of a small region processed in the past, the convolution process is not performed on the small region, and the result of the previous process is output as the result of processing the small region; the small region corresponding to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, The small region corresponding to the feature of the previously processed small region is the small region whose difference from the feature of the previously processed small region is equal to or less than a threshold. Image processing device.

2. 2. The image processing apparatus according to claim 1, wherein the small region in which the difference from the feature is equal to or less than a threshold is a small region in which the difference from the feature for each pixel is equal to or less than a threshold.

3. 3. The image processing apparatus according to claim 2, wherein the small region whose difference from the feature is equal to or smaller than a threshold value has bits other than a predetermined number of lower-order bits that are identical to the feature.

4. 2. The image processing apparatus according to claim 1, wherein the small region in which the difference from the characteristic is equal to or less than a threshold is a small region in which the number of pixels in which the characteristic is different is equal to or less than a threshold.

5. 2. The image processing apparatus according to claim 1, wherein the threshold value is determined in advance so that the accuracy of the processing using the neural network is a predetermined accuracy.

6. 2. The image processing apparatus according to claim 1, wherein the predetermined features are the same within the small region.

7. An image processing method in an image processing device including a neural network that includes convolution processing for an image, an acquisition unit acquires a target image to be processed; a processing unit that processes the target image using a neural network that includes the convolution processing; When the processing unit performs the convolution processing, the processing unit performs the convolution processing for each small region obtained by dividing an input feature map that is an input of the convolution processing, When performing the convolution processing for each of the small regions, if a feature constituting the small region corresponds to a predetermined feature, the convolution processing is not performed for the small region, and a result of processing the predetermined feature is output as a result of processing the small region; If the features constituting the small region correspond to the features of a small region processed in the past, the convolution process is not performed on the small region, and the result of the previous process is output as the result of processing the small region; the small region corresponding to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, The small region corresponding to the feature of the previously processed small region is the small region whose difference from the feature of the previously processed small region is equal to or less than a threshold. Image processing methods.

8. A computer including a neural network that includes convolution processing for images, Acquire a target image to be processed; An image processing program for executing processing of the target image using a neural network including the convolution processing, When performing the convolution processing, the convolution processing is performed for each small region obtained by dividing an input feature map that is an input of the convolution processing; When performing the convolution processing for each of the small regions, if a feature constituting the small region corresponds to a predetermined feature, the convolution processing is not performed for the small region, and a result of processing the predetermined feature is output as a result of processing the small region; If the features constituting the small region correspond to the features of a small region processed in the past, the convolution process is not performed on the small region, and the result of the previous process is output as the result of processing the small region; the small region corresponding to the predetermined feature is the small region whose difference from the predetermined feature is equal to or less than a threshold, The small region corresponding to the feature of the previously processed small region is the small region whose difference from the feature of the previously processed small region is equal to or less than a threshold. Image processing program.

Images