IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

Abstract

When an output feature map, which is the output of a convolution process, is outputted, the output feature map is outputted to a storage unit for each small region obtained by dividing the output feature map. When the output feature map is outputted to the storage unit for each small region, if the features constituting the small region are the same as predetermined features or previously outputted small region features, the predetermined features or the previously outputted small region features are compressed and outputted to the storage unit.

Description

[Technical field]

[0001] The technology disclosed herein relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] When performing inference using a Convolutional Neural Network (CNN), the network is composed of multiple layers, and the convolutional layer performs convolution processing, which involves multiply-and-accumulate operations and activation processing.

[0003] In inference using CNN, the above convolution operations account for the majority of the total processing volume.

[0004] Even when implementing an inference engine using CNN as hardware, the performance of the convolution calculation is directly linked to the performance of the entire engine.

[0005] Examples of convolution operations when the kernel size is 3×3 are shown in Fig. 23 and Fig. 24. Fig. 23 shows an example of performing a convolution operation on a 3×3 input feature map using a 3×3 kernel. In this example, nine product-sum operations are performed to output a 1×1 output feature map.

[0006] Also, Fig. 24 shows an example in which a convolution operation is performed on a (W+2)×(H+2) input feature map using a 3×3 kernel. In this example, nine product-sum operations are repeated while moving the kernel over the input feature map, and a W×H output feature map is output.

[0007] In hardware that performs convolutional operations for CNN, in order to increase throughput, the input feature map is often divided into small regions of a certain fixed size, and a circuit is prepared that can perform multiply-and-accumulate operations on one small region at a time (see Figure 25). Figure 25 shows an example in which a 26x14 input feature map is divided into nine 10x6 small regions, and the calculation circuit performs convolution processing on 32 points (8x4 points) simultaneously using a 3x3 kernel, outputting an 8x4 output feature map. In this example, the dot part of the input feature map is one small region, and the calculation circuit performs convolution processing on 32 points simultaneously for each of the nine small regions, outputting a 24x12 output feature map.

[0008] As one of the methods for speeding up calculations, a method is known in which calculations are skipped when the values ​​of a small area of ​​an input feature map are all zero, as shown in FIG. 26 (for example, Non-Patent Document 1). FIG. 26 shows an example in which the size of the output small area is 4×2, the kernel size is 3×3, and 4-bit data representing 0 to 15 is used. In this example, the size of the small area of ​​the input feature map is 6×4, and the values ​​of the small areas represented by dotted lines are all 0. Since the result of a product-sum operation for 0 is 0, there is no need to perform convolution processing in the calculation circuit, and it is possible to skip the convolution processing of the small area. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W. Keckler, William J. Dally, "SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks", arXiv:1708.04485, 23 May 2017 Summary of the Invention [Problem to be solved by the invention]

[0010] Here, if the size of the small region is increased in order to increase the throughput, the values ​​of the small region in the input feature map rarely become all zero, and sufficient speed-up of the calculation cannot be expected. For example, as shown in FIG. 27A, when the size of the small region in the output feature map is 4×2 (when the size of the small region in the input feature map is 6×4), the values ​​of the small region represented by the dotted line in the input feature map are all zero. On the other hand, as shown in FIG. 27B, when the size of the small region in the output feature map is 8×4 (when the size of the small region in the input feature map is 10×6), the small region represented by the dotted line in the input feature map contains a non-zero value.

[0011] Furthermore, since the size of the small region is directly related to the computational throughput, it is often difficult to change it.

[0012] Furthermore, when outputting the output feature map to memory, the larger the data size, the longer it takes to access the memory, which hinders high-speed calculation.

[0013] The disclosed technology has been developed in consideration of the above points, and aims to provide an image processing device, an image processing method, and an image processing program that can reduce the data size when outputting an output feature map. [Means for solving the problem]

[0014] A first aspect of the present disclosure is an image processing device including a neural network including convolution processing for an image, the image processing device including an acquisition unit that acquires a target image to be processed, and a processing unit that processes the target image using the neural network including the convolution processing, wherein when outputting an output feature map that is the output of the convolution processing, the processing unit outputs the output feature map to a memory unit for each divided small region, and when outputting each small region to the memory unit, if features constituting the small region are the same as predetermined features or features of a small region outputted in the past, the processing unit compresses the predetermined features or the features of the small region outputted in the past and outputs them to the memory unit.

[0015] A second aspect of the present disclosure is an image processing method in an image processing device including a neural network including convolution processing for an image, wherein an acquisition unit acquires a target image to be processed, and a processing unit processes the target image using the neural network including the convolution processing, and when the processing unit outputs an output feature map which is the output of the convolution processing, the output feature map is output to a memory unit for each divided small region, and when outputting each small region to the memory unit, if features constituting the small region are the same as predetermined features or features of a small region output in the past, the predetermined features or the features of the small region output in the past are compressed and output to the memory unit.

[0016] A third aspect of the present disclosure is an image processing program for causing a computer including a neural network including a convolution process for an image to acquire a target image that is to be processed, and process the target image using the neural network including the convolution process, wherein when outputting an output feature map that is the output of the convolution process, the output feature map is output to a memory unit for each divided small region, and when outputting each small region to the memory unit, if features constituting the small region are the same as predetermined features or features of a small region output in the past, the predetermined features or the features of the small region output in the past are compressed and output to the memory unit. Effect of the Invention

[0017] According to the disclosed technology, it is possible to reduce the data size when outputting an output feature map. [Brief description of the drawings]

[0018] [Figure 1] FIG. 13 is a diagram for explaining a small region of an input feature map. [Diagram 2] FIG. 13 is a diagram for explaining a method for skipping convolution processing for a small region of an input feature map. [Diagram 3] FIG. 1 is a schematic block diagram of an example of a computer that functions as an image processing apparatus according to the first and second embodiments. [Figure 4] FIG. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to a first and second embodiments. [Diagram 5] FIG. 2 is a block diagram illustrating a functional configuration of a learning unit of the image processing device according to the first and second embodiments. [Figure 6] FIG. 2 is a block diagram illustrating a functional configuration of an inference unit of the image processing device according to the first and second embodiments. [Figure 7] 4 is a flowchart showing the flow of a learning process according to the first and second embodiments. [Figure 8]1 is a flowchart showing the flow of a convolution process in learning processing and image processing according to the first and second embodiments. [Figure 9] 4 is a flowchart showing the flow of image processing according to the first and second embodiments. [Figure 10A] FIG. 13 is a diagram showing an example of a small region using 4-bit data. [Figure 10B] FIG. 13 is a diagram showing an example of a small region using 6-bit data. [Figure 11] FIG. 13 is a diagram for explaining a method of performing convolution processing on a plurality of small regions in parallel. [Figure 12] 13A and 13B are diagrams for explaining a method for skipping convolution processing for a small region of an input feature map in a modified example. [Figure 13] 13A and 13B are diagrams for explaining a method for skipping convolution processing for a small region of an input feature map in a modified example. [Figure 14] 13A and 13B are diagrams for explaining a method for skipping convolution processing for a small region of an input feature map in a modified example. [Figure 15] FIG. 13 is a diagram for explaining a method of compressing an output feature map and outputting it to a RAM. [Figure 16] FIG. 13 is a diagram illustrating an example of a subregion of an output feature map. [Figure 17] FIG. 13 illustrates an example of sub-regions with overlapping regions in an output feature map. [Figure 18] FIG. 13 illustrates an example of sub-regions with no overlapping regions in an output feature map. [Figure 19] 13A and 13B are diagrams showing an example of an equality flag, a continuity flag, and output data for small regions that are neither equal small regions nor consecutive identical small regions. [Figure 20] 13A and 13B are diagrams illustrating an example of an equality flag, a consecutive flag, and output data for small regions of equality. [Figure 21] 13A and 13B are diagrams illustrating an example of equal flags, consecutive flags, and output data for consecutive identical small regions. [Figure 22]13 is a flowchart showing the flow of learning processing and convolution processing in image processing according to a third embodiment. [Diagram 23] FIG. 13 is a diagram illustrating an example of a convolution process. [Figure 24] FIG. 13 is a diagram illustrating an example of a convolution process. [Diagram 25] FIG. 13 is a diagram illustrating an example of convolution processing for each small region. [Figure 26] FIG. 2 illustrates an example of a subregion of an input feature map. [Figure 27A] FIG. 2 illustrates an example of a subregion of an input feature map. [Figure 27B] FIG. 2 illustrates an example of a subregion of an input feature map. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0020] <Overview of the Disclosed Technology> In the disclosed technology, after the data of the input feature map of the convolution layer is read from a RAM (Random Access Memory) or the like, it is determined for each small region whether all the features in the small region are the same or whether they are consecutively the same small regions. Hereinafter, a small region in the input feature map in which all the features are the same is referred to as a "small region of the same value". Also, a small region in which the features are completely the same as the previous small region is referred to as a "consecutively identical small region". In FIG. 1, as an example of "same value small regions", an example in which all the features in the small region surrounded by a dotted line are 4 is shown. Also, as an example of "consecutively identical small regions", an example in which the features in the small region surrounded by a thin dashed line and the features in the small region surrounded by a thick dashed line are the same is shown. Here, the reason for determining whether the small regions are the same is because the input image often has the same value. Also, the reason for determining whether the small regions are consecutively the same is because the input image often has similar regions. For example, in a 10x6 region of the input image, the 10 points in the top row have a value of 10, and the 50 points in the bottom three rows have a value of 12. In both cases of regions with the same value and consecutive identical small regions, this applies when the input image is not a complex picture region but a flat, uniform region.

[0021] Also, as shown in Figure 2, an equivalence flag indicating the result of the judgment for each small region as to whether or not the small regions are of the same value, a continuity flag indicating the result of the judgment for each small region as to whether or not the small regions are consecutive and identical, and small region data, which is the data for each small region of the input feature map, are input to an arithmetic circuit that performs convolution processing.

[0022] In the arithmetic circuit, if the small regions to be processed are small regions of the same value or consecutive identical small regions, a process skip is performed without performing convolution processing. If consecutive identical small regions are processed, the processing result will be the same as that of the small region processed immediately before, so it is only necessary to continuously output the processing result of the small region processed immediately before, thereby speeding up processing. Figure 2 shows an example in which the features in small region 3 and the features in small region 4 are identical, so a process skip is performed on small region 4.

[0023] In addition, for small regions with the same value, the processing results are limited. If the value indicating the feature is 4 bits, the processing results are 16 patterns, and if the value indicating the feature is 8 bits, the processing results are 256 patterns. The processing results of all patterns are pre-calculated and the pre-calculated result table is stored in RAM, and for each layer processing, the pre-calculated result table is read from RAM to the internal memory of the arithmetic circuit. This speeds up processing because the processing results can be obtained by only referring to the internal memory without performing convolution processing for small regions with the same value. In Figure 2, since small region 2 is a small region with the same value, processing is skipped for small region 2, and the processing results when all features are 4 stored in the pre-calculated result table are output by referring to them.

[0024] In addition, as shown in Figure 2, the output feature map, which is the result of the convolution process by the arithmetic circuit, is written to a RAM, etc. The written output feature map is used as an input feature map in the next layer and onwards, and the above-mentioned determination process and convolution process using processing skip are repeated again.

[0025] [First embodiment] <Configuration of the image processing device according to the first embodiment> FIG. 3 is a block diagram showing the hardware configuration of the image processing device 10 of the first embodiment.

[0026] 3, the image processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, a communication interface (I / F) 17, and an arithmetic circuit 18. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0027] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a learning processing program for performing learning processing of a neural network and an image processing program for performing image processing using a neural network. The learning processing program and the image processing program may be one program, or may be a group of programs composed of multiple programs or modules.

[0028] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0029] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0030] The input unit 15 receives, as input, learning data for training a neural network. For example, the input unit 15 receives, as input, learning data including a target image to be processed and a processing result for the target image that has been obtained in advance.

[0031] Moreover, the input unit 15 receives, as an input, a target image to be processed.

[0032] The display unit 16 is, for example, a liquid crystal display, and displays various information including the processing results. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0033] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0034] The arithmetic circuit 18 executes the convolution process in the convolution layer of the neural network. Specifically, the arithmetic circuit 18 receives a subregion of an input feature map and a kernel, and outputs a subregion of an output feature map that is the result of the convolution process of each point in the subregion.

[0035] Next, a description will be given of the functional configuration of the image processing device 10. Fig. 4 is a block diagram showing an example of the functional configuration of the image processing device 10.

[0036] Functionally, the image processing device 10 includes a learning unit 20 and an inference unit 22, as shown in FIG.

[0037] As shown in FIG. 5, the learning unit 20 includes an acquisition unit 30, a processing unit 32, and an update unit .

[0038] The acquisition unit 30 acquires the target image and the processing result of the input learning data.

[0039] The processing unit 32 processes the target image of the learning data using a neural network including a convolution process. When performing the convolution process, the processing unit 32 performs the convolution process for each small region obtained by dividing an input feature map that is the input of the convolution process. The convolution process for each small region is performed using the arithmetic circuit 18. At this time, the small region data and kernel of the input feature map are input to the arithmetic circuit 18, and an output feature map representing the result of the convolution process for each small region is output from the arithmetic circuit 18.

[0040] Here, the input feature map is divided into small regions as shown in FIG. 1. In FIG. 1, the regions corresponding to the small regions of the output feature map are shown in bold frames, and the small regions of the input feature map include the corresponding small regions of the output feature map and the overlapping regions of the adjacent small regions. In this way, the small regions divided into the input feature map are overlapping regions of the adjacent small regions, and have overlapping regions according to the kernel size of the convolution process. The division of the small regions is determined at the time of design. For example, it may be determined by balancing the hardware constraints (size and power) and the desired throughput (xx fps, etc.). If the kernel size is 3×3, the small regions of the input feature map are 10×6, and the small regions of the output feature map are 8×4, a throughput that can output 32 points simultaneously is obtained. In this case, 288 product-sum calculation circuits (=(3×3)×8×4) are required. In addition, if the input feature map subregion is 6x4 and the output feature map subregion is 4x2, the throughput will be such that only 8 points are output at a time. In this case, 72 product-sum circuits (=(3x3)x4x2) are required. In addition, when implementing product-sum operations in hardware, the division of the subregions is determined at the time of design, so the division of the subregions does not change depending on the layer.

[0041] When performing convolution processing for each small region, if all the features constituting the small region have the same value, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for the predetermined case in which all the features constituting the small region have the same value as the result of processing the small region.

[0042] Specifically, the processing unit 32 judges, for each small region, whether all the features constituting the small region are the same value or not. If it is judged that all the features constituting the small region are the same value, the arithmetic circuit 18 does not perform the convolution process for the small region, and outputs the result of processing for the case where all the features constituting the small region are the same value, which is stored in the pre-computation result table, as the result of processing the small region.

[0043] Furthermore, when performing convolution processing for each small region, if the features constituting the small region are the same as the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the results of processing on the previously processed small region as the results of processing the small region.

[0044] Specifically, the processing unit 32 judges for each small region whether the features constituting the small region are the same as the features of the small region processed immediately before. If it is judged that the features constituting the small region are the same as the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing the immediately previous small region as the result of processing the small region.

[0045] The update unit 34 updates the parameters of the neural network so that the result of processing the target image using the neural network matches the processing result obtained in advance. The update unit 34 also updates the pre-computation result table of each convolution layer based on the updated parameters of the neural network.

[0046] Each process of the processing unit 32 and the updating unit 34 is repeated until a predetermined iteration end condition is satisfied, thereby training the neural network.

[0047] As shown in FIG. 6, the inference unit 22 includes an acquisition unit 40 and a processing unit .

[0048] The acquisition unit 40 acquires an input target image that is to be processed.

[0049] The processing unit 42 processes the target image using a neural network including a convolution process. When performing the convolution process, the processing unit 42 performs the convolution process for each small region obtained by dividing an input feature map that is the input for the convolution process. The convolution process for each small region is performed using the arithmetic circuit 18. At this time, the small region data and kernel of the input feature map are input to the arithmetic circuit 18, and an output feature map representing the result of the convolution process for each small region is output from the arithmetic circuit 18.

[0050] The processing unit 42 judges, for each small region, whether or not all the features constituting the small region have the same value, similarly to the processing unit 32. If it is judged that all the features constituting the small region have the same value, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for the case where all the features constituting the small region have the same value, which is stored in the pre-computation result table, as the result of processing the small region.

[0051] Moreover, like the processing unit 32, the processing unit 42 judges whether or not the features constituting the relevant small region are the same as the features of the sub-region processed immediately before. If it is judged that the features constituting the relevant small region are the same as the features of the sub-region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the relevant small region, and outputs the result of processing the previously processed sub-region as the result of processing the relevant small region.

[0052] The result of processing the target image using the neural network is displayed on the display unit 16.

[0053] <Function of the image processing device according to the first embodiment> Next, the operation of the image processing device 10 according to the first embodiment will be described.

[0054] 7 is a flowchart showing the flow of the learning process by the image processing device 10. The learning process is performed by the CPU 11 reading out a learning process program from the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it. In addition, learning data is input to the image processing device 10.

[0055] In step S100, the CPU 11 functions as the acquisition unit 30 to acquire a target image that is to be processed from the input learning data, and the processing result.

[0056] In step S102, the CPU 11, as the processing unit 32, processes the target image of the learning data using a neural network including convolution processing.

[0057] In step S104, the CPU 11, as the update unit 34, updates the parameters of the neural network so that the results of processing the target image of the learning data using the neural network match the processing results obtained in advance, and updates the pre-calculation result table.

[0058] In step S106, the CPU 11 determines whether or not a predetermined iteration end condition is satisfied. If the iteration end condition is not satisfied, the process returns to step S102, and the processes of the processing unit 32 and the update unit 34 are repeatedly performed. This allows the neural network to learn.

[0059] In step S102, the computation process is performed for each layer of the neural network. Here, the computation process for the convolution layer is realized by the processing routine shown in FIG.

[0060] In step S110, the CPU 11, functioning as the processing unit 32, divides an input feature map, which is to be the input of the convolution layer, into small regions.

[0061] In step S112, the CPU 11, functioning as the processing unit 32, reads out from the RAM 13 the pre-computation result table of the convolution layer.

[0062] In step S114, the CPU 11, as the processing unit 32, processes each of the divided small areas in turn, and determines whether the features constituting the small area to be processed are the same as or identical to the features of the small area processed immediately before.

[0063] In step S116, the CPU 11, as the processing unit 32, outputs each small region data of the input feature map, the pre-computation result table, and the equivalence flag and consecutive flag indicating the judgment result of step S114 to the arithmetic circuit 18. Then, the arithmetic circuit 18 performs convolution processing for each small region. At this time, if the small regions to be processed are not small regions of the same value and are not consecutively the same small regions, the arithmetic circuit 18 performs convolution processing on the small region to be processed. If the small regions to be processed are small regions of the same value, the arithmetic circuit 18 does not perform convolution processing on the small region to be processed, and outputs the result of processing for the case where the features constituting the small regions are the same value stored in the pre-computation result table as the result of processing the small region to be processed.

[0064] Furthermore, if the small regions to be processed are consecutive identical small regions, the arithmetic circuit 18 does not perform convolution processing on the small regions to be processed, but outputs the results of processing on the small region processed immediately before as the results of processing the small regions to be processed.

[0065] Then, the processing routine is terminated, and an output feature map consisting of the processing results for each small region is output as the input feature map for the next layer.

[0066] 9 is a flowchart showing the flow of image processing by the image processing device 10. Image processing is performed by the CPU 11 reading out an image processing program from the ROM 12 or storage 14, deploying it in the RAM 13, and executing it. A target image is input to the image processing device 10.

[0067] In step S120, the CPU 11 functions as the acquisition unit 40 to acquire the input target image.

[0068] In step S122, the CPU 11, functioning as the processing unit 42, processes the target image using the neural network learned by the above-described learning process. Then, the result of processing the target image using the neural network is displayed on the display unit 16.

[0069] In step S122, arithmetic processing is performed for each layer of the neural network. Here, the arithmetic processing for the convolution layer is realized by the processing routine shown in FIG.

[0070] As described above, when the image processing device according to the first embodiment performs convolution processing for each small region of an input feature map, if the features constituting the small region are the same value or are the same as the features of the small region processed immediately before, it does not perform convolution processing for the small region, and outputs the result of processing a predetermined small region with the same value or the result of the immediately previous processing as the result of processing the small region. This makes it possible to speed up processing using a neural network including convolution processing.

[0071] When the size of the small regions divided from the input feature map is increased to improve throughput, or when the bit depth expressing the input feature map is increased to improve CNN operation accuracy, the inside of the small region is rarely all zero in the conventional method, and the operation cannot be accelerated in many cases. For example, as shown in FIG. 10A, when the feature of the input feature map is expressed by 4-bit data, the inside of the small region is all zero, whereas as shown in FIG. 10B, when the feature of the input feature map is expressed by 6-bit data, the inside of the small region is not all zero. On the other hand, in the present embodiment, sufficient operation speedup is expected compared to the conventional method in small regions of the same value or consecutively the same small regions.

[0072] [Second embodiment] The second embodiment differs from the first embodiment in that the arithmetic circuit performs convolution processing on a plurality of small regions in parallel.

[0073] <Outline of the second embodiment> In the second embodiment, the arithmetic circuit performs convolution processing on multiple small regions of an input feature map in parallel, as shown in Fig. 11. Fig. 11 shows an example in which the size of the small regions of the input feature map is 6 × 4, the arithmetic circuit performs convolution processing on four small regions in parallel, and outputs the results of the convolution processing of 32 points simultaneously.

[0074] At this time, it is determined for each small region whether they are small regions of the same value or consecutively the same small regions. In the example of Fig. 11, small region 1 is a small region of the same value, and small region 3 is a consecutively the same small region, so the convolution process of small region 1 and small region 3 is skipped. Also, the convolution process is performed in parallel on small region 0, small region 2, small region 4, and small region 5, and as a result, the processing results for six small regions are obtained at one time.

[0075] In this way, for each small region, a determination is made as to whether it is a small region of the same value or whether it is a series of the same small regions, and by using an arithmetic circuit to perform convolution processing on multiple small regions in parallel, it is possible to increase the probability that the small regions are of the same value and the probability that they are a series of the same small regions, thereby increasing the probability that the convolution processing will be skipped.

[0076] <Configuration of image processing device according to the second embodiment> An image processing apparatus according to the second embodiment will be described below. The same reference numerals will be used to designate the same components as those in the first embodiment, and the description thereof will be omitted.

[0077] The hardware configuration of the image processing device 210 of the second embodiment is similar to the hardware configuration of the image processing device 10 shown in FIG.

[0078] The arithmetic circuit 18 of the image processing device 210 receives small region data of an input feature map and a kernel, repeatedly performs convolution processing on a predetermined number of small regions in parallel, and outputs an output feature map that is the result of the convolution processing of each point in each small region.

[0079] The processing unit 32 of the learning unit 20 processes the target image of the learning data using a neural network including convolution processing. When performing the convolution processing, the processing unit 32 performs the convolution processing for each small region obtained by dividing the input feature map that is the input of the convolution processing. The convolution processing for each small region is executed using the arithmetic circuit 18. At this time, the processing unit 32 judges, for each small region, whether or not the small regions are of the same value, and whether or not the small regions are consecutive and the same small region. The small region data and kernel of the input feature map, the judgment results for each small region, and the pre-computation result table are input to the arithmetic circuit 18, and an output feature map is output from the arithmetic circuit 18.

[0080] Specifically, when performing the convolution process, the processing unit 32 judges, for each small region, whether or not all the features constituting the small region have the same value. If it is judged that all the features constituting the small region have the same value, the arithmetic circuit 18 does not perform the convolution process for the small region, and outputs the result of processing for the case where all the features constituting the small region have the same value, which is stored in the pre-computation result table, as the result of processing the small region.

[0081] Furthermore, the processing unit 32 judges for each small region whether the features constituting the small region are the same as the features of the small region processed immediately before. If it is judged that the features constituting the small region are the same as the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing the immediately previous small region as the result of processing the small region.

[0082] The arithmetic circuit 18 also performs the convolution process in parallel on a predetermined number of small regions, all of whose features are not identical and are not the same as the features of the small region processed immediately before.

[0083] The output feature map, which is the result of the convolution process performed on each small region as described above, becomes the input to the next layer of the neural network.

[0084] The processing unit 42 of the inference unit 22 processes the target image using a neural network including convolution processing, similar to the processing unit 32. When performing convolution processing, the processing unit 42 performs convolution processing for each small region obtained by dividing the input feature map that is the input for the convolution processing. The convolution processing for each small region is executed using the arithmetic circuit 18. At this time, the processing unit 42 judges, for each small region, whether or not the small regions are of the same value, and whether or not the small regions are consecutive and the same small region. The small region data and kernel of the input feature map, the judgment results for each small region, and the pre-computation result table are input to the arithmetic circuit 18, and an output feature map is output from the arithmetic circuit 18.

[0085] Specifically, when performing the convolution process, the processing unit 42 determines for each small region whether all the features constituting the small region have the same value, similar to the processing unit 32. If it is determined that all the features constituting the small region have the same value, the arithmetic circuit 18 does not perform the convolution process for the small region, and outputs the result of processing for the case where all the features constituting the small region have the same value, which is stored in the pre-computation result table, as the result of processing the small region.

[0086] Similarly to the processing unit 32, the processing unit 42 judges for each small region whether the features constituting the small region are the same as the features of the small region processed immediately before. If it is judged that the features constituting the small region are the same as the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing the immediately previous small region as the result of processing the small region.

[0087] The arithmetic circuit 18 also performs the convolution process in parallel on a predetermined number of small regions, all of whose features are not identical and are not the same as the features of the small region processed immediately before.

[0088] The output feature map, which is the result of the convolution process performed on each small region as described above, becomes the input to the next layer of the neural network.

[0089] <Action of the image processing device according to the second embodiment> Next, the operation of the image processing device 210 according to the second embodiment will be described.

[0090] The CPU 11 of the image processing device 210 reads out a learning processing program from the ROM 12 or the storage 14, deploys it in the RAM 13, and executes it to perform processing similar to the learning processing shown in Fig. 7. In addition, learning data is input to the image processing device 210.

[0091] In step S102, arithmetic processing is performed for each layer of the neural network. Here, the arithmetic processing for the convolution layer is realized by the processing routine shown in FIG.

[0092] In step S116, if the features constituting the small region to be processed are the same, the arithmetic circuit 18 does not perform convolution processing on the small region to be processed, and outputs the result of processing for the case where the features constituting the small region are the same, which is stored in the pre-computation result table, as the result of processing the small region to be processed.

[0093] Furthermore, if the features constituting the small region to be processed are the same as the features of the small region processed immediately before, the arithmetic circuit 18 does not perform convolution processing on the small region to be processed, but outputs the results of processing on the small region processed immediately before as the results of processing the small region to be processed.

[0094] The arithmetic circuit 18 also performs convolution processing in parallel on a predetermined number of small regions whose constituent features are not identical and are not the same as the features of the small region processed immediately before.

[0095] The CPU 11 of the image processing device 210 reads out an image processing program from the ROM 12 or the storage 14, deploys it in the RAM 13, and executes it to perform processing similar to the image processing shown in Fig. 9. A target image is input to the image processing device 210.

[0096] In step S122, arithmetic processing is performed for each layer of the neural network. Here, the arithmetic processing for the convolution layer is realized by the processing routine shown in FIG.

[0097] Other configurations and operations of the image processing device 210 of the second embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0098] As described above, in the image processing device according to the second embodiment, the arithmetic circuit is configured to perform convolution processing on a plurality of small regions in parallel, and when performing convolution processing for each small region of the input feature map, if the features constituting the small region are the same value or are the same as the features of the small region processed immediately before, convolution processing is not performed on the small region, and the result of processing on a predetermined small region with the same value or the result of processing immediately before is output as the result of processing the small region. As a result, even in cases where it is not expected to skip convolution processing for small regions with a large bit depth or parallel processing unit and all values ​​being zero, it is expected to speed up processing by skipping processing without decreasing arithmetic accuracy or throughput.

[0099] The present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

[0100] For example, the case where the processing results are calculated in advance for each of all values ​​(for example, 16 values ​​when the feature is expressed by 4-bit data) for the small areas with the same value and stored in the memory has been described as an example, but the present invention is not limited to this. The feature values ​​that are likely to appear as small areas with the same value may be determined in advance by performing a simulation or the like, and the processing results for the small areas with the same value may be obtained in advance only for the top several feature values ​​with the highest occurrence probability, and stored in the pre-calculation result table (see FIG. 12). For small areas with the same value and the processing results are obtained in advance, the convolution process is not performed and the previously obtained processing results are output. FIG. 12 shows an example in which the processing results are obtained in advance only for the small areas with the same value whose feature value is 1, the small areas with the same value whose feature value is 4, and the small areas with the same value whose feature value is 15. In addition, the processing results for the small areas with the same value whose feature value is 0 are not stored in the pre-calculation result table because the processing results are 0. This makes it possible to reduce the size of the pre-calculation result table that stores the previously obtained processing results, leading to a reduction in the amount of memory in the hardware.

[0101] In addition, the case where the processing results are calculated in advance for each of all values ​​(for example, 16 values ​​when the feature is expressed by 4-bit data) for small regions with the same value and stored in memory has been described as an example, but the present invention is not limited to this. When a small region with the same feature value appears for the first time, a convolution process is performed and the processing results are stored in the calculation result table, and when a small region with the same feature value appears for the second or subsequent time, the processing results may be read from the calculation result table without performing the convolution process (see FIG. 13). FIG. 13 shows an example in which a small region with the same feature value of 4 appears for the first time in small region 2, a convolution process is performed, and the processing results are stored in the calculation result table. In this example, when a small region with the same feature value of 4 appears for the second or subsequent time, the processing results are read from the calculation result table without performing the convolution process. In addition, the calculation result table is reset every time each layer is processed. This eliminates the need for laborious pre-calculation processing. In addition, since there is no need to transfer the pre-calculation results of each layer from RAM etc. to the internal memory of the arithmetic circuit for each layer, it is possible to reduce the amount of data transferred to and from RAM etc.

[0102] In addition, although the case where consecutive small regions are determined to be the same or not has been described as an example, the present invention is not limited to this. For example, it may be determined whether or not a small region is the same as a small region processed several times before, and if it is the same as the small region processed several times before, the processing result of the small region processed several times before may be output without performing convolution processing (see FIG. 14). FIG. 14 shows an example in which a small region 4 is determined to be the same as small region 1 using a parameter of a small region interval indicating how many times before the small region was the same as the small region processed several times before, in addition to an identity flag indicating whether or not it is the same as the small region processed several times before, and a processing result the same as that of small region 1 is output without performing convolution processing. In this case, it is necessary to set an upper limit of the small region interval, and store the processing results for the small regions in the internal memory of the arithmetic circuit by the amount of the upper limit of the small region interval. This increases the probability of skipping processing, and thus it is expected to increase the speed.

[0103] [Third embodiment] The third embodiment differs from the first embodiment in that the result of the convolution process by the arithmetic circuit is compressed and stored in the RAM.

[0104] <Outline of the third embodiment> Feature maps in convolution processing often have huge data sizes, and therefore require large amounts of data transfer when reading and writing feature maps from memory such as RAM. Waiting for data transfer from memory often becomes a bottleneck in processing using CNN, so in order to improve CNN processing performance, it is necessary to reduce not only the amount of calculations but also the data size.

[0105] In particular, in the first embodiment, the amount of calculation is reduced by skipping processing, but the amount of transfer for reading from and writing to the memory remains unchanged, so there is a possibility that the gap between the two will become larger.

[0106] Therefore, in the third embodiment, in order to reduce the amount of calculation, the result of the determination as to whether the small regions are of the same value or whether the small regions are consecutive and identical is reused, and the feature map is compressed in accordance with the result of the determination as to whether the small regions are of the same value or whether the small regions are consecutive and identical, without adding a new determination process or flag.

[0107] 15, instead of judging whether the input feature map of the convolution process has the same value or whether the small regions are consecutive and the same, as in the first embodiment, a judgment is made as to whether the output feature map of the convolution process has the same value or whether the small regions are consecutive and the same before the output feature map is output to the RAM 13. Also, a flag indicating the judgment result is output to the RAM 13 together with the output feature map.

[0108] In normal convolution processing, the output feature map of the convolution processing of the previous layer becomes the input feature map of the current layer, so determining whether the output feature map of the previous layer is a sub-region with the same value or a continuous sub-region is the same as determining whether the input feature map of the current layer is a sub-region with the same value or a continuous sub-region, and has no effect on the processing result. Also, in the ROUTE layer, there is no effect on the processing result when the output feature map of several layers before is used as the input feature map of the current layer.

[0109] Therefore, in this embodiment, a determination is made as to whether the output feature map contains small regions of the same value or whether the small regions are consecutive and identical, and the determination result is used to perform a compression process on the output feature map, and the compressed output feature map is output to RAM 13.

[0110] When performing the convolution process of the next layer, the compressed output feature map stored in the RAM 13 is read out and expanded into an input feature map before performing the convolution process.

[0111] In Fig. 15, the size of the small regions in the output feature map is set to 6 x 4, and it is determined for each small region whether they are small regions of the same value or consecutive and the same small regions. In the example of Fig. 15, small region 2 is a small region of the same value, and small region 4 is a consecutive and the same small region, and the data of small region 2 and small region 4 are compressed and stored in RAM 13. In the next layer, the compressed output feature map read from RAM 13 is expanded into the input feature map, and when convolution processing is performed for each small region, the convolution processing of small region 2 and small region 4 is skipped.

[0112] In this way, for each small region of the output feature map, it is determined whether it is a small region of the same value or a series of the same small regions, and the data of the small regions determined to be a small region of the same value or a series of the same small regions is compressed to create a compressed output feature map, which is stored in RAM 13. During the convolution process of the next layer, the compressed output feature map is read from RAM 13 and expanded into the input feature map. This makes it possible to reduce the amount of data transferred when reading from and writing to RAM 13.

[0113] Furthermore, in the convolution processing of the next layer, the convolution processing of small regions that are determined to be equal or consecutive identical small regions is skipped, thereby speeding up the convolution processing.

[0114] <Configuration of image processing device according to the third embodiment> An image processing device according to the third embodiment will be described below. The same reference numerals will be used to designate the same components as those in the first embodiment, and the description thereof will be omitted.

[0115] The hardware configuration of the image processing device 310 of the third embodiment is similar to the hardware configuration of the image processing device 10 shown in FIG.

[0116] The processing unit 32 of the learning unit 20 processes the target image of the learning data using a neural network including convolution processing. When outputting an output feature map that is the output of the convolution processing, the processing unit 32 outputs the output feature map to the RAM 13 for each divided small region. When outputting each small region to the RAM 13, the processing unit 32 judges whether the small regions are small regions of the same value or whether they are consecutively the same small regions. The judgment result for each small region and a compressed output feature map in which the small regions judged to be small regions of the same value or consecutively the same small regions are compressed are output to the RAM 13.

[0117] Here, the output feature map is divided into small regions as shown in FIG. 16. In FIG. 16, the regions corresponding to the small regions of the output feature map are displayed in bold frames. However, the small regions of the output feature map that are units for determining whether or not they are the same small regions and whether or not they are consecutively the same small regions are regions that include overlapping regions that overlap with adjacent small regions (see FIG. 17). In this way, the small regions that are units for determination are overlapping regions that overlap with adjacent small regions and have overlapping regions according to the kernel size of the convolution processing of the next layer. The division of the small regions is determined at the time of design. FIG. 17 shows an example in which the kernel size is 3×3 and the small regions of the output feature map are 4×2, and the small regions that are units for determination are 6×4. In addition, when outputting each small region to the RAM 13, each small region that does not include an overlapping region that overlaps with adjacent small regions is output to the RAM 13 as shown in FIG. 18. In FIG. 18, an example in which the small region that does not include an overlapping region that overlaps with adjacent small regions is 4×2 is shown. When the kernel size of the convolution process in the next layer is 1×1, the small region that serves as the unit of determination does not have an overlapping region that overlaps with adjacent small regions.

[0118] 19, when outputting each small region to RAM 13, if the small region is neither a small region of the same value nor a continuous identical small region, the small region is output as is to RAM 13 without being compressed. Fig. 19 shows an example in which the small region shown in the dotted frame has an equal value flag and a continuity flag indicating that the small region is neither a small region of the same value nor a continuous identical small region, and small region data (8 values) shown in the thick frame that does not include an overlapping region that overlaps with adjacent small regions, are output to RAM 13.

[0119] 20, when outputting each small region to RAM 13, if the small regions are of the same value, the small regions are compressed to one value and output to RAM 13. In FIG. 20, an example is shown in which the small regions to be judged, indicated by the dotted frame, are of the same value, and an equality flag and a consecutive flag indicating that the small regions are not consecutive and identical, and a single value obtained by compressing the small region data are output to RAM 13. In this way, in small regions of the same value, the small region data can be expressed by only one value, so it is sufficient to output only one representative value to RAM 13.

[0120] Also, as shown in Fig. 21, when outputting each small region to RAM 13, if the small regions are consecutive and identical, the small region data is not output to RAM 13. Fig. 21 shows an example in which an equality flag and a consecutive flag indicating that the small regions to be judged, which are indicated by dotted lines, are consecutive and identical, rather than being identical small regions, are output to RAM 13, and no small region data is output. In this way, consecutive and identical small regions have the same small region data as the small region immediately preceding them, so if the judgment result indicates that the small regions are consecutive and identical, there is no need to output the small region data.

[0121] Furthermore, when performing the convolution process, the processing unit 32 reads out the compressed output feature map of the previous layer from the RAM 13, and expands it as an input feature map that is to be input to the convolution process.

[0122] Specifically, the processing unit 32 expands the previous compressed output feature map read from the RAM 13 into small region data for each small region including an overlap region that overlaps with adjacent small regions, and sets the small region data for each small region obtained by dividing the input feature map that is the input for the convolution process. The processing unit 32 also performs convolution processing for each small region obtained by dividing the input feature map that is the input for the convolution process. The convolution processing for each small region is performed using the arithmetic circuit 18. At this time, the processing unit 32 judges, for each small region, whether or not the small regions are of the same value, and whether or not the small regions are consecutively the same small regions. At this time, based on the equivalence flag and consecutive flag of the small region read from the RAM 13, the processing unit 32 judges whether or not the small regions are of the same value, and whether or not the small regions are consecutively the same small regions. Furthermore, the small region data of the input feature map, the kernel, the judgment results for each small region, and the pre-computation result table are input to the arithmetic circuit 18, and the arithmetic circuit 18 outputs the output feature map.

[0123] Furthermore, when the processing unit 32 determines that the small region is an equivalent small region, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the results of processing for the case where all features constituting the small region are equivalent, which are stored in the pre-computation result table, as the result of processing the small region.

[0124] Furthermore, if the processing unit 32 determines that the small regions are consecutive and identical small regions, the arithmetic circuit 18 does not perform convolution processing on the small regions, and outputs the results of processing on the subregion processed immediately before as the results of processing the small region.

[0125] The output feature map, which is the result of the convolution process performed for each small region as described above, is output again for each small region into which the output feature map is divided to the RAM 13. At this time, for each small region, it is determined whether the small region is a small region of the same value and whether the small regions are consecutive and the same small regions, and the determination result for each small region and the compressed output feature map in which the small regions determined to be a small region of the same value or consecutive and the same small regions are compressed are output to the RAM 13.

[0126] The processing unit 42 of the inference unit 22 processes the target image using a neural network including convolution processing, similar to the processing unit 32. When outputting an output feature map that is the output of the convolution processing, the processing unit 42 outputs the output feature map to the RAM 13 for each divided small region. When outputting each small region to the RAM 13, the processing unit 42 judges whether the small regions are small regions of the same value and whether they are consecutive and the same small regions. The judgment result for each small region and the compressed output feature map in which the small regions judged to be small regions of the same value or consecutive and the same small regions are compressed are output to the RAM 13.

[0127] Furthermore, similar to the processing unit 32, when performing the convolution process, the processing unit 42 reads out the immediately previous compressed output feature map from the RAM 13 and expands it into an input feature map that is to be the input for the convolution process.

[0128] Specifically, the processing unit 42 expands the previous compressed output feature map read from the RAM 13 into small region data for each small region including an overlap region that overlaps with adjacent small regions, and sets the small region data for each small region obtained by dividing the input feature map that is the input for the convolution process. The processing unit 42 also performs convolution processing for each small region obtained by dividing the input feature map that is the input for the convolution process. The convolution processing for each small region is performed using the arithmetic circuit 18. At this time, the processing unit 42 judges, for each small region, whether or not the small regions are of the same value, and whether or not the small regions are consecutively the same small regions. At this time, based on the equivalence flag and consecutive flag of the small region read from the RAM 13, the processing unit 42 judges whether or not the small regions are of the same value, and whether or not the small regions are consecutively the same small regions. Furthermore, the small region data of the input feature map, the kernel, the judgment results for each small region, and the pre-computation result table are input to the arithmetic circuit 18, and the arithmetic circuit 18 outputs the output feature map.

[0129] Furthermore, like the processing unit 32, when the processing unit 42 determines that the small region is an equivalent small region, the arithmetic circuit 18 does not perform convolution processing on the small region, and outputs the result of processing for the case where all features constituting the small region are equivalent, which is stored in the pre-computation result table, as the result of processing the small region.

[0130] Furthermore, if processing unit 42, like processing unit 32, determines that the sub-regions in question are consecutive and identical sub-regions, arithmetic circuit 18 does not perform convolution processing on the sub-region, and outputs the results of processing on the sub-region processed immediately before as the results of processing the sub-region in question.

[0131] The output feature map, which is the result of the convolution process performed for each small region as described above, is output again for each small region into which the output feature map is divided to the RAM 13. At this time, for each small region, it is determined whether the small region is a small region of the same value and whether the small regions are consecutive and the same small regions, and the determination result for each small region and the compressed output feature map in which the small regions determined to be a small region of the same value or consecutive and the same small regions are compressed are output to the RAM 13.

[0132] <Action of the image processing device according to the third embodiment> Next, the operation of the image processing device 310 according to the third embodiment will be described.

[0133] The CPU 11 of the image processing device 310 reads out a learning processing program from the ROM 12 or the storage 14, deploys it in the RAM 13, and executes it to perform processing similar to the learning processing shown in Fig. 7. In addition, learning data is input to the image processing device 310.

[0134] In step S102, arithmetic processing is performed for each layer of the neural network. Here, the arithmetic processing for the convolution layer is realized by the processing routine shown in FIG.

[0135] At this time, the output feature map shown in FIG. 15 is obtained in the computation process of the previous convolutional layer, and the compressed output feature map is output to the RAM 13 together with the determination results for each small region as to whether or not the small regions have the same value and whether or not the small regions are consecutive and the same.

[0136] In the example of the output feature map shown in FIG. 15 above, the equality flags and consecutive flags are as follows: Equivalence flags: 0,0,1,0,0,0 Sequential flags: 0,0,0,0,1,0

[0137] The compressed output feature map has a total of 33 values, as shown below. Note that " / " is the separator between small region data. 0,0,0,0,4,9,9,4 / 0,0,0,4,4,9,9,4 / 4 / 4,9,9,4,3,8,8,3 / / 4,4,4,4,3,7,7,7

[0138] The data size output to the RAM 13 is 1 bit*12+4 bit*33=144 bits. Since the data size without compression is 4 bit*48=192 bits, the data size is reduced by compressing the output feature map.

[0139] In step S200, the CPU 11, functioning as the processing unit 32, reads out the previous compressed output feature map from the RAM 13, and expands it into an input feature map that is to be input for the convolution process.

[0140] For example, the input feature map expanded from the above equality flags, consecutive flags, and compressed output feature map has a total of 48 values, as shown below. 0,0,0,0,4,9,9,4 / 0,0,0,4,4,9,9,4 / 4,4,4,4,4,4,4,4,4,4 / 4,9,9,4,3,8,8,3 / 4,9,9,4,3,8,8,3 / 4,4,4,4,3,7,7,7

[0141] In step S110, the CPU 11, functioning as the processing unit 32, divides an input feature map, which is to be the input of the convolution layer, into small regions.

[0142] For example, the image is divided into sub-regions having adjacent sub-regions and overlapping regions as follows: 0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,9,9,4,4,4,4,9,9,4,4 / 0,0,0,0,0,0,0,0,4,4,4,9,9,4,4,4,9,9,4,4,4,4,9,4,4 / 4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4 / 4,4,9,9,4,4,4,4,9,9,4,4,3,3,8,8,3,3,3,3,8,8,3,3 / 4,4,9,9,4,4,4,4,9,9,4,4,3,3,8,8,3,3,3,3,8,8,3,3 / 4,4,4,4,4,4,4,4,4,4,4,4,4,4,3,3,7,7,7,7,3,3,7,7,7,7

[0143] In step S112, the CPU 11, functioning as the processing unit 32, reads out from the RAM 13 the pre-computation result table of the convolution layer.

[0144] In step S114, the CPU 11, as the processing unit 32, processes each divided small area in turn, and determines, based on the equivalence flag and the continuity flag, whether the features constituting the small area to be processed are equivalent or the same as the features of the small area processed immediately before.

[0145] In step S116, the CPU 11, as the processing unit 32, outputs each small region data of the input feature map, the pre-calculation result table, and the equality flag and consecutive flag indicating the determination result of step S114 to the arithmetic circuit 18. Then, the arithmetic circuit 18 performs a convolution process for each small region.

[0146] At this time, if the small regions to be processed are not small regions of the same value and are not consecutively the same small regions, the arithmetic circuit 18 performs convolution processing on the small regions to be processed. If the small regions to be processed are small regions of the same value, the arithmetic circuit 18 does not perform convolution processing on the small regions to be processed, and outputs the results of processing for the cases where the features constituting the small regions are the same value, stored in the pre-computation result table, as the results of processing the small regions to be processed.

[0147] Furthermore, if the small regions to be processed are consecutive identical small regions, the arithmetic circuit 18 does not perform convolution processing on the small regions to be processed, but outputs the results of processing on the small region processed immediately before as the results of processing the small regions to be processed.

[0148] In step S202, the CPU 11, as the processing unit 32, divides an output feature map consisting of the processing results for each small region into small regions each having an adjacent small region and an overlapping region. Then, the CPU 11, as the processing unit 32, processes each divided small region in order, and determines whether the features constituting the small region to be processed are the same as or the same as the features of the small region processed immediately before.

[0149] In step S204, the CPU 11, as the processing unit 32, performs compression processing of the output feature map using the determination result of step S202, outputs the compressed output feature map to the RAM 13 together with the equivalence flag and continuity flag indicating the determination result of step S202, terminates the processing routine, and proceeds to the calculation processing of the next layer.

[0150] The CPU 11 of the image processing device 310 reads out an image processing program from the ROM 12 or the storage 14, deploys it in the RAM 13, and executes it to perform processing similar to the image processing shown in Fig. 9. A target image is input to the image processing device 310.

[0151] In step S122, arithmetic processing is performed for each layer of the neural network. Here, the arithmetic processing for the convolution layer is realized by the processing routine shown in FIG.

[0152] Other configurations and operations of the image processing device 310 of the third embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0153] As described above, when outputting an output feature map that is the output of convolution processing, the image processing device according to the third embodiment outputs the output feature map to the RAM for each divided small region, and when outputting each small region to the RAM, compresses and outputs to the RAM small regions that have the same value or consecutive identical small regions. This makes it possible to reduce the data size when outputting the output feature map.

[0154] In addition, the unit of the small region for determining whether or not the small regions are the same in value or the same in succession may be a small region without overlapping regions, but since the unit of the determination is a small region with overlapping regions according to the kernel size of the convolution processing of the next layer, the area of ​​the determination unit is larger when the unit of the determination is a small region with overlapping regions than when the unit of the determination is a small region without overlapping regions. Since the smaller the area of ​​the determination unit, the more likely it is that the small regions are the same in value or continuous, the data compression performance may be degraded when the unit of the determination is a small region with overlapping regions than when the unit of the determination is a small region without overlapping regions. On the other hand, when outputting the output feature map, if the small regions without overlapping regions are used as units to determine whether or not the small regions are the same in value or the same in succession, when performing the convolution processing, it is necessary to further determine whether or not the small regions are the same in value or the same in succession using the small region with overlapping regions according to the kernel size of the convolution processing of the next layer as units. In this embodiment, a small area having an overlapping area according to the kernel size of the convolution processing of the next layer is used as a unit of judgment, so that additional judgment processing or flag data is not required, and simple compression processing is possible.

[0155] In this way, by compressing the output feature map using the determination results for each small region in order to reduce the amount of calculation in the convolution process, it becomes possible to compress the output feature map without adding any processing or data to the first embodiment described above, thereby suppressing wait times for data transfer and speeding up the entire image processing using a neural network.

[0156] In addition, increasing the size of the small regions leads to suppression of deterioration in compression performance caused by setting the unit of judgment to small regions with overlapping regions. For example, if M is the width of the small region without overlapping regions, N is the height of the small region without overlapping regions, and k is the kernel size (k=3 for a 3x3 kernel), the area ratio of the area of ​​the small region without overlapping regions to the area of ​​the small region without overlapping regions is expressed as ((M*N) / ((M+k-1)*(N+k-1))). Since this ratio approaches 1 as M and N become larger, the difference between including and not including overlapping regions becomes smaller as the size of the small regions becomes larger, and the difference in compression performance also becomes smaller. It is also possible to determine appropriate values ​​of M and N by simulating data compression performance for the target neural network model with different values ​​of M and N in advance.

[0157] The present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

[0158] For example, in the above third embodiment, the case where it is determined whether or not consecutive small regions are the same is described as an example, but the present invention is not limited to this. For example, it may be determined whether or not a small region is the same as a small region output several times ago, and if it is the same as a small region processed several times ago, data reduction may be performed without outputting data of the small region to the RAM. For example, in addition to the same flag indicating whether or not it is the same as a small region output several times ago, a parameter of a small region interval indicating how many times ago it was the same as the small region may also be used to determine which small region it is the same as. This is expected to further improve compression performance.

[0159] In the above embodiments, the case where it is determined whether or not a small region is a small region of the same value has been described as an example, but the present invention is not limited to this. It may be determined whether or not a small region is identical to a predetermined feature pattern other than a small region of the same value. For example, in the convolution process, a processing result for a predetermined frequent feature pattern may be obtained in advance, and it may be determined whether or not a small region is identical to the predetermined frequent feature pattern, and if the small region is identical to the predetermined frequent feature pattern, the previously obtained processing result may be output without performing the convolution process.

[0160] In addition, when outputting the output feature map, a representative value for a predetermined frequently occurring feature pattern may be determined in advance, and it may be determined whether or not the small region is identical to the predetermined frequently occurring feature pattern. If the small region is identical to the predetermined frequently occurring feature pattern, it may be compressed to the representative value and output to the RAM.

[0161] In addition, the image processing device has been described as having a learning unit and an inference unit, but is not limited thereto. The device having the learning unit and the device having the inference unit may be configured as separate devices. When there are significant hardware constraints such as power and size, it is preferable to configure the device having the learning unit and the device having the inference unit as separate devices. For example, the device having the inference unit is mounted on a drone or used as an IoT or edge device. Also, configuring one device to have a learning unit and an inference unit is generally used when performing high-speed learning using hardware located in a data center, such as cloud computing.

[0162] In addition, various processes that the CPU reads and executes software (programs) in the above embodiment may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and dedicated electric circuits that are processors having a circuit configuration designed exclusively for executing specific processes, such as ASICs (Application Specific Integrated Circuits). Furthermore, the learning process and image processing may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). Moreover, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0163] In addition, in each of the above embodiments, the learning processing program and the image processing program are described as being stored (installed) in advance in the storage 14, but the present invention is not limited to this. The programs may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network.

[0164] The following supplementary notes are further disclosed regarding the above embodiment.

[0165] (Additional note 1) An image processing device including a neural network including a convolution process for an image, Memory, at least one processor coupled to the memory; Including, The processor, Acquire a target image to be processed; Processing the target image using a neural network including the convolution process; When outputting an output feature map which is an output of the convolution processing, the output feature map is output to a storage unit for each divided small region; When outputting each of the small regions to the storage unit, if a feature constituting the small region is the same as a predetermined feature or a feature of a small region output in the past, the predetermined feature or the feature of the small region output in the past is compressed and output to the storage unit. Image processing device.

[0166] (Additional note 2) A non-transitory storage medium storing a program executable by a computer including a neural network including a convolution process for an image to perform image processing, The image processing includes: Acquire a target image to be processed; Processing the target image using a neural network including the convolution process; When outputting an output feature map which is an output of the convolution processing, the output feature map is output to a storage unit for each divided small region; When outputting each of the small regions to the storage unit, if a feature constituting the small region is the same as a predetermined feature or a feature of a small region output in the past, the predetermined feature or the feature of the small region output in the past is compressed and output to the storage unit. Non-transitory storage medium. [Explanation of symbols]

[0167] 10, 210, 310 Image processing device 11 CPU 13 RAM 18 Arithmetic circuit 20 Learning Department 22 Reasoning part 30, 40 Acquisition Department 32, 42 Processing section 34 Update Department

Claims

1. An image processing device including a neural network including a convolution process for an image, an acquisition unit for acquiring a target image to be processed; a processing unit that processes the target image using a neural network including the convolution processing, When outputting an output feature map that is an output of the convolution process, the processing unit outputs the output feature map to a storage unit for each divided small region; When outputting each of the small regions to the storage unit, if a feature constituting the small region is the same as a predetermined feature or a feature of a small region output in the past, the predetermined feature or the feature of the small region output in the past is compressed and output to the storage unit. Image processing device.

2. The processing unit includes: When performing the convolution process using a neural network including the successive convolution processes, reading out an output feature map of the previous convolution processing from the storage unit, and performing the convolution processing for each small region obtained by dividing an input feature map that is an input of the convolution processing; When performing the convolution process for each small region, if the features constituting the small region are the same as predetermined features or features of a small region previously processed, the convolution process is not performed for the small region, and the result of processing the predetermined features or the result of processing the small region previously is output as the result of processing the small region.

2. The image processing device according to claim 1.

3. The processing unit includes:

3. The image processing device according to claim 1, wherein a small region that overlaps with an adjacent small region and has an overlap region of a size corresponding to a kernel size of the subsequent convolution processing is treated as a small region obtained by dividing the output feature map, and a determination is made as to whether the features constituting the small region are the same as the predetermined features or the features of a small region previously output.

4. 4. The image processing apparatus according to claim 1, wherein the predetermined characteristics are the same characteristics within the small regions.

5. An image processing method in an image processing device including a neural network including a convolution process for an image, comprising: The acquisition unit acquires a target image to be processed, The processing unit processes the target image using a neural network including the convolution process; When the processing unit outputs an output feature map which is an output of the convolution processing, the processing unit outputs the output feature map to a storage unit for each divided small region; When outputting each of the small regions to the storage unit, if a feature constituting the small region is the same as a predetermined feature or a feature of a small region output in the past, the predetermined feature or the feature of the small region output in the past is compressed and output to the storage unit. Image processing methods.

6. A computer including a neural network that includes a convolution process for an image is provided. Acquire a target image to be processed; An image processing program for executing processing of the target image using a neural network including the convolution processing, When outputting an output feature map which is an output of the convolution processing, the output feature map is output to a storage unit for each divided small region; When outputting each of the small regions to the storage unit, if a feature constituting the small region is the same as a predetermined feature or a feature of a small region output in the past, the predetermined feature or the feature of the small region output in the past is compressed and output to the storage unit. Image processing program.

Images