Image processing device and image processing method

Abstract

To provide an image processing device, an image processing method and a computer program which reduce a processing time by making reference to pixels in a feature image while skipping pixels in the case of performing a convolution operation.SOLUTION: In an image processing device, a convolutional neutral network (CNN) processing unit comprises a processing unit 406 as operation means for acquiring a pixel from a feature image and performing a convolution operation based on the acquired pixel. The processing unit 406 can acquire pixels not neighboring each other from a feature image when acquiring pixels from a feature pixel.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to convolution processing. [Background technology]

[0002] In recent years, advances in deep learning have improved the accuracy of image recognition. Convolutional Neural Networks (CNN) are known as a method used in deep learning.

[0003] In a CNN, multiple layers are connected hierarchically, and multiple feature images exist within each layer. Figure 2 shows an example of a CNN configuration with four layers, each containing four feature images. In a CNN, convolution processing is performed using learned coefficients (weighting coefficients) and the pixels of the feature images (feature data). The convolution processing is a product-sum operation, and includes multiple multiplications and cumulative additions. The arrows in Figure 2 represent product-sum operations.

[0004] The feature image in the current layer is calculated using the feature image in the previous layer and the coefficients corresponding to the previous layer. To calculate one feature image in the current layer, information on multiple feature images in the previous layer is required. The product-sum operation to obtain the feature image in the current layer is performed according to the following (Equation 1).

[0005]

number

[0006] Here, n is the index of the feature image in the current layer, and m (m = 1 to M) is the index of the feature image in the previous layer. i,j (n) indicates the feature data (product-sum operation result) at position (i, j) in the feature image with index = n in the current layer. i,j (m) indicates the feature data at position (i, j) in the feature image I(m) with index = m in the previous layer. x,y(m, n) indicates the coefficient between the feature image of index n in the current layer and the feature data at position (x, y) in the feature image of index m in the previous layer. In (Equation 1), the coefficient (C 0,0 (m,n)~C X-1,Y-1 There are (X×Y) (m,n) elements, which differ depending on the feature image. X and Y are variables indicating the reference range. The number of product-sum operations required to calculate the feature data of the current layer is (M×X×Y).

[0007] After performing the above product-sum operation (filtering), the product-sum operation result O is calculated based on the CNN network structure. i,j (n) is used to perform activation processing, pooling, etc. to obtain the feature image of the current layer.

[0008] CNNs are also used in applications such as image segmentation. Dilated convolution, described in Non-Patent Document 1, is a technology for improving the accuracy of image segmentation with low computational cost. When performing dilated convolution, a product-sum operation is performed according to the following (Equation 2).

[0009]

number

[0010] Here, the variable R is the magnification of the dilated convolution process. When the variable R is 1, (Equation 2) becomes the same as (Equation 1). The larger the value of the variable R, the wider the reference range in the feature image of the previous layer. After dilation, the reference range changes from (X × Y) to [R × (X-1) + 1] × [R × (Y-1) + 1]. In this calculation, no coefficients are skipped, but because every (R-1) pieces of feature data in the feature image are processed, feature data in the horizontal or vertical direction is skipped when referenced.

[0011] The product-sum operation in which the coefficient (weighting coefficient) C used in the product-sum operation (Equation 2) in the filter processing is replaced with the coefficient C' in the expanded (extended) filter of Equation 4 described later is shown in Equation 3 below.

[0012]

number

[0013] The variables in equation (3) that are common to equations (1) and (2) are as described above, and therefore will not be described again. x,y (m, n) is shown in the following equation (4).

[0014]

number

[0015] If x and y are multiples of R, the coefficient C' x,y The value of (m,n) is the coefficient C x / R,y / R It is the same as (m, n) and is a significant value (effective coefficient). On the other hand, if x, y are not multiples of R, the coefficient C' x,y The value of (m,n) is 0, which means that the calculation is omitted.

[0016]

number

[0017] is a floor function that outputs the largest integer less than or equal to X. Because CNNs perform a large number of product-sum operations, when they are applied to embedded systems such as mobile terminals and in-vehicle devices, it is necessary to reduce the amount of feature data and coefficients transferred, perform product-sum operations efficiently, and shorten the overall processing time. Patent Documents 1 and 2 propose a configuration that processes multiple feature data in parallel. [Prior art documents] [Patent documents]

[0018] [Patent Document 1] US2020 / 0410036 [Patent Document 2] US10861123 [Non-patent literature]

[0019] [Non-Patent Document 1] Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, and Yizhou Yu. FastFCN: Rethinking dilated convolution in the backbone for semantic segmentation, 2019. CoRR, abs / 1903.11816. Summary of the Invention [Problem to be solved by the invention]

[0020] Dilated convolution can achieve high recognition accuracy with low computational cost, but since the coordinates of the referenced data and memory addresses are not continuous, there is a possibility that the processing efficiency of the hardware will decrease.

[0021] In the method described in Patent Document 1, output data is calculated in parallel using a systolic array configuration. Although it is possible to perform the dilated convolution process described in Non-Patent Document 1 by using the zero coefficients shown in (Equation 4) while referring to the network parameters, it is not possible to skip and refer to the feature data of the feature image I(m).

[0022] In the method described in Patent Document 2, output data is calculated in parallel using multiple different feature data and common coefficients. When performing the dilated convolution process described in Non-Patent Document 1, it is possible to use the zero coefficients shown in (Equation 4). If each pixel and each coefficient is read out in a single stroke, the dilated convolution process can be performed efficiently. However, since it is not possible to skip and reference the feature data of the feature image I(m), it is not possible to omit the processing of all zero coefficients. The present invention provides a technology for reducing processing time by skipping and referencing pixels in the feature image when performing convolution calculations. [Means for solving the problem]

[0023] In one aspect of the present invention, a feature image A specific pixel storage unit among a group of two-dimensionally arranged pixel storage units having a memory in which pixel information for each pixel is stored and a multiplexer. from The aforementioned pixel information a first acquisition means for acquiring the The pixel acquired by the first acquisition means information A calculation means for performing a convolution calculation based on Equipped with the pixel storage unit is connected to a pixel storage unit storing adjacent pixels in the feature image and a pixel storage unit storing non-adjacent pixels in the feature image, and pixel information can be shifted between the pixel storage units by selecting the multiplexer; when the first acquisition means acquires pixel information that is non-adjacent to the acquired pixel information in the feature image, the multiplexer of the specific pixel storage unit is selected to shift pixel information from the pixel storage unit that stores the non-adjacent pixel to the specific pixel storage unit; When the first acquisition means acquires pixel information adjacent to the acquired pixel information in the feature image, the multiplexer of the specific pixel storage unit is selected to shift pixel information from the pixel storage unit storing the adjacent pixel to the specific pixel storage unit. It is characterized by: [Effects of the Invention]

[0024] According to the present invention, when performing a convolution operation, the processing time can be reduced by referencing pixels in the feature image while skipping them. [Brief explanation of the drawings]

[0025] [Figure 1] 13 is a flowchart of the operation of the CNN processing unit 1305. [Figure 2] A diagram showing an example of the configuration of a CNN. [Figure 3] FIG. 4 is a block diagram showing an example of the configuration of a processing unit 406. [Figure 4] FIG. 13 is a block diagram showing an example of the configuration of a CNN processing unit 1305. [Figure 5] FIG. 3 shows an example of the configuration of a processing unit 309. [Figure 6] FIG. 3 is a diagram showing an example of the configuration of a storage unit 306. [Figure 7] FIG. 3 is a diagram showing an example of the configuration of a storage unit 305. [Figure 8] FIG. 10 is a diagram showing the correspondence between pixels and positions. [Figure 9] FIG. 10 is a diagram showing the correspondence between filter coefficients and positions. [Figure 10] 10A and 10B are diagrams showing pixel transfer patterns and directions, and coefficient transfer patterns and directions. [Figure 11] 10A and 10B are diagrams showing examples of pixel transfer and coefficient transfer. [Figure 12] A diagram showing an example of CNN and convolution processing. [Figure 13] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus. [Figure 14] FIG. 10 is a diagram showing pixel sets, coefficient filters, and convolution processing results. [Figure 15] 10 is a flowchart showing details of the process in step S108. [Figure 16] FIG. 3 is a block diagram showing an example of the configuration of a storage unit 306. [Figure 17] FIG. 10 is a diagram showing an example of an extended coefficient filter. [Figure 18] FIG. 1 shows an example of applying cascaded dilated convolution processing to a convolutional neural network. [Figure 19] FIG. 10 is a diagram showing an example of cascaded coefficient filters for dilated convolution processing. [Figure 20] FIG. 1 is a diagram showing an example of a convolutional neural network for image segmentation processing. [Figure 21] FIG. 13 is a block diagram showing an example of the configuration of a CNN processing unit 1305. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0027] [First embodiment] First, an example of the hardware configuration of an image processing device that performs convolution operations in a hierarchical neural network (such as the above-mentioned CNN) will be described using the block diagram of FIG. 13. Computer devices such as PCs (personal computers), tablet terminal devices, and smartphones can be applied to the image processing device according to this embodiment. Furthermore, the configuration of the image processing device is not limited to the configuration shown in FIG. 13, and any configuration may be adopted as long as it is capable of performing the processes described below as being performed by the image processing device.

[0028] The input unit 1301 is a user interface such as a keyboard, a mouse, or a touch panel screen, and can be operated by a user to input various instructions to the CPU 1306. Note that the input unit 1301 is not limited to being included in the image processing apparatus.

[0029] The data storage unit 1302 is a large-capacity information storage device such as a hard disk drive. The data storage unit 1302 stores an OS (operating system) and computer programs and data for causing the CPU 1306 to execute or control each process described as being performed by the image processing device. The computer programs and data stored in the data storage unit 1302 are loaded into the RAM 1308 as appropriate under the control of the CPU 1306, and become the subject of processing by the CPU 1306.

[0030] The data storage unit 1302 may be a memory device such as a flexible disk, CD-ROM, CD-R, DVD, memory card, CF card, smart media, SD card, memory stick, xD picture card, or USB memory.

[0031] The data storage unit 1302 does not necessarily have to be included in the image processing apparatus, but may be provided on a wired and / or wireless network such as a LAN or the Internet. In this case, the image processing apparatus accesses the data storage unit 1302 via the communication unit 1303 to read and write computer programs and data from and to the data storage unit 1302. The communication unit 1303 performs data communication with external devices via the network.

[0032] The display unit 1304 has a liquid crystal screen or a touch panel screen, and displays the processing results of the CPU 1306 as images, text, etc. Note that the display unit 1304 is not limited to being included in the image processing device. The display unit 1304 may also be a projection device such as a projector that projects images and text. The input unit 1301 and the display unit 1304 may also be integrated to form a touch panel screen.

[0033] The CNN processing unit 1305 inputs the image to be processed (input image) stored in the RAM 1308 into the input layer of the CNN, performs arithmetic processing in each layer of the CNN, and stores the results of the arithmetic processing in the RAM 1308 or the data storage unit 1302.

[0034] The CPU 1306 executes various processes using computer programs and data stored in the RAM 1308 and the ROM 1307. As a result, the CPU 1306 controls the operation of the entire image processing device, and executes or controls each process that will be described as being performed by the image processing device.

[0035] The method for storing computer programs and data in RAM 1308 is not limited to a specific method, and for example, computer programs and data received from an external device via communication unit 1303 may be temporarily stored in data storage unit 1302 and then stored in RAM 1308. Furthermore, computer programs and data received from an external device via communication unit 1303 may be directly stored in RAM 1308.

[0036] The CPU 1306 can perform various processes related to the input image using the "results of arithmetic processing on the input image" stored in the RAM 1308 or the data storage unit 1302 by the CNN processing unit 1305. Examples of the "various processes related to the input image" include face detection from the input image, face recognition in the input image, and counting the number of faces in the input image. The CPU 1306 stores the results of various processes including such "various processes related to the input image" in the RAM 1308 or the data storage unit 1302. Note that although the number of CPUs 1306 is one in FIG. 13, there may be two or more.

[0037] The ROM 1307 stores setting data for the image processing apparatus, computer programs and data related to the startup of the image processing apparatus, computer programs and data related to the basic operation of the image processing apparatus, and the like.

[0038] The RAM 1308 has an area for storing computer programs and data loaded from the data storage unit 1302 or the ROM 1307, and an area for storing the results of arithmetic processing output from the CNN processing unit 1305. The RAM 1308 also has an area for storing computer programs and data received from an external device via the communication unit 1303, and a work area used when the CPU 1306 or the image processing unit 1309 executes various processes. In this way, the RAM 1308 can provide various areas as needed.

[0039] When an image processing unit 1309 receives an image processing instruction from the CPU 1306, it performs image processing such as adjusting the range of pixel values of each pixel of the image on the image stored in the RAM 1308 or data storage unit 1302.

[0040] The input unit 1301 , data storage unit 1302 , communication unit 1303 , display unit 1304 , CNN processing unit 1305 , CPU 1306 , ROM 1307 , RAM 1308 , and image processing unit 1309 are all connected to a system bus 1310 .

[0041] Next, an example of the configuration of a CNN will be explained using Figure 2. The CNN shown in Figure 2 has four layers (Layer 1, Layer 2, Layer 3, Layer 4). Layer 1 has four feature images (feature images (1,1), (1,2), (1,3), (1,4)), Layer 2 has four feature images (feature images (2,1), (2,2), (2,3), (2,4)), Layer 3 has four feature images (feature images (3,1), (3,2), (3,3), (3,4)), and Layer 4 has four feature images (feature images (4,1), (4,2), (4,3), (4,4)).

[0042] In layer 1, the product-sum operation of (Equation 2) using a coefficient filter is performed on each of the feature images (1,1), (1,2), (1,3), and (1,4), to generate the feature images (2,1), (2,2), (2,3), and (2,4)) of layer 2 as output feature images.

[0043] In layer 2, the feature images (2,1), (2,2), (2,3), and (2,4) are each subjected to a product-sum operation using a coefficient filter (Equation 2) to generate the feature images (3,1), (3,2), (3,3), and (3,4)) of layer 3 as output feature images.

[0044] In layer 3, the feature images (3,1), (3,2), (3,3), and (3,4) are each subjected to a product-sum operation using a coefficient filter (Equation 2) to generate the feature images (4,1), (4,2), (4,3), and (4,4)) of layer 4 as output feature images.

[0045] Furthermore, the expansion convolution processing magnification R in layer 1 is set to 1, the expansion convolution processing magnification R in layer 2 is set to 2, and the expansion convolution processing magnification R in layer 3 is set to 4. The size of the coefficient filter before expansion is 3x3, and in each layer, a product-sum operation is performed with the feature image using an expansion coefficient filter obtained by expanding the coefficient filter before expansion according to the expansion convolution processing magnification corresponding to that layer.

[0046] An example of extending a coefficient filter in each layer is shown on the left side of Fig. 17. In layer 1, as shown in a frame 1701, a product-sum operation of (Equation 2) is performed on a feature image using an extended coefficient filter (coefficient filter C'x,y on the right side) obtained by extending the coefficient filter before extension (coefficient filter Cx,y on the left side) according to an extension convolution processing magnification R=1. When R=1, there is no change in size between the coefficient filter before extension and the coefficient filter after extension.

[0047] In layer 2, as shown in a frame 1702, a product-sum operation of (Equation 2) is performed on the feature image using an expanded coefficient filter (right-side coefficient filter C'x,y) obtained by expanding the coefficient filter before expansion (left-side coefficient filter Cx,y) according to an expansion convolution processing magnification R=2. The expanded coefficient filter is a 5×5 coefficient filter obtained by inserting one idle coefficient (for example, a coefficient of "0") between the coefficients of the coefficient filter before expansion.

[0048] In layer 3, as shown in frame 1703, a product-sum operation of (Equation 2) is performed on the feature image using an expanded coefficient filter (right-side coefficient filter C'x,y) obtained by expanding the coefficient filter before expansion (left-side coefficient filter Cx,y) according to an expanded convolution processing magnification R=4. The expanded coefficient filter is a 9x9 coefficient filter obtained by inserting three idle coefficients between the coefficients of the coefficient filter before expansion. In this way, the expanded convolution processing magnification R differs for each layer in the CNN.

[0049] An example of CNN and convolution processing is shown in Figure 12. Feature data is extracted from the same position in four feature images 1201 in layer 1, and the result of a product-sum operation with a coefficient filter is calculated. This result becomes the feature data for the same position in the next layer, layer 2, and a feature image 1202 containing that feature data is obtained.

[0050] An example configuration of the above-mentioned CNN processing unit 1305 is shown in the block diagram of Fig. 4. The functional units shown in Fig. 4 may be implemented by hardware or by software (computer program). In the latter case, the computer program is stored in the data storage unit 1302. The operation of the CNN processing unit 1305 will be explained with reference to the flowchart in Fig. 1.

[0051] In step S101, the control unit 401 reads out the coefficient filter before expansion, the input feature image (feature image of layer 1 in the example of FIG. 2), and information about the structure of the CNN (structural information) from the data storage unit 1302 or the RAM 1308, and stores them in the data holding unit 408. The structural information includes, for example, the amount of calculation of the product-sum operation, the number of layers, the size of the feature image, the number of feature images in each layer, etc.

[0052] Then, under the control of the control unit 401, the processes of steps S103 to S111 are performed for each layer in the CNN. In the example of Fig. 2, first, layer 1 is set as the target layer, and the processes of steps S103 to S111 are performed for layer 1. Next, layer 2 is set as the target layer, and the processes of steps S103 to S111 are performed for layer 2. Next, layer 3 is set as the target layer, and the processes of steps S103 to S111 are performed for layer 3.

[0053] In step S103, the reading unit 405, under the control of the control unit 401, acquires the dilated convolution processing magnification R corresponding to the target layer from the data storage unit 408. If the target layer is layer 1, the reading unit 405 acquires "1" as the dilated convolution processing magnification R; if the target layer is layer 2, the reading unit 405 acquires "2" as the dilated convolution processing magnification R; and if the target layer is layer 3, the reading unit 405 acquires "4" as the dilated convolution processing magnification R.

[0054] In this way, the expansion convolution processing magnification R can be set for each layer. Note that the expansion convolution processing magnification does not have to be set for each layer, and may be set for each output feature image or for each group of output feature images, for example.

[0055] Then, under the control of the control unit 401, the processes of steps S105 to S111 are performed for each output feature image generated in the target layer (the feature image of the next layer (the layer that will become the next target layer) following the target layer). In the example of FIG. 2, when the target layer is layer 1, the processes of steps S105 to S111 are first performed to generate a feature image (2,1), and then the processes of steps S105 to S111 are performed to generate a feature image (2,2). Similarly, the processes of steps S105 to S111 are next performed to generate a feature image (2,3), and then the processes of steps S105 to S111 are performed to generate a feature image (2,4). In step S105, the control unit 401 initializes the convolution operation result stored in the storage unit 503 (FIG. 5) of the processing unit 406 to 0.

[0056] Then, under the control of the control unit 401, the processes of steps S107 to S108 are performed for each feature image in the target layer. In the example of Fig. 2, when the target layer is layer 1, first, feature image (1,1) is set as the target feature image, and steps S107 to S108 are performed for this target feature image. Next, feature image (1,2) is set as the target feature image, and steps S107 to S108 are performed for this target feature image. Next, feature image (1,3) is set as the target feature image, and steps S107 to S108 are performed for this target feature image. Next, feature image (1,4) is set as the target feature image, and steps S107 to S108 are performed for this target feature image.

[0057] In step S107, if the target layer is layer 1, the control unit 401 transfers the input feature image stored in the data storage unit 408 to the storage unit 402 as a target feature image, and transfers the pre-extension coefficient filter stored in the data storage unit 408 to the storage unit 404. On the other hand, if the target layer is layer A (A is an integer equal to or greater than 2), the feature image generated by the processing unit 407 for layer A has been transferred to and stored in the storage unit 402 as a target feature image, and the pre-extension coefficient filter has already been transferred to the storage unit 404, so step S107 may be omitted.

[0058] In step S108, under the control of the control unit 401, the processing unit 406 reads the target feature image stored in the storage unit 402, 405 The convolution operation is performed using the extended convolution processing magnification R read out by the storage unit 404 and the coefficient filter held in the holding unit 404. Details of the processing in step S108 will be described later with reference to FIG.

[0059] When the process proceeds to step S110, the convolution operation has been performed on all feature images in the layer to be processed, and the results of the convolution operation (convolution operation results) have been stored in the storage unit 503 (FIG. 5). Therefore, in step S110, under the control of the control unit 401, the processing unit 407 performs activation processing by performing an operation based on the following (Equation 5) based on the respective convolution operation results stored in the storage unit 503.

[0060]

number

[0061] Here, f(x) is the activation function, and x is the result of the convolution operation. Here, the activation function is realized using a ReLU (Rectified Linear Unit), but the activation function is not limited to ReLU and may be realized using other nonlinear functions or quantization functions. In addition, the processing unit 407 performs pooling processing based on the result of the activation processing according to the layer information, and adjusts the size of the output feature image as necessary.

[0062] In step S111, the control unit 401 stores the result of the activation pooling process (output feature image) obtained in step S110 in the storage unit 402 as a feature image of the next layer following the target layer (the layer that will be the next target layer).

[0063] Next, details of the processing in step S108 above will be explained with reference to the flowchart in FIG. 15. An example of the configuration of the processing unit 406 is shown in the block diagram of FIG. 3. In the following, it is assumed that the target feature image is stored in the storage unit 402 so as to be readable in units of pixel blocks (pixel sets) each having a size of 6 pixels x 6 pixels. For example, the control unit 401 divides the target feature image into a plurality of pixel sets (with overlapping portions with adjacent pixel sets), and stores each pixel set in the storage unit 402. In the following, as shown in FIG. 14(a), a pixel (feature data) at position (i, j) (1≦i, j≦6) in the pixel set 1401 is stored as D i、j It is written as follows.

[0064] In the following description, it is assumed that a coefficient filter of 3x3 size is stored in the storage unit 404. As shown in FIG. 14(b), the coefficients at (i, j) (1≦i, j≦3) in the coefficient filter are represented by C i、j It is written as follows.

[0065] In this embodiment, there are four arithmetic units, which multiply pixel values of different pixels by a common coefficient in parallel, and as shown in FIG. 14(c), four convolution processing results (O 1,1 , O 1,2 , O 2,1 , ~O 2,2) are calculated in parallel.

[0066] In the following, the processes of steps S1503 to S1512 are performed for each pixel set in the target feature image. In step S1503, the control unit 311 sets, in the setting unit 301, a function (Equation 8) that defines a pixel transfer pattern, a function (Equation 9) that defines a coefficient transfer pattern, a function (Equation 10) that defines an interval between pixels to be transferred, and a function (Equation 11) that defines an interval between coefficients to be transferred. Equations 8 to 11 will be described later.

[0067] In step S1504, the control unit 311 selects one pixel set from the unselected pixel sets in the target feature image stored in the holding unit 402 as a selected pixel set, reads out the selected pixel set from the holding unit 402, and stores it in the storage unit 306. The control unit 311 also selects a corresponding coefficient filter from the coefficient filters held in the holding unit 404 as a selected coefficient filter, reads out the selected coefficient filter from the holding unit 404, and stores it in the storage unit 305.

[0068] In step S1505, the control unit 311 initializes the processing unit 406 by setting an initial value for the result of the convolution processing and initializing a variable m (the variable m used in this embodiment and subsequent embodiments is different from the variable m used in (Equation 4) and earlier) indicating the processing order of the coefficients to 1. The initial value of the result of the convolution processing is related to the input feature image; when processing the first input feature image, the initial value of the result of the convolution processing is set to 0, and when processing subsequent feature images, the initial value of the result of the convolution processing becomes the processing result of the previous feature image.

[0069] In step S1506, the kernel of the coefficient filter is scanned, and convolution processing (extended convolution processing) is performed by a product-sum operation using the selected pixel set and the selected coefficient filter. In this embodiment, the convolution processing result is obtained by calculating the following (Equation 6) and (Equation 7).

[0070]

number

[0071] Here, i is an index indicating a position in the horizontal direction, and j is an index indicating a position in the vertical direction. Dx(m) is a function that returns the horizontal position corresponding to variable m in the pixel set, and Dy(m) is a function that returns the vertical position corresponding to variable m in the pixel set. Cx(m) is a function that returns the horizontal position corresponding to variable m in the selection coefficient filter. C y(m) is a function that returns the vertical position corresponding to the variable m in the selection coefficient filter. Details of the process in step S1506 will be described later.

[0072] In step S1512, the processing unit 309 calculates the four convolution processing results (O 1,1 , O 1,2 , O 2,1 , O 2,2 ) is stored in the memory 310. The processing unit 407 uses the convolution processing result stored in the memory 310 to perform activation processing and pooling processing.

[0073] Next, the details of the process in step S1506 above will be described. In step S1506, the processes in steps S1508 to S1510 are repeated the number of times equal to the number of coefficients in the selection coefficient filter.

[0074] In step S1508, the transfer control unit 307 transfers the functions of (Equation 8) and (Equation 10) set in the setting unit 301 to the storage unit 306. Then, using these functions transferred to the storage unit 306, the transfer control unit 307 calculates the pixel D at the position (Dx(m), Dy(m)) in the selected pixel set. Dx(m)、Dy(m) The transfer control unit 304 transfers the functions of (Equation 9) and (Equation 11) set in the setting unit 301 to the storage unit 305. Then, using these functions transferred to the storage unit 305, the transfer control unit 304 obtains the coefficient C at the position (Cx(m), Cy(m)) in the selection coefficient filter. Cx(m)、Cy(m) Get.

[0075] An example configuration of storage unit 306 will be described with reference to Figure 6. Storage unit 306 has multiple pixel storage units. There are two types of pixel storage units (pixel storage unit A and pixel storage unit B), each connected to surrounding pixel storage units and arranged two-dimensionally. Pixel storage unit A is connected to processing unit 309, while pixel storage unit B is not connected to processing unit 309. Pixel storage unit A and pixel storage unit B have pixel storage memory for storing pixels and a multiplexer.

[0076] In FIG. 6, the storage unit 306 has 36 pixel storage units and can support processing with an extended convolution processing magnification R of 1 or 2. The configuration of pixel storage units B other than pixel storage unit A in block 601 is shown within dashed line 603. Each pixel storage unit B can store one pixel. Based on the function of (Equation 8) and the function of (Equation 10) stored in the setting unit 301, pixel storage unit B selects pixels stored in the eight pixel storage units surrounding it (one above, one below, one to the left, one to the right, two above, two below, two to the left, and two to the right) and stores them in its own pixel storage memory. Furthermore, one pixel storage unit B can supply pixels in parallel to the eight surrounding pixel storage units. There are four parallel arithmetic units, and because the four pixels processed in parallel are different, there are four pixel storage units A (pixel storage units 1-1, 1-2, 2-1, and 2-2) in block 601. The configuration of pixel storage unit A is shown within dashed line 602. The configuration of pixel storage unit A is the same as the configuration of pixel storage unit B, and one pixel storage unit A can store one pixel. Pixel storage unit A selects pixels from the eight pixel storage units surrounding itself (one above, one below, one to the left, one to the right, two above, two below, two to the left, and two to the right) based on the function of (Equation 8) and the function of (Equation 10) stored in setting unit 301, and stores the pixels in its own pixel storage memory. Furthermore, one pixel storage unit can supply pixels in parallel to the eight surrounding pixel storage units and to the output destination of storage unit 306. When the dilation convolution processing magnification R is 1, "one above, one below, one to the left, and one to the right of itself" is selected as the above-mentioned "surroundings of itself," and when the dilation convolution processing magnification R is 2, "two above, two below, two to the left, and two to the right of itself" is selected as the above-mentioned "surroundings of itself."

[0077] An example configuration of the storage unit 305 will be described with reference to Fig. 7. The storage unit 305 has a plurality of coefficient storage units. There are two types of coefficient storage units (coefficient storage unit A and coefficient storage unit B), each connected to surrounding coefficient storage units and arranged two-dimensionally. Coefficient storage unit A is connected to the processing unit 309, while coefficient storage unit B is not connected to the processing unit 309. Coefficient storage unit A and coefficient storage unit B each have a coefficient storage memory for storing coefficients and a multiplexer.

[0078] In FIG. 7, the storage unit 305 has nine coefficient storage units, and the configuration of the coefficient storage units B other than the coefficient storage unit A surrounded by the dashed line 701 is shown within the dashed line 703. One coefficient storage unit B can store one coefficient. The coefficient storage unit B selects coefficients held in the four surrounding coefficient storage units (one above, one below, one to the left, and one to the right of itself) based on the function of (Equation 9) and the function of (Equation 11) set in the setting unit 301, and stores the selected coefficients in its own coefficient storage memory. In addition, one coefficient storage unit B can supply coefficients in parallel to the four surrounding coefficient storage units. Although there are four parallel arithmetic units, the coefficients processed in parallel are the same, so there is one coefficient storage unit A (coefficient storage unit 1-1) within the dashed line 701. The configuration of the coefficient storage unit A is shown within the dashed line 702. The configuration of the coefficient storage unit A is the same as the configuration of the coefficient storage unit B, and one coefficient storage unit A can store one coefficient. Coefficient storage unit A selects coefficients stored in the four coefficient storage units surrounding it (one above, one below, one to the left, and one to the right) based on the function of (Equation 9) and the function of (Equation 11) stored in setting unit 301, and stores the selected coefficients in its own coefficient storage memory. Also, one coefficient storage unit can supply coefficients in parallel to the four surrounding coefficient storage units and to the output destination of storage unit 305. Here, (Equations 8) to (Equation 11) are shown below.

[0079]

number

[0080] Initial values are set for Dx(1), Dy(1), Cx(1), and Cy(1). If the value of variable m is 1, Dx(1), Dy(1), Cx(1), and Cy(1) are used as Dx(m), Dy(m), Cx(m), and Cy(m), respectively. If the value of variable m is 2 or more, Dx(m), Dy(m), Cx(m), and Cy(m) are calculated according to (Equation 8) to (Equation 11). Ix() and Jx() are shift functions in the horizontal direction, and Iy() and Jy() are shift functions in the vertical direction. P D (), P C () will be discussed later.

[0081] In step S1509, the transfer control unit 307 transfers the four pixels acquired in step S1508 (in this embodiment, four parallel computing units are used, and thus the four pixels are acquired and transferred in parallel) to the processing unit 309. In addition, the transfer control unit 304 transfers one coefficient acquired in step S1508 to the processing unit 309.

[0082] An example configuration of the processing unit 309 will be described using the block diagram of Fig. 5. As shown in Fig. 5, the processing unit 309 has (M x N) sets of multipliers 501, adders 502, and storage units 503, and is capable of parallel processing of product-sum operations using (M x N) pixels and one coefficient. In this embodiment, there are four parallel operation units, and (M x N) is 4. Since the initial value of the processing result was set in step S1505, when variable m is 1, this initial value is stored in the storage unit 503.

[0083] In step S1510, the processing unit 309 performs the product-sum operation (convolution processing) of the above (Equation 6) and (Equation 7) by "multiplying the pixel by the coefficient" and "accumulating the multiplication result" using the multiplier 501 and the adder 502. The processing unit 309 then stores the result of the convolution processing in the storage unit 503. In this embodiment, since the calculation unit is four-parallel, the product (P i,j,m ,i=1,2,j=1,2) are calculated in parallel, and the calculated product is processed as a result (O i,j, i=1,2, j=1,2) can be added in parallel.

[0084] If the value of variable m has not reached the number of coefficients in the coefficient filter (XxY=9) at the end of step S1510, control unit 311 increments the value of variable m by 1 and repeats the processes of steps S1508 to S1510 again. On the other hand, if the value of variable m has reached the number of coefficients in the coefficient filter (XxY=9) at the end of step S1510, the process proceeds to step S1512.

[0085] <Example of dilated convolution processing> Here, a processing example of the sum-of-products operation (step S1506) of one pixel set and one coefficient filter will be described. Before starting the sum-of-products operation, the processing unit 406 is initialized in step S1505. Therefore, the convolution processing result O 1,1 ~O 2,2 The value of variable m is set to 1, and the process proceeds to step S1508.

[0086] P in (Equation 8) D (m), P in (Eq. 9) C Regarding (m), if the value of variable m is 1, P C (m), P D (m), Ix(), I y Without calculating (), set (Dx(1), Dy(1)) = (Cx(1), Cy(1)) = (1,1).

[0087] In step S1508, the transfer control unit 307 transfers pixel D 1,1 and coefficient C 1,1 As shown in FIG. 8, when the value of the variable m is 1, four pixels (D 1,1 ~D 2,2 ) are obtained. The dashed line 802 shows an example of pixels stored in the storage unit 306 of FIG. 6, and the 36 pixel storage units are used to obtain 36 pixels (D 1,1 ~D 6,6 ) is stored.

[0088] As shown in FIG. 9, when the value of the variable m is 1, one coefficient (C 1,1 ) are obtained. The dashed line 902 shows an example of the coefficients stored in the storage unit 305 of FIG. 6, and the nine coefficient storage units store nine coefficients (C 1,1 ~C 3,3 ) is stored.

[0089] Pixel transfer pattern (m, P D (m), Ix(P D (m)),Iy(P D (m)), direction, Dx(m), Dy(m)) and coefficient transfer pattern (m, P C (m), Jx(P C (m)),Jy(P C An example of the direction of the pixel storage units (Cx(m), Cy(m)) and the direction of the pixel storage units (Cx(m), Cy(m)) will be explained using FIG. 10. As shown in FIG. 6, there are four pixel storage units connected to the output, but only the pixel storage unit 1-1 at the top left will be explained here. The pixel storage unit 1-1 stores the output result (O 1,1 ), and the pixel transfer start position is the top left. As shown in Figure 7, the coefficient storage unit connected to the output is coefficient storage unit 1-1. The coefficient transfer start position is the top left.

[0090] An example of pixel transfer and coefficient transfer (convolution processing magnification R=2) will be explained using FIG. 11. When the value of variable m is 1, D 1,1 and C 1,1 Add the product of to the initial value, and use (Equation 6) to get O 1,1 Calculate the O shown in Figure 11. 1,2 , O 2,1 , O 2,2 About O 1,1 Since the results can be obtained in a similar manner to that of step S1501, a description thereof will be omitted. In this embodiment, it is possible to calculate four convolution processing results in parallel. After that, the value of variable m is changed to 2, and the process proceeds to step S1508.

[0091] If the value of variable m is 2 or greater, P D (2) is 0, and Ix(P D (2)) and Iy(P D(Dx(2), Dy(2)) is (2,0). Then, from the above (Equation 8) and (Equation 10), (Dx(2), Dy(2)) is calculated as (3,1). As shown in FIG. 6, there are eight values of the shift function, and the input pixel can be directly selected using the configuration of dashed line 602 or dashed line 603, resulting in fast processing speed.

[0092] P C (2) is 0, and Ix(P C (2)) and Iy(P C (2))) is (1,0). Then, from the above (Equation 9) and (Equation 11), (Cx(2), Cy(2)) is calculated as (2,1). As shown in FIG. 7, there are four types of shift function values, and the coefficients can be directly selected using the configuration of dashed line 702 or dashed line 703, resulting in fast processing speed.

[0093] In step S1508, the transfer control unit 307 transfers pixel D 3、1 The transfer control unit 304 acquires the coefficient C 2、1 As shown in FIG. 8, when the value of the variable m is 2, four pixels (D 2,1 ~D 3,2 ) is obtained as the output pixel. The value of the shift function varies depending on the expansion convolution processing magnification R.

[0094] Dashed line 802 indicates the pixel held in storage unit 306 in Fig. 6 when the expansion convolution processing magnification R is 1. Because the shift direction of the output pixel is to the right, pixel storage unit 1-1 in storage unit 306 selects a pixel from pixel storage unit 2-1 and holds it in the pixel storage memory in Fig. 6. The pixel in pixel storage unit 1-1 is replaced by the pixel in pixel storage unit 2-1, and the state of dashed line 802 changes to the state of dashed line 804.

[0095] Dashed line 808 indicates the pixels stored in storage unit 306 in Fig. 6 when the expansion convolution processing magnification R is 2. As shown in 805 and 806, the shift direction of the output pixel is to the right, so pixel storage unit 1-1 in storage unit 306 selects a pixel from pixel storage unit 3-1 and stores it in the pixel storage memory in Fig. 6. The pixel in pixel storage unit 1-1 is replaced by the pixel in pixel storage unit 3-1, and the state of dashed line 807 changes to the state of dashed line 808.

[0096] In this way, according to this embodiment, it is possible to read out data other than the data adjacent to the read data (data two or more pixels away from the read data in the row direction), which means that non-adjacent pixels can be obtained from the feature image.

[0097] As shown in FIG. 9, when the value of the variable m is 2, one coefficient (C 2,1 ) is obtained as the output coefficient. Processing using the shift function is not related to the dilation convolution processing magnification R.

[0098] Dashed line 902 (corresponding to 901) indicates the coefficients stored in storage unit 305 shown in Fig. 7. Because the shift direction of the output coefficients is to the right, coefficient storage unit 1-1 in storage unit 305 selects the coefficients from coefficient storage unit 2-1 and stores them in the coefficient storage memory in Fig. 8. The coefficients in coefficient storage unit 1-1 are replaced with the coefficients in coefficient storage unit 2-1, and the state of dashed line 902 changes to the state of dashed line 904.

[0099] Coefficient transfer pattern (m, P D (m), Ix(P D (m)),Iy(P D (m)), direction, Dx(m), Dy(m)) and coefficient transfer pattern (m, P C (m), Jx(P C (m)),Jy(P C An example of the direction, Cx(m), Cy(m) and the direction will be explained using FIG. 10. As shown in FIG. 10, the storage unit 1-1 stores the output result (O 1,1), where the pixel shift direction is right. Here, the coefficient shift direction is right.

[0100] An example of pixel transfer and coefficient transfer (convolution processing magnification R=2) will be explained using FIG. 11. When the value of variable m is 2, D 3,1 and C 2,1 The product of D calculated earlier 1,1 and C 1,1 Add to the product of (Equation 6) 1,1 Calculate the O shown in Figure 11. 1,2 , O 2,1 , O 2,2 About O 1,1 Since it can be obtained in the same way as above, the explanation thereof will be omitted.

[0101] Thereafter, the value of the variable m is changed to 3, and the process proceeds to step S1508. As shown in FIGS. 10, 11, and 12, the output result O is calculated based on the transfer pattern using (Equation 6) to (Equation 11). 1,1 ~O 2,2 In this way, multiplication and cumulative addition are repeated. When the value of variable m reaches 9, the output result (O 1,1 , O 1,2 , O 2,1 , O 2,2 ) is stored in memory 310 as a result of the dilation convolution process.

[0102] As described above, in this embodiment, the shift amount and direction of pixels and coefficients are selected based on the transfer pattern, the expansion convolution processing magnification, etc., so that each can be transferred efficiently. Also, by having one pixel storage unit and one coefficient storage unit receive data from eight adjacent units, unnecessary calculations due to zero coefficients are eliminated, and all pixels and coefficients required for the product-sum calculation can be output. Therefore, expansion convolution processing can be performed efficiently.

[0103] [Second embodiment] In the following embodiments, differences from the first embodiment will be described, and unless otherwise specified, the following description will be given assuming that the present embodiment is the same as the first embodiment. In this embodiment, processing is performed at multiple expansion convolution processing magnifications R using hierarchical pixel storage units.

[0104] An example of the configuration of the storage unit 306, which is composed of hierarchical pixel storage units, will be described using the block diagram in FIG. 16. In the example in FIG. 16, there are three layers, with three layer variables: X0, X1, and X2. Pixels can be selected from the pixel storage units of the three layers and output with the same throughput as the configuration in FIG. 6 described in the first embodiment. The multiple pixel storage units have multiple multiplexers. The layers here are unrelated to the layers of the CNN.

[0105] In the first hierarchical layer 1601, there are two types of shift amounts (1 or 0). Pixel storage unit AA 1604 is connected to pixel selection unit AB 1605 at the same position in the second hierarchical layer 1602, and to pixel storage units adjacent to pixel storage unit AA 1604 above, below, left, and right. When pixel selection unit AB 1605 at the same position in the second hierarchical layer 1602 is selected, the value of variable X0 of the layer becomes 0; otherwise, the value of variable X0 becomes 1.

[0106] In the second hierarchical layer 1602, there are two types of shift amounts (2 or 0). Pixel selection unit AB1605 is connected to pixel selection unit AC1606 at the same position in the third hierarchical layer 1603, and to pixel selection units that are not adjacent to pixel selection unit AB1605 in the vertical or horizontal directions. When pixel selection unit AC1606 at the same position in the third hierarchical layer 1603 is selected, the value of variable X1 of the layer becomes 0, but otherwise the value of variable X1 becomes 1.

[0107] In the third layer 1603, there are two types of shift amounts (4 or 0). Pixel selection unit AC1606 is connected to pixel storage unit AA1604 at the same position in the first layer 1601, and to pixel selection units that are not adjacent to pixel selection unit AC1606 in the vertical or horizontal directions. If pixel storage unit AA1604 at the same position in the first layer 1601 is selected, the value of variable X2 of the layer becomes 0; otherwise, the value of variable X2 becomes 1.

[0108] When Ix() and Iy() are calculated using Equation 10, the expansion convolution processing magnification R is calculated using Equation 12 below.

[0109]

number

[0110] According to this, the maximum value of the dilation convolution processing magnification R is 7, and seven types of dilation convolution processing magnification R from 1 to 7 can be supported. Based on the dilation convolution processing magnification R, the horizontal spacing or vertical spacing of the read data can be determined.

[0111] When the number of layers is H, the maximum value of the dilation convolution processing magnification R is (2 H -1), and 1 to (2 H -1) of (2 H -1) types of extended convolution processing magnifications can be supported. An example of the configuration of the CNN processing unit 1305 according to this embodiment is shown in the block diagram of FIG. 21. Here, a variable indicating a layer is X i (i=1, . . . , H). In this case, the expansion convolution processing magnification R can be calculated based on the following (Equation 13).

[0112]

number

[0113] Thus, according to this embodiment, the hierarchical pixel storage unit can achieve a circuit cost H times lower (2 H-1) Since various types of extended convolution processing magnifications R can be supported, there is an advantage in that flexibility can be increased with low hardware costs.

[0114] [Third embodiment] In the first and second embodiments, examples in which the present invention is applied to CNN have been described, but the present invention is not limited to this and may be applied to applications other than CNN (applications such as image processing).

[0115] In the first and second embodiments, examples have been described in which the present invention is applied to a CNN that handles two-dimensional data, but the present invention is not limited to this and may also be applied to a CNN that handles one-dimensional data or data of three or more dimensions.

[0116] In the first embodiment, an example having 36 pixel storage units was described, but the number of pixel storage units in the horizontal and vertical directions may be determined depending on the application to accommodate various input data sizes.

[0117] In the first embodiment, an example has been described in which processing with an extended convolution processing magnification R of 1 or 2 can be supported. However, by increasing the number of inputs to the pixel storage unit shown in FIG. 6, processing with an extended convolution processing magnification R of 3 or more can be supported.

[0118] In the first and second embodiments, examples of transferring pixels (data) in the horizontal and vertical directions have been described, but this is not limited to two-dimensional data transfer, and may also be achieved by simplified one-dimensional or three or more-dimensional data transfer.

[0119] In the second embodiment, an example was described in which the results selected from the pixel storage units on the top, bottom, left, and right sides of the first layer 1601 and the second layer 1602 are transferred to the next layer, but hierarchical pixel storage units may also be provided for each of the four directions (top, bottom, left, and right).

[0120] In the first embodiment, the convolution process is described when the size of the coefficient filter before expansion is 3×3, but the size of the coefficient filter before expansion is not limited to 3×3 and may be any size such as 2×2.

[0121] 17 shows an example of coefficient filter expansion when the size of the coefficient filter before expansion is 2 × 2. As shown in a box 1704, the coefficient filter before expansion (coefficient filter Cx, y on the left) is expanded according to an expansion convolution processing magnification R = 1 to obtain the coefficient filter after expansion (coefficient filter C'x, y on the right). When R = 1, the size of the coefficient filter before expansion and the coefficient filter after expansion remains the same.

[0122] As shown in a frame 1705, the coefficient filter before expansion (coefficient filter Cx, y on the left) is expanded according to the expansion convolution processing magnification R=2 to obtain the coefficient filter after expansion (coefficient filter C'x, y on the right). The coefficient filter after expansion is a 3×3 coefficient filter obtained by inserting one idle coefficient between the coefficients of the coefficient filter before expansion.

[0123] As shown in a frame 1706, the coefficient filter before expansion (coefficient filter Cx, y on the left) is expanded according to the expansion convolution processing magnification R=4 to obtain the coefficient filter after expansion (coefficient filter C'x, y on the right). The coefficient filter after expansion is a 5×5 coefficient filter obtained by inserting three idle coefficients between the coefficients of the coefficient filter before expansion.

[0124] Also, if the size of the coefficient filter before expansion is 2 × 2, the coordinate index is corrected and O is calculated using the following (Equation 14): i,j (n) can also be calculated. In this formula, the expansion convolution processing magnification R is a multiple of 2.

[0125]

number

[0126] If the coefficient filter before expansion is of size 2x2, it can be approximated to a coefficient filter of size 3x3 using two cascaded coefficient filters. 19 An example of cascaded coefficient filters for dilated convolution processing is shown in FIG.

[0127] When the expansion convolution processing magnification R is 2, the size of the coefficient filter is expanded from 2 × 2 to 3 × 3. Because the positions of the 0 coefficients remain the same, the cascaded coefficient filters can be approximated as a coefficient filter whose size is expanded from 3 × 3 to 5 × 5.

[0128] When the expansion convolution processing magnification R is 4, the size of the coefficient filter is expanded from 2 × 2 to 5 × 5. Because the positions of the 0 coefficients remain the same, the cascaded coefficient filters can be approximated as a coefficient filter whose size is expanded from 5 × 5 to 9 × 9.

[0129] figure 18 This shows an example of applying cascaded dilated convolution processing to a convolutional neural network. There are two convolution processing layers and three sets of feature images (C1, C1, C2, or C1, C2, C2). When used in conjunction with depthwise convolution, the coefficient filters of the cascaded dilated convolution processing have 8 × C1, 4 × C1 × (C2 + 1), or 4 × (C1 + 1) × C2 coefficients. When not used in conjunction with depthwise convolution, the coefficient filters of the cascaded dilated convolution processing have 4 × (C1 + C2) × C2 coefficients or 4 × (C1 + C2) × C1 coefficients. The coefficient filter, whose size is expanded from 3 × 3 to 5 × 5, has 9 × C1 × C2 coefficients. By adjusting the numbers C1 and C2 of the feature images, the number of coefficients in the cascaded dilation convolution process can be made smaller than the target to be approximated, which has the effect of reducing the coefficients.

[0130] An example of a convolutional neural network for image segmentation processing is shown in Figure 20. This convolutional neural network includes cascade-connected dilated convolution processing. The convolutional neural network shown in Figure 20 can be realized by performing dilated convolution processing with a filter size of 2 × 2 before dilation using the configuration of the first or second embodiment.

[0131] In the first and second embodiments, an example has been described in which the feature image is processed in units of pixel sets, but the above processing may be performed on the feature image as one pixel set.

[0132] Furthermore, the numerical values, processing timing, processing order, processing subject, data (information) configuration / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0133] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0134] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0135] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0136] 1301: Input unit 1302: Data storage unit 1303: Communication unit 1304: Display unit 1305: CNN processing unit 1306: CPU 1307: ROM 1308: RAM 1309: Image processing unit 1310: System bus

Claims

1. a first acquiring means for acquiring pixel information from a specific pixel storage unit among a group of two-dimensionally arranged pixel storage units, the group including a memory storing pixel information for each pixel of the characteristic image and a multiplexer; a calculation means for performing a convolution calculation based on the pixel information acquired by the first acquisition means; Equipped with the pixel storage unit is connected to a pixel storage unit storing adjacent pixels in the feature image and a pixel storage unit storing non-adjacent pixels in the feature image, and pixel information can be shifted between the pixel storage units by selecting the multiplexer; When the first acquisition means acquires pixel information that is non-adjacent to the acquired pixel information in the feature image, the multiplexer of the specific pixel storage unit is selected to shift pixel information from the pixel storage unit that stores the non-adjacent pixel to the specific pixel storage unit; When the first acquisition means acquires pixel information adjacent to the acquired pixel information in the feature image, the multiplexer of the specific pixel storage unit is selected to shift pixel information from the pixel storage unit storing the adjacent pixel to the specific pixel storage unit.

1. An image processing device comprising:

2. moreover, a second acquisition means for acquiring coefficients from the coefficient filter; The calculation means performs a convolution calculation based on the pixel information acquired by the first acquisition means and the coefficients acquired by the second acquisition means.

2. The image processing device according to claim 1, wherein:

3. 3. The image processing device according to claim 2, wherein the calculation means performs a convolution operation for each layer in a hierarchical neural network based on pixel information acquired by the first acquisition means from a feature image in the layer and coefficients acquired by the second acquisition means from a coefficient filter in the layer.

4. The image processing device according to claim 3, characterized in that the first acquisition means selects pixel information to be acquired from the feature image according to any of a value set for each layer, a value set for the feature image for each layer, and a value set for a group of feature images for each layer.

5. The image processing device described in claim 3 or 4, characterized in that the first acquisition means acquires non-adjacent pixel information from the feature images at intervals corresponding to either a value set for each layer, a value set for the feature images for each layer, or a value set for a group of feature images for each layer.

6. 6. The image processing device according to claim 3, wherein the second acquisition means acquires non-adjacent coefficients from the coefficient filter at intervals according to a value set for each layer, a value set for a feature image for each layer, or a value set for a group of feature images for each layer.

7. 7. The image processing device according to claim 5, wherein the interval is a horizontal interval or a vertical interval.

8. 8. The image processing device according to claim 1, wherein the non-adjacent pixel information is pixel information that is non-adjacent to one another in the horizontal direction or the vertical direction.

9. 9. The image processing apparatus according to claim 2, wherein the calculation means comprises a plurality of means for performing a product-sum calculation between pixels and coefficients.

10. There are a plurality of the specific pixel storage units, 10. The image processing device according to claim 1, wherein the pixel information acquired by the first acquisition means from the specific pixel storage unit is supplied to the calculation means in parallel.

11. 1. An image processing method, comprising: a first acquisition step of acquiring pixel information from a specific pixel storage unit among a group of two-dimensionally arranged pixel storage units each having a memory in which pixel information for each pixel of the characteristic image is stored and a multiplexer; a calculation step of performing a convolution calculation based on the pixel information acquired in the first acquisition step; Equipped with the pixel storage unit is connected to a pixel storage unit storing adjacent pixels in the feature image and a pixel storage unit storing non-adjacent pixels in the feature image, and pixel information can be shifted between the pixel storage units by selecting the multiplexer; In the first acquisition step, when acquiring pixel information of a pixel that is not adjacent to the acquired pixel information in the feature image, the multiplexer of the specific pixel storage unit is caused to select the pixel information so as to shift the pixel information from the pixel storage unit that stores the non-adjacent pixel to the specific pixel storage unit; In the first acquisition step, when acquiring pixel information adjacent to the acquired pixel information in the feature image, the multiplexer of the specific pixel storage unit is caused to select the pixel information so as to shift the pixel information from the pixel storage unit storing the adjacent pixel to the specific pixel storage unit. An image processing method comprising:

12. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 10.

Images