Arithmetic device, arithmetic method, and computer program

Abstract

To reduce the calculation volume of a convolutional calculation.SOLUTION: A calculation device has a calculation processing unit that executes a predetermined process by inputting predetermined data into a neural network having a plurality of convolutional layers that are continuously provided from an input side to an output side. Each of the plurality of convolutional layers performs a convolutional calculation using kernels with different dilate sizes defined for each convolutional layer for at least one of a plurality of dimensions including a channel direction and a spatial direction for individual features included in a predetermined range of a feature map having a plurality of dimensions including the channel direction and the spatial direction, which are output from the immediately preceding layer and input to the convolutional layer concerned.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a computing device, a computing method, and a computer program using a neural network. [Background technology]

[0002] Conventionally, a neural network configuration has been proposed that performs recognition according to the results of learning based on the contents of data (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Andrew G. Howard et. al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, 2017. Summary of the Invention

[0004] The arithmetic device according to the present disclosure has a calculation processing unit that executes a predetermined process by inputting predetermined data to a neural network having a plurality of convolutional layers arranged in succession from the input side to the output side, and each of the plurality of convolutional layers executes a convolution operation using a kernel with a different dilate size determined for each convolutional layer for at least one of the plurality of dimensions including the channel direction and the spatial direction, for each feature included in a predetermined range of a feature map having a plurality of dimensions including the channel direction and the spatial direction, which is output from the immediately preceding layer and input to the convolutional layer.

[0005] The computation method according to the present disclosure is a computation method in which a computer executes a predetermined process by inputting predetermined data to a neural network having a plurality of convolution layers arranged in succession from the input side to the output side, and each of the plurality of convolution layers executes a convolution computation using a kernel with a different dilate size determined for each convolution layer for at least one of the plurality of dimensions including the channel direction and the spatial direction, for each feature included in a predetermined range of a feature map having a plurality of dimensions including the channel direction and the spatial direction, which is output from the immediately preceding layer and input to the convolution layer.

[0006] A computer program according to the present disclosure is a computer program that causes a computer to execute a predetermined process by inputting predetermined data to a neural network having a plurality of convolutional layers arranged in succession from the input side to the output side, and each of the plurality of convolutional layers executes a convolution operation using a kernel with a different dilate size determined for each convolutional layer for at least one of the plurality of dimensions including the channel direction and the spatial direction, for each feature included in a predetermined range of a feature map having a plurality of dimensions including the channel direction and the spatial direction, which is output from the immediately preceding layer and input to the convolutional layer. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of a computer. [Diagram 2] FIG. 2 is a functional block diagram of a processor. [Diagram 3] FIG. 1 is a schematic diagram showing the structure of a neural network. [Figure 4] FIG. 2 is a schematic diagram showing an example of a structure of a feature map. [Diagram 5] 1A is a schematic diagram showing an example of the structure of a kernel having a dilate size of 1, FIG. 1B is a schematic diagram showing an example of the structure of a kernel having a dilate size of 2, and FIG. 1C is a schematic diagram showing an example of the structure of a kernel having a dilate size of 4. [Figure 6]13 is a processing flowchart of a calculation processing unit. [Figure 7] 1A-1C are schematic diagrams showing different structures of neural networks. [Figure 8] FIG. 13 is a schematic diagram showing yet another different structure of a neural network. [Figure 9] FIG. 13 is a schematic diagram showing yet another different structure of a neural network. [Figure 10] 1A is a schematic diagram showing another example of the structure of a kernel having a dilate size of 1; FIG. 1B is a schematic diagram showing another example of the structure of a kernel having a dilate size of 2; and FIG. 1C is a schematic diagram showing another example of the structure of a kernel having a dilate size of 4. [Figure 11] FIG. 1 is a schematic diagram illustrating a configuration of a camera equipped with a computing device according to an embodiment. [Figure 12] 1 is a process flowchart of a camera equipped with a computing device according to an embodiment. [Figure 13] FIG. 2 is a schematic diagram showing the structure of a neural network according to one embodiment. [Figure 14] FIG. 11 is a schematic diagram illustrating an example of processing using a neural network. [Figure 15] FIG. 11 is a schematic diagram illustrating another example of processing using a neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The computing device, the computing method, and the computer program will be described in detail below with reference to the drawings, but it should be understood that the present invention is not limited to the drawings or the embodiments described below.

[0009] FIG. 1 is a schematic diagram showing a schematic configuration of a computer according to an embodiment of the present disclosure, and FIG. 2 is a functional block diagram of a processor.

[0010] In this embodiment, a computer 1, which is an example of a calculation device, executes a predetermined process by inputting input data to a neural network. To this end, the computer 1 includes an input / output interface 11, a memory 12, and a processor 13.

[0011] The input / output interface 11 has an interface circuit for accepting data to be processed by the computer 1 or for outputting data processed by the computer 1. The input / output interface 11 includes, for example, a communication interface circuit for connecting the computer 1 to a communication network, or a peripheral device interface circuit for connecting the computer 1 to various peripheral devices such as a keyboard and a display. The input / output interface 11 accepts the input of predetermined data and outputs the processing results.

[0012] The memory 12 is an example of a storage unit, and includes at least one of a semiconductor memory, a magnetic disk device, and an optical disk device. The memory 12 stores a driver program, an operating system program, an application program, data, and the like used in processing by the processor 13. For example, the memory 12 stores a computer program for implementing the arithmetic processing of the present disclosure as an application program. The memory 12 also stores a group of parameters (number of layers, layer configuration, kernel, weighting coefficient, and the like) for defining a neural network as data. Various programs may be installed in the memory 12 from a computer-readable portable recording medium using a known setup program or the like. The computer-readable portable recording medium is, for example, a CD-ROM (Compact Disc Read-Only Memory), a DVD-ROM (DVD Read-Only Memory), and the like.

[0013] The processor 13 includes one or more processors and their peripheral circuits. The processor 13 is a processing circuit that comprehensively controls the overall operation of the computer 1, and is, for example, a CPU (Central Processing Unit). The processor 13 may include a GPU (Graphics Processing Unit) that can process a large amount of routine calculations in parallel. The processor 13 controls the operation of the input / output interface 11, etc., so that various processes of the computer 1 are executed by appropriate means based on programs, etc. stored in the memory 12. The processor 13 executes processes based on programs (operating system programs, driver programs, application programs, etc.) stored in the memory 12. The processor 13 can also execute multiple programs (application programs, etc.) in parallel.

[0014] 2, the processor 13 has an arithmetic processing unit 131 for executing calculations by the neural network NN1. The arithmetic processing unit 131 is a functional module implemented by a program executed on a processor included in the processor 13. Alternatively, the arithmetic processing unit 131 may be implemented in the computer 1 as an independent integrated circuit, microprocessor, or firmware.

[0015] The arithmetic processing unit 131 executes a predetermined process by inputting predetermined data to the neural network NN1. The predetermined process is, for example, object detection for identifying the type and position of an object shown in an image, region segmentation for classifying an image by pixel, and the like.

[0016] The neural network NN1 has an upper layer UL1, a middle layer ML1, and a downstream layer DL1.

[0017] FIG. 3 is a schematic diagram showing the structure of the neural network NN1, FIG. 4 is a schematic diagram showing an example of the structure of a feature map, and FIG. 5 is a schematic diagram showing an example of the structure of a kernel (or filter).

[0018] The neural network NN1 is a neural network having, for example, a CNN (Convolutional Neural Network) type architecture. The neural network NN1 receives, for example, image data input, executes a predetermined calculation, and detects an object shown in the image data. The neural network NN1 is trained in advance according to the backpropagation method using a plurality of images showing the object to be detected and information indicating the position, type, etc. of the object on each image as teacher data.

[0019] In this case, the neural network NN1 performs a convolution operation on the input image data in the upstream layer UL1, and outputs a feature map with the same resolution as the image data. Then, the upstream layer UL1 performs a pooling process on the feature map, generates a feature map with a reduced resolution, and outputs the feature map to the middle layer ML1. The middle layer ML1 performs a convolution operation using multiple convolution layers, and outputs a feature map with the same resolution as the input feature map. The downstream layer DL1 performs a pooling process on the input feature map, and generates a feature map with a reduced resolution. Multiple bounding boxes are output from the finally obtained feature maps of various resolutions, and NMS (Non-Maximum Suppression) processing is performed on the multiple bounding boxes to detect objects.

[0020] The neural network NN1 may have different upstream, middle, and downstream layers downstream of the above-mentioned upstream layer UL1, middle layer ML1, and downstream layer DL1. In this case, the different upstream layer may include the downstream layer DL1 and perform processing such as object detection.

[0021] The neural network NN1 may also be trained for semantic segmentation. In this case, the upper layer UL1 performs convolution and pooling on the input image data in the same manner as in the case of object detection to generate a feature map with reduced resolution, and outputs it to the middle layer ML1. The middle layer ML1 performs convolution using multiple convolution layers, and outputs a feature map with the same resolution as the input feature map to the downstream layer DL1. The downstream layer DL1 performs pooling on the input feature map in the same manner as the upper layer UL1 to generate a feature map with reduced resolution. The feature map output from the neural network NN1 is subjected to transposed convolution to generate a feature map with increased resolution, and each pixel of the image is classified into multiple classes, thereby performing semantic segmentation.

[0022] The upstream layer UL1 has an input layer that accepts input of image data, one or more convolution layers that perform convolution operations on the input image data, and outputs a feature map 100 having multiple dimensions. Furthermore, the upstream layer UL1 may have a pooling layer, and may output the resulting feature map to the middle layer ML1 after lowering the resolution of the resulting feature map by the pooling layer.

[0023] 4 shows the structure of the feature map 100 output from the upstream layer UL1. The feature map 100 has a structure in which multiple features are arranged in multiple dimensions. In the feature map 100, the multiple features are arranged in a spatial direction having at least the H (height) direction and the W (width) direction, and further arranged in the C (channel, depth) direction.

[0024] In this embodiment, the nodes shown in Fig. 3 correspond to the channels of the feature map 100. That is, the feature map 100 has a number of channels according to the number of nodes included in the final layer L10 located at the most output side of the upstream layer UL1. For the sake of simplicity of explanation, eight nodes (N101-N108) included in the final layer L10 are illustrated in Fig. 3. Note that the number of nodes and the number of features to be convolved are not limited to this embodiment.

[0025] The middle layer ML1 has a plurality of convolution layers L11-L13 arranged successively from the input side to the output side. The convolution layers L11-L13 perform a convolution operation for at least one of the multiple dimensions of the feature map. In this embodiment, the convolution layers L11-L13 perform a convolution operation for at least the C direction and output the result of the operation.

[0026] Each of the multiple convolution layers performs a convolution operation on each feature included in a predetermined range of a feature map having multiple dimensions that is output from the immediately preceding layer and input to the convolution layer. That is, each convolution layer has the same number of nodes as the number of channels of the input feature map, of which eight corresponding to N101-N108 are shown in FIG. 3. The nodes N111-N118 included in the convolution layer L11 perform a multiply-and-accumulate operation between the features output by the nodes N101-N108 included in the final layer L10 of the upstream layer UL1 and the corresponding weighting coefficients of the kernel. For the convolution operation in each convolution layer, a kernel with a different dilate size is used for at least one of the multiple dimensions. In this embodiment, a kernel with a different dilate size is used for each convolution layer for the C direction.

[0027] In Figure 3, a line connecting a node in an input layer (e.g., convolutional layer L11) and a node in an output layer (e.g., convolutional layer L12) indicates that the features output by the node in the input layer are used in the calculation of the node in the output layer (hereinafter also referred to as "connected").

[0028] The convolutional layers L11-L13 perform convolutional operations on individual features included in a predetermined range in the C direction of the feature map. In FIG. 3, the middle layer and the predetermined range are indicated by a double frame line.

[0029] Nodes N111-N118 included in the convolutional layer L11 perform convolution operations using a kernel with a dilate size of 1 on each of the features output from nodes N101-N108 of the feature map output from the final layer L10 of the immediately preceding layer, the upstream layer UL1.

[0030] FIG. 5(a) is a schematic diagram showing an example of the structure of a kernel K11 with a dilate size of 1. In FIGS. 5(a)-(c), the hatched squares in each kernel indicate that a weighting factor is set. Also, the unshaded squares in each kernel indicate that no weighting factor is set (or 0 is set as the weighting factor). That is, only the weighting factor corresponding to the hatched squares is used as an input for the convolution operation. In the kernel K11 shown in FIG. 5(a), a weighting factor W112 is set at a position corresponding to the target node, and weighting factors W111 and W113 are set at positions that are 1 away from the target node.

[0031] Convolution calculations are performed in the convolution layer L11 by sequentially changing the target node (stride=1) and applying the kernel K11 to each feature included in a predetermined range of the feature map output from the final layer L10 of the upstream layer UL1, which is the immediately preceding layer. For example, node N112 adds the product of the feature output from node N101 and the weighting factor W111, the product of the feature output from node N102 and the weighting factor W112, and the product of the feature output from node N103 and the weighting factor W113, and outputs the result. In this specification, the stride refers to the interval of the positions at which the kernel is applied, based on the feature map input to the midstream layer.

[0032] In the convolution layer L12, a convolution operation is performed using a kernel K12 having a dilate size of 2 as shown in FIG. 5(b), and in the convolution layer L13, a convolution operation is performed using a kernel K13 having a dilate size of 4 as shown in FIG. 5(c). In the kernel K12 shown in FIG. 5(b), a weighting factor W122 is set at a position corresponding to the target node, and weighting factors W121 and W123 are set at positions at a distance of 2 from the target node. In the kernel K12, a weighting factor is not set at a position at a distance of 1 from the target node. In the kernel K13 shown in FIG. 5(c), a weighting factor W132 is set at a position corresponding to the target node, and weighting factors W131 and W133 are set at positions at a distance of 4 from the target node. In the kernel K13, a weighting factor is not set at positions at distances of 1, 2, and 3 from the target node.

[0033] In this way, in the middle layer ML1, convolutional calculations are performed using kernels with different dilate sizes for each convolutional layer by multiple convolutional layers L11-L13 arranged consecutively from the input side to the output side.

[0034] A portion of features included in a predetermined range of the feature map input to the most input convolutional layer L11 is combined with each feature output from nodes N131-N138 included in a predetermined range of the feature map output from the most output convolutional layer L13. The portion of features refers to features included in a range separated by the sum of the dilate sizes of the kernels used in each of the convolutional layers L11-L13 (1+2+4=7).

[0035] In FIG. 3, for example, node N135 in convolutional layer L13 is connected to node N121 and node N125 in convolutional layer L12.

[0036] In the convolutional layer L12, the node N121 is connected to the node N111 and the node N113 of the convolutional layer L11. The node N125 is connected to the node N113, the node N114, and the node N117 of the convolutional layer L11. Therefore, the node N135 of the convolutional layer L13 is connected to the nodes N111, N113, N114, and N117 of the convolutional layer L11 via the node N121 and the node N125 of the convolutional layer L12.

[0037] In the convolutional layer L11, the node N111 is connected to the node N101 and the node N102 in the final layer L10 of the upstream layer UL1. The node N113 is connected to the node N102, the node N103, and the node N104 in the final layer L10 of the upstream layer UL1. The node N115 is connected to the node N104, the node N105, and the node N106 in the final layer L10 of the upstream layer UL1. The node N117 is connected to the node N106, the node N107, and the node N108 in the final layer L10 of the upstream layer UL1. Therefore, node 135 in convolutional layer L13 is connected to nodes N101-N108 in the final layer L10 of the upstream layer UL1 via nodes N121 and N125 in convolutional layer L12 and nodes N111, N113, N114, and N117 in convolutional layer L11.

[0038] In this way, each feature output from nodes N131-N138 of the convolutional layer L13 is combined with all the C-direction features output from nodes N101-N108 of the final layer L10 of the upstream layer UL1.

[0039] The feature map output from the convolution layer L13 is input to the downstream layer DL1. The downstream layer DL1 has an output layer at the output side that outputs the calculation result by the neural network NN1. The output layer can identify the most likely type of detection target type for a region of interest on an image, for example, by using a softmax function as an activation function. The output layer can also output a confidence level for each type by using a sigmoid function as an activation function. The downstream layer DL1 may include a transposed convolution layer that performs a transposed convolution calculation, a pooling layer, etc., at the input side of the output layer.

[0040] 6 is a process flowchart of the arithmetic processing unit 131. The arithmetic processing unit 131 inputs predetermined data to the neural network NN1 and executes the following processes.

[0041] First, the calculation processing unit 131 causes the upstream layer UL1 of the neural network NN1 to execute a predetermined calculation on predetermined data and output the feature map 100 (step S1). The feature map 100 includes features corresponding to a predetermined range that is the target of the convolution calculation in the convolution layers L11-L13.

[0042] Next, the calculation processing unit 131 executes a convolution calculation in the unprocessed and most input-side layer among the multiple convolution layers L11-L13 included in the middle layer ML1 of the neural network NN1 (step S2). In the convolution calculation in step S2, a kernel with a dilate size preset for each convolution layer is used. In this embodiment, a kernel with a dilate size of 1 is used in the convolution layer L11, a kernel with a dilate size of 2 is used in the convolution layer L12, and a kernel with a dilate size of 4 is used in the convolution layer L13.

[0043] Next, the calculation processing unit 131 determines whether or not the convolution calculations by all of the convolution layers L11-L13 included in the middle layer ML1 have been completed (step S3).

[0044] If it is determined that the convolution calculations by all the convolution layers have not been completed (step S3: N), the calculation processing unit 131 executes the process of step S2. The unprocessed convolution layers are associated with a dilate size different from the dilate size associated with the processed convolution layers. Therefore, by repeatedly executing the process of step S2 in this manner, the convolution calculations using kernels with different dilate sizes are executed in each of the multiple convolution layers.

[0045] If it is determined that the convolution calculations by all the convolution layers have been completed (step S3: Y), the calculation processing unit 131 inputs the output of the middle layer ML1 to the downstream layer DL1 and causes the downstream layer DL1 to execute a predetermined calculation (step S4). The calculation processing unit 131 obtains the calculation result output from the final layer of the downstream layer DL1 and ends the calculation by the neural network NN1.

[0046] A feature map (C in =C out =256), the 3×3 kernel (k H =k W The computational load when a convolution operation using θ(θ)=3) is performed by MobileNets described in Non-Patent Document 1 can be calculated by the following Equation 1.

[0047] {(k H ×k W ×C in )+(1×1×C in ×C out )}×H×W = (2304+65536)×H×W = 67840×H×W (Formula 1)

[0048] On the other hand, when the calculation device of the present disclosure performs calculations using the same kernel for the same feature map, the calculation load is calculated by the following Equation 2.

[0049] C in ×log2C in ×k H ×kW ×H×W = 256×8×3×3×H×W = 18432×H×W (Formula 2)

[0050] By performing calculations using the neural network NN1 in this manner, the calculation device can reduce the amount of calculations for a feature map having a large number of features in the dimension to be calculated.

[0051] The dilate size of the kernel used in the convolutional layers L11-L13 is 1 (=2 0 ), 2(=2 1 ), 4(=2 2 ), which is a power of 2. Setting the dilate size in this way enables convolution operations that combine more features with fewer convolution layers.

[0052] The dilate size of the kernel used in the convolutional layer may be a power of a natural number greater than or equal to 2. For example, if the dilate size is 1 (=3 0 ), 3(=3 1 ), 9(=3 2 A kernel that is a power of 3, such as x = 1, may be used. Furthermore, the dilate size of the kernel is not limited to a power of 2 or more, and may be set appropriately so that a required range of features can be combined by a predetermined number of convolutional layers.

[0053] In the present disclosure, the structure of the neural network is not limited to the structure shown in Fig. 3. For example, the dilate sizes used in the convolution calculations in multiple convolution layers included in the middle layer of the neural network do not have to be set in ascending order such as 1, 2, 4. Fig. 7 is a schematic diagram for explaining a different structure of the neural network.

[0054] The neural network NN2 shown in FIG. 7 has an upper layer UL2, a middle layer ML2, and a downstream layer DL2.

[0055] The upstream layer UL2 is similar to the upstream layer UL1 in the neural network NN1, and the downstream layer DL2 is similar to the downstream layer DL1, so detailed explanations will be omitted.

[0056] The middle layer ML2 has a plurality of convolution layers L21-L23 arranged successively from the input side to the output side. The nodes N211-N218 included in the convolution layer L21 perform a convolution operation using a kernel with a dilate size of 4 on each feature output from the nodes N201-N208 of the feature map output from the final layer L20 of the upstream layer UL2, which is the immediately preceding layer. The nodes N221-N228 included in the convolution layer L22 perform a convolution operation using a kernel with a dilate size of 1 on each feature output from the nodes N211-N218 of the feature map output from the convolution layer L21, which is the immediately preceding layer. The nodes N231-N238 included in the convolution layer L23 perform a convolution operation using a kernel with a dilate size of 2 on each feature output from the nodes N221-N228 of the feature map output from the convolution layer L22, which is the immediately preceding layer.

[0057] In this way, when performing convolution operations using kernels with different dilate sizes in multiple consecutive convolution layers, the dilate sizes do not need to be changed in ascending or descending order from the input side to the output side, and the order can be arbitrary.

[0058] Each feature output from nodes N231-N238 of the output-side convolutional layer L23 is combined with all C-direction features input from nodes N201-N208 of the final layer L20 of the upstream layer UL2 to nodes N211-N218 of the convolutional layer L21.

[0059] Furthermore, multiple convolutional layers included in the middle layer of the neural network may perform convolutional operations with a stride of 2 or more. Figure 8 is a schematic diagram illustrating yet another structure of the neural network.

[0060] The neural network NN3 shown in FIG. 8 has an upper layer UL3, a middle layer ML3, and a downstream layer DL3.

[0061] The upper layer UL3 is similar to the upper layer UL1 in the neural network NN1, and therefore a detailed description thereof will be omitted.

[0062] The middle layer ML3 has multiple convolutional layers L31-L33 arranged consecutively from the input side to the output side. Nodes N311-N318 included in the convolutional layer L31 perform convolutional calculations using a kernel with a dilate size of 4 on individual features output from nodes N301-N308 of the feature map output from the final layer L30 of the upstream layer UL3, which is the immediately preceding layer.

[0063] In the convolution layer L32, a convolution operation is performed using a kernel with a dilate size of 1. The stride of the convolution operation at this time is 2. That is, when a convolution operation is performed at node N321, a convolution operation is not performed at adjacent node N322. Also, a convolution operation is performed at node N323, which is located at a distance of 2 from node N321.

[0064] In the convolution layer L33, a convolution operation is performed using a kernel with a dilate size of 2. At this time, the stride of the convolution operation is 4. That is, the convolution operation is performed at nodes N331 and N335, and the convolution operation is not performed at nodes N332-N334 and nodes N336-N338.

[0065] Each feature output from nodes N331 and N335 of the output-side convolutional layer L33 is combined with all C-direction features input from nodes N301-N308 of the final layer L30 of the upstream layer UL3 to nodes N311-N318 of the convolutional layer L31.

[0066] A layer L34 included in the downstream layer DL3 executes a transpose convolution operation using the features output from nodes N331 and N335.

[0067] Thus, in the neural network NN3, the middle layer ML3 includes convolution layers L32 and L33 that perform convolution operations with a stride of 2 or more. Therefore, in the neural network NN3, the amount of calculations on the output side is reduced compared to convolution layers that perform convolution operations with a stride of 2 or more.

[0068] Although an example in which the strides are set to 1, 2, and 4 in the convolutional layers L31-L33 has been described, the present disclosure is not limited thereto. That is, it is sufficient that a convolutional layer that executes a convolutional operation with a stride of 2 or more is placed at any position in the middle layer ML3. Note that placing a convolutional layer that executes a convolutional operation with a stride of 2 or more at a position close to the input side is effective in reducing the amount of calculation.

[0069] In addition, each feature included in the predetermined range that is the subject of the convolution operation in the midstream layer may be combined with a feature that is not included in the predetermined range. Figure 9 is a schematic diagram illustrating yet another structure of the neural network.

[0070] The neural network NN4 shown in Fig. 9 has an upper layer UL4, a middle layer ML4, and a downstream layer DL4. The downstream layer DL4 is similar to the downstream layer DL1 in the neural network NN1, so a detailed description thereof will be omitted.

[0071] The upstream layer UL4 has nodes N401-N408 corresponding to a predetermined range that is the target of the convolution operation in the midstream layer ML4, as well as nodes N400 and N409 outside the predetermined range.

[0072] The middle layer ML4 has a plurality of convolutional layers L41-L43 arranged successively from the input side to the output side. The convolutional layer L41 has nodes N411-N418 included in the predetermined range, as well as nodes N410 and N419 outside the predetermined range. The convolutional layer L42 has nodes N421-N428 included in the predetermined range, as well as nodes N420 and N429 outside the predetermined range. The convolutional layer L43 has nodes N431-N438 included in the predetermined range, as well as nodes N430 and N439 outside the predetermined range.

[0073] A node N410 outside the predetermined range in the convolutional layer L41 is coupled to a node N400 outside the predetermined range and a node N401 included in the predetermined range in the final layer L40 of the upstream layer UL4. A node N420 outside the predetermined range in the convolutional layer L42 is coupled to a node N410 outside the predetermined range and a node N411 included in the predetermined range in the convolutional layer L41. A node N430 outside the predetermined range in the convolutional layer L43 is coupled to a node N420 outside the predetermined range and a node N421 included in the predetermined range in the convolutional layer L42.

[0074] Node N419 outside the predetermined range in convolutional layer L41 is coupled to node N408 included in the predetermined range and to node N409 outside the predetermined range in the final layer L40 of the upstream layer UL4. Node N429 outside the predetermined range in convolutional layer L42 is coupled to node N418 included in the predetermined range and to node N419 outside the predetermined range in convolutional layer L41. Node N439 outside the predetermined range in convolutional layer L43 is coupled to node N428 included in the predetermined range and to node N429 outside the predetermined range in convolutional layer L42.

[0075] In this way, in the neural network NN4, the nodes included in the predetermined range are connected to the nodes outside the predetermined range, so that the features output by the calculations at the nodes included in the predetermined range can be used for the calculations at the nodes outside the predetermined range.

[0076] Also, the convolution operation in the middle layer ML1 may be performed in a dimension other than the C direction, for example, the HW direction. In the convolution operation performed in the HW direction (two dimensions), a two-dimensional kernel is used. FIG. 10 is a schematic diagram showing another example of the kernel structure. In FIG. 10(a)-(c), it is shown that a weighting factor is set in the hatched squares in each kernel. In FIG. 10(a)-(c), a weighting factor is not set in the unhatched squares in each kernel.

[0077] Fig. 10 shows examples of kernels used in convolution calculations performed in the HW direction, and Fig. 10(a), (b), and (c) show examples of the structures of kernels K21, K22, and K23 with die size of 1, 2, and 4, respectively. Note that the two dimensions convoluted using the kernels shown in Fig. 10(a)-(c) are not limited to the H direction and the W direction, and may be, for example, the H direction and the C direction, or the W direction and the C direction.

[0078] Furthermore, in the arithmetic device of the present disclosure, the arithmetic processing unit 131 may execute convolution for three or more dimensions using a kernel (not shown) having a three or more dimensional structure.

[0079] Next, an example in which the arithmetic device according to an embodiment is mounted on a camera will be described below. Fig. 11 is a schematic diagram showing the configuration of a camera mounted with the arithmetic device according to an embodiment.

[0080] The arithmetic device according to this embodiment is a microcontroller 4 mounted on a camera 2. The camera 2 is an example of an imaging device, and includes an image sensor 3, a microcontroller 4, and a lens 5. The image sensor 3 is an example of an imaging element, and converts light from a subject collected by the lens 5 into image data. The microcontroller 4 is an integrated circuit that integrates a processor, a memory, and an input / output interface. The microcontroller 4 executes calculations based on image data input from the image sensor 3, and outputs a signal indicating an area in the image data to be the target of autofocus. The microcontroller 4 also operates as a focus control unit, and outputs a focus control signal that controls the lens 5 to focus on a predetermined area represented in the image data. The lens 5 is an example of an optical system, and collects light from a subject on the image sensor 3. The lens 5 has an autofocus mechanism 51, and the autofocus mechanism 51 focuses the lens 5 on the subject based on the focus control signal output from the microcontroller 4.

[0081] Fig. 12 is a process flowchart of a camera including a computing device according to an embodiment. The camera 2 starts the process of Fig. 12 in response to detection of an action instructing execution of autofocus, such as a half-press of a shutter button (not shown) by a user. The camera 2 may repeatedly execute the process of Fig. 12 while the action instructing execution of autofocus is being detected.

[0082] First, the image sensor 3 transmits image data corresponding to the collected light to the microcontroller 4 (step S10).

[0083] Next, the microcontroller 4 inputs the received image data to the neural network and executes a calculation to identify an area corresponding to the object to be recognized that is included in the image data (step S20). The calculation process by the neural network is similar to the flowchart of the calculation by the neural network shown in FIG.

[0084] Next, the microcontroller 4 generates a focus control signal for focusing on an area corresponding to the object identified by the calculation, and transmits it to the autofocus mechanism 51. The autofocus mechanism 51 changes the position of the focusing lens included in the lens 5 based on the focus control signal received from the microcontroller 4 (step S30). Whether or not the specified area of ​​the image is in focus is determined by a method such as a contrast AF method or a phase difference AF method.

[0085] The microcontroller 4 includes a processing unit 431 having a neural network NN4, instead of the processing unit 131 having the neural network NN1. FIG. 13 is a schematic diagram showing the structure of a neural network NN5 included in the processing device mounted on the camera 2.

[0086] The neural network NN5 has an upper layer UL5, a middle layer ML5, and a downstream layer DL5.

[0087] The upstream layer UL5 differs from the upstream layer UL1 in the neural network NN1 in that it has a first upstream layer UL5A and a second upstream layer UL5B, each of which executes a predetermined calculation, but is otherwise similar.

[0088] The middle layer ML5 has a first middle layer ML5A and a second middle layer ML5B that perform a predetermined operation. The middle layer ML5 also has a plurality of convolution layers L51-L52 that are successively provided from the input side to the output side. The nodes N511-N514 included in the convolution layer L51 perform a convolution operation using a kernel with a dilate size of 1 on each feature output from the nodes N501-N504 of the feature map output from the final layer L50 of the upstream layer UL5, which is the immediately preceding layer. The nodes N515-N518 included in the convolution layer L51 perform a convolution operation using a kernel with a dilate size of 1 on each feature output from the nodes N505-N508 of the feature map output from the final layer L50 of the upstream layer UL5. The kernel used in the convolution operation of the nodes N515-N518 may have parameters different from the kernel used in the convolution operation of the nodes N511-N514.

[0089] Similarly, in the convolution layer L52, a convolution operation is performed using a kernel with a dilute size of 2.

[0090] In this way, in the middle layer ML5, convolutional calculations are performed using kernels with different dilate sizes for each convolutional layer by multiple convolutional layers L51-L52 arranged consecutively from the input side to the output side.

[0091] A part of features included in a predetermined range of the feature map input to the most input convolutional layer L51 is combined with each feature output from the nodes N521-N524 included in a predetermined range of the feature map output from the most output convolutional layer L52. The part of features refers to features included in a range separated by the sum of the dilate sizes (1+2=3) of the kernels used in each of the convolutional layers L51-L52. That is, all features input from the nodes N501-N504 of the final layer of the final layer L50 of the upstream layer UL5 to the nodes N511-N514 of the convolutional layer L51 are combined with each feature output from the nodes N525-N528 of the convolutional layer L52.

[0092] In the neural network NN5, node N524 in convolutional layer L52 is connected to node N516 in convolutional layer L51, which has a distance of 2 outside the predetermined range. Therefore, features input to node N516 from nodes N505-N507 are also connected to node N524.

[0093] The downstream layer DL5 differs from the downstream layer DL1 in the neural network NN1 in that it has a first downstream layer DL5A and a second downstream layer DL5B, each of which executes a predetermined calculation, but is otherwise similar.

[0094] FIG. 14 is a schematic diagram illustrating an example of processing using a neural network.

[0095] The first upstream layer UL5A, the first middle layer ML5A, and the first downstream layer DL5A of the neural network NN5 can detect a rectangular area including a predetermined identification object (e.g., a cow) by learning the teacher data. The first upstream layer UL5A, the first middle layer ML5A, and the first downstream layer DL5A may have an architecture similar to that of an SSD (Single Shot MultiBox Detector). In this case, the first upstream layer UL5A performs a convolution operation and outputs a feature map with the same resolution as the resolution of the image PIC output by the image sensor 3. Then, the first upstream layer UL5A performs a pooling process on the feature map to generate a feature map with a reduced resolution and outputs it to the first middle layer ML5A. The first middle layer ML4A performs a convolution operation and outputs a feature map with the same resolution as the resolution of the input feature map. The first downstream layer DL5A performs a pooling process and generates a feature map with a reduced resolution. The microcontroller 4 detects multiple bounding boxes from the feature maps of various resolutions finally obtained, and performs NMS processing on the multiple bounding boxes. By processing in this way, the neural network NN5 detects a rectangular area BB1 including the object to be classified from the image PIC.

[0096] There is a high possibility that the points included in the rectangular area BB1 are points that the user of the camera 2 is paying attention to. Therefore, the autofocus mechanism 51 controls the lens 5 so as to focus on the points included in the rectangular area BB1. Here, the rectangular area BB1 includes a point P1 that corresponds to the identification object and a point P2 that does not correspond to the identification object. If the autofocus mechanism 51 is controlled so as to focus on the point P2 that does not correspond to the identification object, there is a high possibility that the autofocus control will not be in line with the user's intention.

[0097] On the other hand, the second upstream layer UL5B, the second middle layer ML5B, and the second downstream layer DL5B can divide a region having a predetermined feature (for example, corresponding to an object to be identified) by learning the teacher data. The second upstream layer UL5B, the second middle layer ML5B, and the second downstream layer DL5B may have an architecture similar to U-Net, for example. In this case, the second upstream layer UL5B executes a convolution operation and a pooling process, and outputs a feature map having a lower resolution than the resolution of the image PIC output by the image sensor 3. The second middle layer ML5B executes a convolution operation, and outputs a feature map having the same resolution as the resolution of the input feature map to the second downstream layer DL5B. The second downstream layer DL5B executes a transposed convolution operation. The second downstream layer DL5B may also execute a transposed convolution operation. The microcontroller 4 classifies the pixels of the image PIC into a plurality of classes. By carrying out this processing, the neural network NN5 divides the area of ​​the image PIC into areas R1 and R2 corresponding to the class of objects to be identified, and an area R3 corresponding to a class other than the objects to be identified.

[0098] The microcontroller 4 mounted on the camera 2 can reduce the amount of calculation required to execute the above-mentioned processing by using the neural network NN5.

[0099] The microcontroller 4 of this embodiment can execute more efficient calculations by using the neural network NN5. Fig. 15 is a schematic diagram for explaining another example of processing using the neural network NN5.

[0100] In the neural network NN5 of this embodiment, some nodes included in the first middle layer ML5A and the second middle layer ML5B are connected. The above-mentioned learning is performed on such a neural network NN5. In the trained neural network NN5, the first middle layer ML5A and the second middle layer ML5B are connected, and the feature amount on the first middle layer ML5A side that performs object detection and the feature amount on the second middle layer ML5B side that performs region division are mixed. The mixing ratio is larger nearer the boundary between the first middle layer ML5A and the second middle layer ML5B.

[0101] By inputting the image PIC to such a neural network NN5, a region that is the intersection of the rectangular region BB1 and the regions R1 and R2 corresponding to the object to be identified is specified, and the region R4 that includes only the object to be identified is identified. Since the region R4 does not include any points that do not correspond to the object to be identified, it is possible to perform autofocus control according to the user's intention by controlling the autofocus mechanism 51 so that the focus is set on the point P3 included in the region R4. The arithmetic device of the present disclosure can perform such convolution calculations suitable for autofocus control with a small amount of calculation.

[0102] It should be understood that those skilled in the art can make various changes, substitutions, and alterations thereto without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0103] 1. Computer 131 Processing unit NN1 Neural Network

Claims

1. The neural network has a plurality of convolution layers arranged in succession from an input side to an output side, and a calculation processing unit executes a predetermined process by inputting predetermined data to the neural network, Each of the plurality of convolution layers performs a convolution operation using kernels with different dilate sizes determined for each of the convolution layers in the channel direction for all features in the channel direction of a feature map having a plurality of dimensions including a channel direction and a spatial direction, the feature map being output from a previous layer and input to the convolution layer; The dilate size, which differs for each of the plurality of convolutional layers, is a power of n (n is a natural number equal to or greater than 2) and is set so that all of the features of the feature map input to the most input convolutional layer are combined with individual features of the feature map output from the most output convolutional layer. Computing device.

2. 2. The computing device according to claim 1, wherein all of the features of the feature map output from the most output convolutional layer are combined with features included in a range separated by a sum of dilate sizes of kernels used in each of the plurality of convolutional layers of the feature map input to the most input convolutional layer.

3. 3. The computing device according to claim 1, wherein each feature included in a predetermined range of the feature map input to the most input convolutional layer is combined with a feature included in a range spaced apart by a sum of the dilate sizes of kernels used in each of the plurality of convolutional layers in the predetermined range of the feature map output from the most output convolutional layer.

4. The arithmetic device according to any one of claims 1 to 3, wherein one convolutional layer included in the plurality of convolutional layers executes a convolutional operation with a stride of 2 or more.

5. The arithmetic device according to claim 4 , wherein one convolutional layer included in the plurality of convolutional layers performs a convolutional operation with a stride based on the dilate size.

6. The arithmetic device according to claim 4 or 5, wherein one convolutional layer included in the plurality of convolutional layers performs a convolutional operation with a stride that is twice the dilate size.

7. The arithmetic device according to claim 6 , wherein one convolutional layer included in the plurality of convolutional layers executes a convolutional operation with a stride of 4 when the dilate size is 2.

8. The computing device according to any one of claims 2 to 7, wherein the neural network further comprises a transposed convolution layer connected to the plurality of convolution layers and configured to perform a transposed convolution operation.

9. an imaging element that outputs image data according to the light collected by the optical system; 9. A computing device according to claim 1, wherein the computing unit of the computing device executes a process of identifying an area corresponding to an object to be identified among image areas represented in the image data by inputting the image data as the predetermined data to the neural network; a focus control unit that controls the optical system so as to focus on an area corresponding to the identified object.

10. A computing method in which a computer executes a predetermined process by inputting predetermined data into a neural network having a plurality of convolution layers arranged successively from an input side to an output side, comprising the steps of: Each of the plurality of convolution layers performs a convolution operation using kernels with different dilate sizes determined for each of the convolution layers in the channel direction for all features in the channel direction of a feature map having a plurality of dimensions including a channel direction and a spatial direction, the feature map being output from a previous layer and input to the convolution layer; The dilate size, which differs for each of the plurality of convolutional layers, is a power of n (n is a natural number equal to or greater than 2) and is set so that all of the features of the feature map input to the most input convolutional layer are combined with individual features of the feature map output from the most output convolutional layer. Calculation method.

11. A computer program for causing a computer to execute a predetermined process by inputting predetermined data into a neural network having a plurality of convolution layers arranged successively from an input side to an output side, Each of the plurality of convolution layers performs a convolution operation using kernels with different dilate sizes determined for each of the convolution layers in the channel direction for all features in the channel direction of a feature map having a plurality of dimensions including a channel direction and a spatial direction, the feature map being output from a previous layer and input to the convolution layer; The dilate size, which differs for each of the plurality of convolutional layers, is a power of n (n is a natural number equal to or greater than 2) and is set so that all of the features of the feature map input to the most input convolutional layer are combined with individual features of the feature map output from the most output convolutional layer. Computer program.

Images