Neural network system

Abstract

To make it possible to reduce the computational complexity of convolution.SOLUTION: A neural network system 100 has a separable convolutional subnetwork 150, the separable convolutional subnetwork has a plurality of separable convolutional neural network layers arranged in a stacked manner in sequence, each of the plurality of separable convolutional neural network layers applies a first grouping convolution to an input to the separable convolutional neural network layer, the input to the first grouping convolution including a plurality of channels, and the first grouping convolution is a spatial convolution in which the channels of the input to the first grouping convolution are divided into groups in the channel direction, convolved, and combined to form an output.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a neural network system, a processing method of a neural network system, and a program.

Background Art

[0002] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output of received inputs. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to another layer within the network, such as the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of an individual set of parameters.

[0003] A convolutional neural network generally includes convolutional neural network layers and, in some cases, fully connected neural network layers and other types of neural network layers. Convolutional neural network layers are sparsely connected, and each node in the convolutional layer receives input only from a subset of the nodes in the next lowest neural network layer. Some convolutional neural network layers have nodes that share weights with other nodes in the layer. However, nodes in a fully connected layer receive input from each node in the next lowest neural network layer.

[0004] Patent Document 1 describes a neural network system implemented by one or more computers and configured to receive an input image and generate a network output of the input image. The neural network system includes a plurality of separable convolutional neural network layers, each consisting of separable convolutional subnetworks arranged sequentially in a stack. Each separable convolutional neural network layer is configured to generate a layer output by separately applying both depth convolution and point convolution during processing of the input to the separable convolutional neural network layer. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Special Publication No. 2019-535084 [Overview of the project] [Problems that the invention aims to solve]

[0006] The neural network system described in Patent Document 1 has the problem of enormous memory usage and processing time due to the enormous amount of computation required for convolution.

[0007] The purpose of this disclosure is to enable a reduction in the computational complexity of convolution. [Means for solving the problem]

[0008] The neural network system has a separable convolutional subnetwork, the separable convolutional subnetwork has a plurality of separable convolutional neural network layers arranged in a stack in order, each of the plurality of separable convolutional neural network layers applies a first grouping convolution to the input to the separable convolutional neural network layer, the input to the first grouping convolution includes a plurality of channels, the first grouping convolution is a spatial convolution that divides the channels of the input to the first grouping convolution into groups in the channel direction, convolves them, and combines them to produce an output. Each of the multiple separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and the second grouping convolution separately to the input to the separable convolutional neural network layer, the second grouping convolution being a spatial convolution that divides the channels calculated by the first grouping convolution into groups in the channel direction, convolves them, and combines them to obtain the output. ru. [Effects of the Invention]

[0009] According to this disclosure, the computational complexity of convolution can be reduced. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example of a neural network system configuration. [Figure 2] This is a block diagram showing an example configuration of the start module. [Figure 3] This is a block diagram showing an example of the configuration of an intermediate module. [Figure 4] This is a block diagram showing an example of the final module configuration. [Figure 5] This is a flowchart illustrating the processing method of a neural network system. [Figure 6] This is a block diagram showing an example of a hardware configuration. [Figure 7] This is a schematic diagram of a convolutional neural network layer. [Modes for carrying out the invention]

[0011] (First Embodiment) Similar reference numbers and names in various drawings refer to the same elements.

[0012] Figure 1 is a block diagram showing an example of the configuration of a neural network system 100 according to the first embodiment. The neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, and the systems, components, and techniques described later may be implemented.

[0013] The neural network system 100 has an input subnetwork 104, a separable convolutional subnetwork 150, and an output subnetwork 112, and takes an input image 102 as input and outputs a classification output 114 of the input image 102. The separable convolutional subnetwork 150 has a start module 106, an intermediate module 108, and a final module 110. The neural network system 100 processes the input image 102 using a deep neural network, such as a convolutional neural network.

[0014] The neural network system 100 has an input subnetwork 104. The input subnetwork 104 includes one or more convolutional neural network layers. The input subnetwork 104 receives an input image 102, processes the input image 102, and generates an initial output 116.

[0015] The separable convolutional subnetwork 150 has one or more start modules 106. One or more start modules 106 receive an initial output 116, process the initial output 116, and generate a start output 118.

[0016] The separable convolutional subnetwork 150 has one or more intermediate modules 108. One or more intermediate modules 108 receive the start output 118 as input, process the start output 118, and generate an intermediate output 120.

[0017] The final module 110 receives the intermediate output 120, processes the intermediate output 120, and generates a separable convolutional output 122 of the separable convolutional sub-network 150.

[0018] The neural network system 100 has an output sub-network 112. The output sub-network 112 receives the separable convolutional output 122. The output sub-network 112 processes the separable convolutional output 122 to generate a classification output 114 of the input image 102.

[0019] The architecture of the output sub-network 112 is specific to the task performed by the neural network system 100. For example, when the neural network system 100 performs an image classification task, the output sub-network 112 includes a softmax layer and, optionally, one or more fully connected layers preceding the softmax layer. These layers process the separable convolutional output 122 to generate a classification output 114 of the input image 102.

[0020] As described above, the neural network system 100 receives the input image 🔢. Next, the neural network system 100 uses the separable convolutional sub-network 150 to process the input image 102 and generate a classification output 114 of the input image 102.

[0021] The neural network system 100 may be configured to generate an output of any type of score, regression, or classification based on the input image 102, that is, it may be configured to perform any type of image processing task. The score or classification output generated by the neural network system 100 depends on the task that the neural network system 100 is configured to verify.

[0022] The neural network system 100 generates one or more scores or classification outputs in the output layer of the convolutional subnetwork 150, or in any suitable hidden layer among the one or more hidden layers of the convolutional subnetwork 150. For example, in the task of image classification or recognition, the outputs generated by the neural network system 100 for a given image may be scores for each of a set of object categories. Each score represents the likelihood that the image contains images of objects belonging to that category.

[0023] In another example, for an object detection task, the output generated by the neural network system 100 can identify the location, size, or both of the target object in the input image. In yet another example, the output generated by the neural network system 100 can be the output of one of the network's hidden layers and provided for use as a numerical representation of the input image, i.e., a numerical embedding.

[0024] The separable convolutional subnetwork 150 includes multiple separable convolutional neural network layers arranged in a stack. Each separable convolutional neural network layer includes a grouped pointwise convolution, i.e., a spatial convolution in which the input is divided into groups along the channel direction, convolved, and then combined to form the output. Following this, the separable convolutional neural network layer is followed by a pointwise convolution, i.e., a normal convolution with a 1x1 window, in which the computer-calculated channels are projected onto a new channel space. The pointwise convolution is a pointwise convolution.

[0025] Grouped convolution involves dividing the input channel Cin into G subgroups and grouping them. Then, using G kernels Kg(Cin / G), each responsible for convolution only within the Cin / G channel portion, the same processing as in normal convolution is performed in parallel, independently for each of the G groups. While normal convolution requires spatial filtering, grouped convolution decomposes the calculation into combinations of input channel Cin and kernel Kg × G groups, thus reducing the computational complexity by a factor of 1 / G. Furthermore, depthwise convolution can be considered as the case when the input channel Cin and the number of groups G are equal. Therefore, from here on, grouped convolution will be described as the case excluding the case where the input channel Cin and the number of groups G are equal.

[0026] Each separable convolutional neural network layer is configured to apply both grouping convolution and point-direction convolution separately while processing the input to the separable convolutional neural network layer. For example, the separable convolutional neural network layer first performs grouping convolution on the received input, dividing it into groups in the channel direction and convolving them, then combining them to produce an output; that is, it performs grouping convolution to generate an intermediate output. Then, the separable convolutional neural network layer performs a 1x1 convolution on the intermediate output, that is, point-direction convolution, to produce a layer output.

[0027] As another example, a separable convolutional neural network layer may first perform a 1x1 convolution on the received input to produce an intermediate output, and then perform a grouped convolution on the intermediate output to produce a layer output.

[0028] As another example, a separable convolutional neural network layer generates an intermediate output by performing a 1x1 convolution on the received input, i.e., a point-direction grouping convolution where the input is divided into groups in the channel direction, convolved, and then combined to form the output. Then, the separable convolutional neural network layer generates a layer output by performing a normal convolution on the intermediate output. In this case, the point-direction grouping convolution is decomposed into calculations for G groups of input channel Cin and kernel Kg, thus reducing the computational complexity by a factor of 1 / G. Here, when G is greater than 1, the overall computational complexity can be reduced.

[0029] Furthermore, by dividing the system into groups, the number of channels referenced by each group can be reduced while maintaining the dependency between input and output channels, allowing each group to learn a different kernel. This reduces computational cost through parallelization and improves expressive power through wider representation, without diminishing the learning effect.

[0030] Specifically, a separable convolutional subnetwork 150 includes a module array comprising one or more start modules 106, followed by one or more intermediate modules 108, and followed by one final module 110. For example, a separable convolutional subnetwork 150 may include three start modules 106. A separable convolutional subnetwork 150 may include eight intermediate modules 108.

[0031] Each module contains multiple separable convolutional neural network layers arranged sequentially in a stack. Each module is configured to process its module input and produce a module output of that module input. Each module then provides this module output as input to the next module. Each module provides the separable convolutional output 122 of the separable convolutional subnetwork 150 when there is no module following the current module, i.e., when the module is the last module in this sequence. Exemplary configurations of the starting module 106, intermediate module 108, and final module 110 will be described in detail later with reference to Figures 2 to 4.

[0032] Figure 2 is a block diagram showing an example configuration of the start module 106 in Figure 1. The start module 106 has a first stack 220, a maximum pooling layer 210, a concatenation layer 212, and a first pass-through convolutional layer 216, and takes an initial output 116 as input and outputs a start output 118.

[0033] The start module 106 receives an initial output 116 as input. In some cases, the initial output 116 is the initial output generated by the input subnetwork 104 from the input image 102. In other cases, the initial output 116 is the module output generated by the previous start module. In still other cases, the initial output 116 is the input image.

[0034] The starting module 106 has a first pass-through convolutional layer 216. The first pass-through convolutional layer 216 processes the initial output 116 to produce a first pass-through output 218. The first pass-through convolutional layer 216 is a 1x1 convolutional layer.

[0035] The starting module 106 has a first stack 220. The first stack 220 has a separable convolutional layer 204, a rectified linear unit (ReLU) activation layer 206, and a separable convolutional layer 208, which takes the initial output 116 as input and outputs the first stack output 222. The separable convolutional layers 204 and 208 are separable convolutional neural network layers. The first stack 220 processes the initial output 116 to generate the first stack output 222.

[0036] Each of the separable convolutional layers 204 and 208 generates a layer output by sequentially applying both grouping convolution and point-direction convolution separately during the processing of the input to the separable convolutional layer. The ReLU activation layer 206 is removable. The separable convolutional layer 204 provides its output as the input to the next separable convolutional layer 208. If there is no layer following the current layer, the separable convolutional layer 208 provides its output as the first stack output 222.

[0037] In one example, a ReLU activation layer 206 precedes one or more separable convolutional layers 204 and 208 in the first stack 220. For example, a ReLU activation layer 206 precedes a separable convolutional layer 208. The ReLU activation layer 206 applies a ReLU activation function to the output of the preceding separable convolutional layer 204 to produce a ReLU output. This ReLU output is provided as input to the next separable convolutional layer 208 that follows the ReLU activation layer 206.

[0038] In another example, one or more separable convolutional layers 204 and 208 in the first stack 220 are followed by a batch normalization layer that performs batch normalization. The batch normalization layer enables faster learning and more robust initialization by normalizing parts of the model architecture and normalizing each training minimum batch of training data. Batch normalization is described in detail in "Batch normalization: Accelerating deep network training by reducing internal covariate shift" by S. Ioffe and C. Szegedy in the proceedings of the 32nd International Conference on Machine Learning, pp. 448-456, 2015.

[0039] The starting module 106 has a max pooling layer 210. The max pooling layer 210 performs max pooling on the first stack output 222 to produce a max pooled output 224. Max pooling is Max Pooling.

[0040] The start module 106 has a concatenation layer 212. The concatenation layer 212 concatenates the first pass-through output 218 and the max-pooled output 224 to generate the start output 118 of the start module 106. For example, the concatenation layer 212 concatenates the tensors generated by the first pass-through convolutional layer 216 and the max-pooling layer 210 along the depth dimension to generate a single tensor of the start module 106, i.e., the start output 118.

[0041] The start output 118 is then provided as input to another start module 106 or intermediate module 108.

[0042] Figure 3 is a block diagram showing an example configuration of the intermediate module 108 in Figure 1. The intermediate module 108 has more separable convolutional neural network layers than the start module 106, and unlike the start module 106, it does not include a maximum pooling layer.

[0043] The intermediate module 108 has a second stack 320, a second pass-through convolutional layer 322, and a concatenation layer 316, and takes the start output 118 as input and outputs the intermediate output 120. The intermediate module 108 takes the start output 118 as input. For example, the start output 118 is the start output generated by the start module 106. In other examples, the start output 118 is the module output generated by the previous intermediate module 108.

[0044] The intermediate module 108 has a second pass-through convolutional layer 322. The second pass-through convolutional layer 322 processes the start output 118 to generate a second pass-through output 324. The second pass-through convolutional layer 322 is a 1x1 convolutional layer.

[0045] The intermediate module 108 has a second stack 320. The second stack 320 has ReLU activation layers 304, 308, 312 and separable convolutional layers 306, 310, 314, and takes the start output 118 as input and outputs the second stack output 326. The separable convolutional layers 306, 310, 314 are separable convolutional neural network layers. The second stack 320 processes the start output 118 to generate the second stack output 326.

[0046] Specifically, each of the separable convolutional layers 306, 310, and 314 generates a layer output by sequentially applying both grouped convolution and point-direction convolution separately during the processing of the input to the separable convolutional layer. The ReLU activation layers 304, 308, and 312 are removable. Separable convolutional layers 306 and 310 provide their outputs as inputs to the next separable convolutional layers 310 and 314, respectively. If there is no layer following the current layer, separable convolutional layer 314 provides its output as a second stack output 326.

[0047] For example, ReLU activation layers 304, 308, and 312 precede each of the separable convolutional layers 306, 310, and 314, respectively. For example, ReLU activation layer 304 precedes separable convolutional layer 306. ReLU activation layer 308 precedes separable convolutional layer 310. ReLU activation layer 312 precedes separable convolutional layer 314.

[0048] The ReLU activation layer 304 applies the ReLU activation function to the start output 118 to generate an intermediate output. This intermediate output is provided as input to the subsequent separable convolutional layer 306. The ReLU activation layer 308 applies the ReLU activation function to the output of the previous separable convolutional layer 306 to generate an intermediate output. This intermediate output is provided as input to the next separable convolutional layer 310. The ReLU activation layer 312 applies the ReLU activation function to the output of the previous separable convolutional layer 310 to generate an intermediate output. This intermediate output is provided as input to the next separable convolutional layer 314.

[0049] Furthermore, a batch normalization layer may be placed after one or more of the separable convolutional layers 306, 310, and 314.

[0050] The intermediate module 108 has a coupling layer 316. The coupling layer 316 couples a second pass-through output 324 and a second stack output 326 to generate an intermediate output 120 of the intermediate module 108. The intermediate output 120 is provided as an input to another intermediate module 108 or the final module 110.

[0051] Figure 4 is a block diagram showing an example configuration of the final module 110 in Figure 1. The final module 110 has a third stack 420, a maximum pooling layer 412, a third pass-through convolutional layer 432, a concatenation layer 414, a fourth stack 430, and an average pooling layer 426. The final module 110 takes an intermediate output 120 as input and outputs a separable convolutional output 122.

[0052] The final module 110 receives the intermediate output 120 as input. The intermediate output 120 is an intermediate output generated by the intermediate module 108. The final module 110 has a third pass-through convolutional layer 432. The third pass-through convolutional layer 432 processes the intermediate output 120 to generate a third pass-through output 434. The third pass-through convolutional layer 432 is a 1x1 convolutional layer.

[0053] The final module 110 has a third stack 420. The third stack 420 has ReLU activation layers 404, 408 and separable convolutional layers 406, 410. The separable convolutional layers 406 and 410 are separable convolutional neural network layers. The third stack 420 processes the intermediate output 120 to produce a third stack output 436.

[0054] Specifically, each of the separable convolutional layers 406 and 410 applies both grouping convolution and point-direction convolution sequentially and separately during processing of the input to the separable convolutional layer to generate a layer output. The ReLU activation layers 404 and 408 are removable. The separable convolutional layer 406 provides its output as input to the next separable convolutional layer 410. If there is no layer following the current layer, the separable convolutional layer 410 provides its output as a third stack output 436.

[0055] For example, a ReLU activation layer 404 precedes a separable convolutional layer 406. A ReLU activation layer 408 precedes a separable convolutional layer 410. The ReLU activation layer 404 applies the ReLU activation function to the intermediate output 120 to generate an intermediate output. This intermediate output is provided as input to the subsequent separable convolutional layer 406. The ReLU activation layer 408 applies the ReLU activation function to the output of the previous separable convolutional layer 406 to generate an intermediate output. This intermediate output is provided as input to the next separable convolutional layer 410.

[0056] Furthermore, a batch normalization layer may be placed after one or more of the separable convolutional layers 406 and 410.

[0057] The final module 110 has a max pooling layer 412. The max pooling layer 412 performs max pooling on the third stack output 436 to produce a max pooled output 438.

[0058] The final module 110 has a concatenation layer 414. The concatenation layer 414 concatenates the third pass-through output 434 and the max pooled output 438 to produce a concatenated output 440. For example, the concatenation layer 414 concatenates the tensors generated by the third pass-through convolutional layer 432 and the max pooling layer 412 along the depth dimension to produce a single tensor, i.e., the concatenated output 440.

[0059] The final module 110 has a fourth stack 430. The fourth stack 430 has ReLU activation layers 416, 422 and separable convolutional layers 418, 424. The fourth stack 430 processes the concatenated output 440 to produce a fourth stack output 442.

[0060] Specifically, each of the separable convolutional layers 418 and 424 applies both grouping convolution and point-direction convolution sequentially and separately during processing of the input to the separable convolutional layer to generate a layer output. The ReLU activation layers 416 and 422 are removable. The separable convolutional layer 418 provides its output as input to the next separable convolutional layer 424. The separable convolutional layer 424 provides its output as a fourth stack output 442.

[0061] For example, a ReLU activation layer 416 precedes a separable convolutional layer 418. A ReLU activation layer 422 precedes a separable convolutional layer 424. The ReLU activation layer 416 applies the ReLU activation function to the concatenated output 440 to produce an intermediate output. This intermediate output is provided as input to the subsequent separable convolutional layer 418. The ReLU activation layer 422 applies the ReLU activation function to the output of the previous separable convolutional layer 418 to produce an intermediate output. This intermediate output is provided as input to the next separable convolutional layer 424.

[0062] Furthermore, a batch normalization layer may be placed after one or more of the separable convolutional layers 418 and 424.

[0063] The final module 110 has an average pooling layer 426. The average pooling layer 426 performs average pooling on the fourth stack output 442 to generate a separable convolutional output 122. Average pooling is the same as above. The separable convolutional output 122 is the output of the separable convolutional subnetwork 150 in Figure 1.

[0064] Figure 5 is a flowchart illustrating the processing method of the neural network system 100 in Figure 1. The flowchart in Figure 5 is performed by one or more computer systems located in one or more locations. The neural network system 100 in Figure 1 performs the processing shown in the flowchart in Figure 5.

[0065] In step S501, the neural network system 100 receives the input image 102.

[0066] In step S502, the neural network system 100 processes the input image 102 using the neural network system 100, which includes a separable convolutional subnetwork 150, to generate a classification output 114 of the input image 102. For example, the input subnetwork 104 processes the input image 102 to generate an initial output 116. The separable convolutional subnetwork 150 processes the initial output 116 to generate a separable convolutional output 122. The output subnetwork 112 processes the separable convolutional output 122 to generate a classification output 114.

[0067] The separable convolutional subnetwork 150 includes a module array comprising one or more start modules 106, followed by one or more intermediate modules 108, and then one final module 110. Each module in this array comprises multiple separable convolutional neural network layers arranged in a stack in order. The separable convolutional subnetwork 150 processes the input image 102 or initial output 116 through each of the modules in the array to generate a separable convolutional output 122.

[0068] For example, the neural network system 100 uses an output subnetwork 112 to process a separable convolutional output 122 to generate a classification output 114 of the input image 102. In another example, the output subnetwork 112 may include a softmax layer and optionally include one or more fully connected layers preceding the softmax layer. These layers process the separable convolutional output 122 to generate a classification output 114 of the input image 102. In yet another example, the output subnetwork 112 may include a logistic regression layer and optionally include one or more fully connected neural network layers preceding the logistic regression layer. These layers process the separable convolutional output 122 to generate a classification output 114 of the input image 102.

[0069] The output of the neural network system 100 can be any type of score, regression, or classification output, based on the image processing task that the neural network system 100 is configured to perform. For example, the neural network system 100 may be configured to perform the task of image classification or recognition. In that case, the output of the neural network system 100 may include individual scores for each of a given set of object categories. The score for a given object category represents the likelihood that the input image contains an image of an object belonging to that object category.

[0070] In another example, for an object detection task, the output of the neural network system 100 can identify the location, size, or both of the target object in the input image. In yet another example, the output of the neural network system 100 can be the output of one of the network's hidden layers and can be provided for use as a numerical representation of the input image, i.e., a numerical embedding.

[0071] In step S503, the neural network system 100 provides the user with a classification output 114. For example, the output of the neural network system 100 may be data that identifies one or more object categories with the highest scores. The neural network system 100 may provide the network output, or the derived output, for display on the user interface of the user device. Alternatively, the neural network system 100 may store the network output, or the data derived from the network output, in association with the input image 102.

[0072] The neural network system 100 can train a separable convolutional subnetwork 150 on labeled training data using supervised learning techniques, such as stochastic gradient descent with backpropagation. Similarly, the neural network system 100 can train an input subnetwork 104 and an output subnetwork 112. In other words, the neural network system 100 can train a separable convolutional subnetwork 150, etc., using supervised learning techniques to determine the training values ​​of the subnetwork 150's parameters from its initial values.

[0073] Figure 6 is a block diagram showing an example of the hardware configuration of the neural network system 100 shown in Figure 1. The neural network system 100 is, for example, a personal computer or a smartphone. In this embodiment, the neural network system 100 will be described as a personal computer. The neural network system 100 has a CPU 601, ROM 602, RAM 603, HDD 604, display 605, keyboard 606, pointing device 607, data communication unit 608, and data bus 609.

[0074] The CPU (Central Processing Unit / Processor) 601 comprehensively controls the neural network system 100 and, for example, reads a program stored in ROM 602 into RAM 603 and executes it, thereby realizing the function shown in Figure 1. In Figure 6, there is one CPU 601, but it may be composed of multiple CPUs. ROM 602 is a general-purpose ROM and stores, for example, programs executed by the CPU 601. RAM 603 is a general-purpose RAM and is used, for example, as working memory to temporarily store various information when the CPU 601 executes a program. HDD (Hard Disk Drive) 604 is a storage medium (storage unit) for storing image files and processing results such as image classification.

[0075] The display 605 is a display unit that displays the user interface (UI) of this embodiment and a score that identifies the object category of the image data (hereinafter also referred to as "image") to the user. The keyboard 606 and pointing device 607 accept user instructions. The display 605 may also have a touch sensor function. The keyboard 606 is used, for example, when the user inputs an image to be classified on the UI displayed on the display 605. The pointing device 607 is used, for example, when the user clicks a button on the UI displayed on the display 605.

[0076] The data communication unit 608 communicates with external devices via a wired or wireless network. For example, the data communication unit 608 transmits a score that identifies the object category of an image to a server that can communicate with the neural network system 100. The data bus 609 connects the blocks in Figure 6 so that they can communicate with each other.

[0077] Note that the configuration shown in Figure 6 is merely an example and is not limited to this configuration. For example, the neural network system 100 does not have a display 605, and the UI may be displayed on an external display.

[0078] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0079] The neural network system 100 can be implemented in one or more storage media encoded by instructions. When these instructions are implemented by one or more computers, the neural network system 100 is implemented in one or more computers.

[0080] This embodiment can achieve one or more of the following advantages: The image processing neural network system 100 can achieve results that are comparable to or better than state-of-the-art technologies when performing a variety of image processing tasks, such as image classification or object recognition. In addition, the image processing neural network system 100 has far more parameters than those described above, and stores parameter values ​​using far more computing resources, namely memory and processing time. Therefore, the neural network system 100 can achieve better results than conventional neural networks. In other words, the neural network system 100 can achieve these results with a smaller computer footprint than conventional neural networks, meaning it uses less memory and requires less processing time.

[0081] As described above, the neural network system 100 has separable convolutional subnetworks 150, takes an input image 102 as input, and generates a network output of the input image 102.

[0082] The separable convolutional subnetwork 150 has multiple separable convolutional neural network layers arranged in a stack in order, such as a first stack 220, a second stack 320, a third stack 420, or a fourth stack 430. Each of the multiple separable convolutional neural network layers applies at least a first grouping convolution to its input. The input to the first grouping convolution includes multiple channels. The first grouping convolution is a spatial convolution that divides the channels of the input to the first grouping convolution into groups in the channel direction, convolves them, and combines them to obtain the output.

[0083] Specifically, each of the multiple separable convolutional neural network layers generates a layer output by sequentially applying a first grouping convolution and a point-direction convolution to the input to the separable convolutional neural network layer. The point-direction convolution is a 1x1 convolution that projects the channels computed by the first grouping convolution onto a new channel space.

[0084] Furthermore, each of the multiple separable convolutional neural network layers may generate a layer output by sequentially applying a first grouping convolution and a normal convolution separately to the input to the separable convolutional neural network layer. The normal convolution is a normal convolution on the channel computed by the first grouping convolution.

[0085] The neural network system 100 has an input subnetwork 104 and an output subnetwork 112. The input subnetwork 104 processes the network input to generate an initial output 116. The input subnetwork 104 has one or more convolutional neural network layers. The output subnetwork 112 processes the outputs of separable convolutional subnetworks 150 to generate a network output.

[0086] The separable convolutional subnetwork 150 has one or more start modules 106. The start module 106 processes the initial output 116 to generate the output of the start module 106.

[0087] The start module 106 has a first pass-through convolutional layer 216, a first stack 220, a first max-pooling layer 210, and a first concatenation layer 212. The first pass-through convolutional layer 216 processes the input of the start module 106 to produce a first pass-through output 218. The first stack 220 is the first stack of separable convolutional neural network layers and processes the input of the start module 106 to produce a first stack output 222. The first max-pooling layer 210 performs max-pooling on the first stack output 222 to produce a first max-pooled output 224. The first concatenation layer 212 concatenates the first pass-through output 218 and the first max-pooled output 224 to produce an output of the start module 106. For example, a separable convolutional subnetwork 150 has three start modules 106.

[0088] The first pass-through convolutional layer 216 is a 1x1 convolutional layer. The first rectified linear unit activation layer 206 is provided before one or more of the separable convolutional neural network layers 204,208 in the first stack 220. Alternatively, the first batch normalization layer may be provided after one or more of the separable convolutional neural network layers 204,208 in the first stack 220.

[0089] The separable convolutional subnetwork 150 has one or more intermediate modules 108. The intermediate module 108 processes the output of the start module 106 to generate the output of the intermediate module 108.

[0090] The intermediate module 108 has a second passthrough convolutional layer 322, a second stack 320, and a second concatenation layer 316. The second passthrough convolutional layer 322 processes the input of the intermediate module 108 to produce a second passthrough output 324. The second stack 320 is a second stack of separable convolutional neural network layers that processes the input of the intermediate module 108 to produce a second stack output 326. The second concatenation layer 316 concatenates the second passthrough output 324 and the second stack output 326 to produce an output of the intermediate module 108. For example, a separable convolutional subnetwork 150 has eight intermediate modules 108.

[0091] The second pass-through convolutional layer 322 is a 1x1 convolutional layer. The second rectified linear unit activation layers 304, 308, 312 are provided before one or more of the separable convolutional neural network layers 306, 310, 314 in the second stack 320. Alternatively, the second batch normalization layer may be provided after one or more of the separable convolutional neural network layers 306, 310, 314 in the second stack 320.

[0092] The separable convolutional subnetwork 150 has a final module 110. The final module 110 processes the output of the intermediate module 108 to produce the output of the final module 110.

[0093] The final module 110 has a third pass-through convolutional layer 432, a third stack 420, a second max pooling layer 412, a third concatenation layer 414, a fourth stack 430, and an average pooling layer 426. The third pass-through convolutional layer 432 processes the output of the intermediate module 108 to produce a third pass-through output 434. The third stack 420 is a third stack of separable convolutional neural network layers that processes the output of the intermediate module 108 to produce a third stack output 436. The second max pooling layer 412 performs max pooling on the third stack output 436 to produce a second max pooled output 438. The third concatenation layer 414 concatenates the third pass-through output 434 and the second max pooled output 438 to produce a concatenated output 440.

[0094] The third pass-through convolutional layer 432 is a 1x1 convolutional layer. A third rectified linear unit activation layer 404,408 is provided before one or more of the separable convolutional neural network layers 406,410 in the third stack 420. Alternatively, a third batch normalization layer may be provided after one or more of the separable convolutional neural network layers 406,410 in the third stack 420.

[0095] The fourth stack 430 is a fourth stack of separable convolutional neural network layers that processes the concatenated output 440 to produce a fourth stack output 442. The mean pooling layer 426 performs mean pooling on the fourth stack output 442 to produce the output of the final module 110.

[0096] A fourth rectified linear unit activation layer 416,422 is provided before one or more of the separable convolutional neural network layers 418,424 in the fourth stack 430. Alternatively, a fourth batch normalization layer may be provided after one or more of the separable convolutional neural network layers 418,424 in the fourth stack 430.

[0097] As described above, according to this embodiment, the neural network system 100 can reduce the computational complexity of convolution, thereby reducing memory usage and processing time.

[0098] (Second embodiment) In the first embodiment, we described an example of reducing computational complexity without reducing learning effectiveness by using grouped convolution compared to performing spatial convolution on the entire model. While this method reduces the load of performing spatial convolution, the computational complexity of point-direction convolution increases as the number of channels increases. The second embodiment aims to reduce the computational complexity of point-direction convolution without reducing learning effectiveness. The differences between the second embodiment and the first embodiment will be explained below.

[0099] In the second embodiment, the separable convolutional layers in Figures 2-4 first perform grouping convolution with respect to the input, dividing it into groups in the channel direction and convolving them, then combining them to produce an output, that is, they perform grouping convolution to generate an intermediate output. Next, the separable convolutional layers in Figures 2-4 perform 1x1 convolution with respect to the intermediate output, that is, point-direction grouping convolution, dividing it into groups in the intermediate channel direction, convolving it, and combining them to produce an output, thereby generating a layer output.

[0100] In this case, the separable convolutional layer is decomposed into calculations for G groups of combinations of input channel Cin and kernel Kg, thus reducing the computational complexity by a factor of 1 / G. In this case, based on the output channel Cout, the number of groups in the grouped convolution g, and the number of groups in the point-direction grouped convolution Cin / g, the quadratic equation f(g) in g and the discriminant D are expressed by the following equations.

[0101]

number

[0102] When f(g) < 0, D ≥ 0, and a solution g exists. Therefore, using grouped convolution reduces the computational complexity compared to using depthwise convolution under the above conditions. Here, depthwise convolution can be considered as the case when the input channel Cin is equal to the number of groups G.

[0103] Furthermore, this embodiment learns kernels for different groups while maintaining the dependency between input and output channels, even when the number of channels referenced by each group is reduced by dividing the system into groups. As a result, it is expected that computational costs will be reduced through parallelization, as well as expressive power will be improved through wide representation, without reducing the learning effect.

[0104] As another example, a separable convolutional layer performs a 1x1 convolution on the received input, i.e., a pointwise grouped convolution where the input is divided into groups along the channel direction, convolved, and then combined to produce an intermediate output. The separable convolutional layer then performs a normal convolution on the intermediate output to produce the layer output. Pointwise grouped convolution is also known as Grouped Pointwise Convolution.

[0105] Figure 7 is a schematic representation of a convolutional neural network layer. In a typical convolution (Conv 3x3), the 3x3 convolution from input to output can be expanded in both the spatial and channel directions, with the direction of convolution for each point shown by the solid line.

[0106] Depthwise convolution and pointwise convolution can be represented as shown in Figure 7. Similarly, grouped convolution and grouped pointwise convolution can also be represented as shown in Figure 7.

[0107] In this embodiment, a separable convolutional layer generates a layer output by performing grouping convolution and point-direction grouping convolution with respect to the intermediate output. In this case, it is shown that even if the separable convolutional layer is separated with respect to the intermediate output channels by grouping, the output channels can form the final output by performing convolution across channels that are not convolved with the intermediate output channels. In other words, a separable convolutional layer can generate a convolutional layer output that references all channels.

[0108] As described above, the neural network system 100 has a separable convolutional subnetwork 150. The separable convolutional subnetwork 150 has a plurality of separable convolutional neural network layers arranged in a stack in order, such as a first stack 220, a second stack 320, a third stack 420, or a fourth stack 430.

[0109] Each of the multiple separable convolutional neural network layers generates a layer output by sequentially applying a first grouping convolution and a second grouping convolution to the input to the separable convolutional neural network layer. The first grouping convolution is a spatial convolution that divides the channels of the input to the first grouping convolution into groups in the channel direction, convolves them, and then combines them to produce the output. The second grouping convolution is a spatial convolution that divides the channels computed by the first grouping convolution into groups in the channel direction, convolves them, and then combines them to produce the output.

[0110] As described above, according to this embodiment, the neural network system 100 can reduce the computational amount of convolution in the point direction without reducing the learning effect.

[0111] (Third Embodiment) In the second embodiment, an example of reducing the computational amount of convolution in the point direction without reducing the learning effect was described. The purpose of the third embodiment is to reduce the computational amount without reducing the learning effect when using a method of increasing the number of channels in the intermediate layer in order to improve the expressiveness. Hereinafter, the differences between the third embodiment and the first embodiment will be described.

[0112] In the third embodiment, the separable convolutional layer in FIGS. 2 to 4 first performs a 1×1 convolution on the received input, that is, performs grouped convolution in the input channel direction by dividing into groups, combining and outputting as a point-direction grouped convolution to generate an intermediate output. Next, the separable convolutional layer in FIGS. 2 to 4 performs grouped convolution by dividing the intermediate output into groups in the channel direction, combining and outputting, that is, performs grouped convolution to generate a second intermediate output. Next, the separable convolutional layer in FIGS. 2 to 4 performs a 1×1 convolution on the second intermediate output, that is, performs grouped convolution by dividing the second intermediate output into groups in the channel direction, combining and outputting as a point-direction grouped convolution to generate a layer output.

[0113] Here, let the expansion rate t of the intermediate number of channels, the output channel Cout, the number of groups g of the grouped convolution, and the number of groups Cin / g of the first and second point-direction grouped convolutions be set. Then, under the condition of 1 < g < Cin, the separable convolutional layer can reduce the computational amount by using grouped convolution compared to the case of using depthwise convolution.

[0114] Note that any one of these point-direction grouped convolutions may generate an output from a normal convolutional layer.

[0115] As described above, the neural network system 100 has a separable convolutional subnetwork 150. The separable convolutional subnetwork 150 has a plurality of separable convolutional neural network layers arranged in a stack in order, such as a first stack 220, a second stack 320, a third stack 420, or a fourth stack 430.

[0116] Each of the multiple separable convolutional neural network layers generates a layer output by sequentially applying a first grouping convolution, a second grouping convolution, and a third grouping convolution to the input to the separable convolutional neural network layer. The first grouping convolution is a spatial convolution that divides the channels of the input to the first grouping convolution into groups in the channel direction, convolves them, and combines them to produce the output. The second grouping convolution is a spatial convolution that divides the channels calculated by the first grouping convolution into groups in the channel direction, convolves them, and combines them to produce the output. The third grouping convolution is a spatial convolution that divides the channels calculated by the second grouping convolution into groups in the channel direction, convolves them, and combines them to produce the output.

[0117] As described above, according to this embodiment, when the neural network system 100 uses a method to increase the number of channels in the intermediate layer in order to improve expressiveness, it is possible to reduce the computational load without reducing the learning effect.

[0118] (Other embodiments) This specification uses the term “configured” in relation to systems and computer program components. In the case of a system of one or more computers that will be configured to perform a particular operation or action, the system has software, firmware, hardware, or a combination thereof installed on it. These, in operation, cause the system to perform the operation or action. In the case of one or more computer programs that will be configured to perform a particular operation or action, the programs include instructions, which, when executed by a data processing device, cause the device to perform the operation or action.

[0119] Embodiments include the above-described structures and their structural equivalents, and can be implemented as tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof. Embodiments of the subject matter described herein are executed by one or more computer programs, i.e., data processing devices, or can be implemented as one or more modules of computer program instructions encoded in a tangible non-temporary storage medium for controlling its operation. The computer storage medium can be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, program instructions may be encoded in artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals generated to encode information to be transmitted to a receiving device suitable for execution by a data processing device.

[0120] The term "Data Processing Apparatus" refers to data processing hardware, including, for example, a programmable processor, a computer, or multiple processors or computers. It also encompasses all types of devices and machines for processing data. An apparatus may also be, or further include, a dedicated logic circuit mechanism, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). Optionally, in addition to hardware, an apparatus may include code that creates an environment for executing computer programs. This code may, for example, constitute processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0121] Computer programs are sometimes also called programs, software, software applications, apps, modules, software modules, scripts, or code. Computer programs can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages. They can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. Programs can, but are not necessarily, correspond to files in a file system. A program may be stored in a single file dedicated to the program in question, as part of a file holding other programs or data stored in a markup language document, such as one or more scripts; or in multiple coordinate files, such as files storing one or more modules, subprograms, or parts of code. Computer programs can be deployed to run on one or more computers interconnected by a data communication network, either located at one site or distributed across multiple sites.

[0122] The methods and logic flows described herein can be carried out by one or more programmable computers running one or more computer programs to perform functions by manipulating input data and producing outputs. The methods and logic flows can also be carried out by dedicated logic circuit mechanisms, such as FPGAs or ASICs, or by a combination of dedicated logic circuit mechanisms and one or more programmed computers.

[0123] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. The central processing unit will receive instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for issuing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be complemented by or incorporated into a dedicated logic circuit mechanism. A computer may also include, or be operably coupled to, one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks. The computer then receives data from or transfers data to or both. However, a computer does not necessarily need to have such devices. Moreover, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), or mobile audio or video player. Furthermore, computers can be embedded in game consoles, Global Positioning System (GPS) receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives, but these are just a few examples.

[0124] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices. Computer-readable media also include all forms of non-volatile memory, media, and memory devices, including magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs.

[0125] To enable interaction with the user, a computer has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user. The computer also has a keyboard and pointing device, such as a mouse or trackball, from which the user can provide input. Interaction with the user can also be achieved using other types of devices. For example, the feedback given to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback. Input from the user can be received in any form, including acoustic, speech, or tactile input. A computer can interact with the user by sending and receiving documents to and from the device the user is using, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. A computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0126] A data processing device for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for handling the common computationally intensive parts of machine learning training or outcomes, i.e., inference, and the workload.

[0127] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0128] Embodiments of the subject matter described herein can be implemented in a computing system. The computing system includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer. This client computer has a graphical user interface, a web browser, or an app. Through these, a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0129] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship arises when computer programs run on separate computers and have a client / server relationship with each other. For example, a server sends data, such as an HTML page, to a user device, for the purpose of displaying the data on a device acting as a client and receiving user input from a user interacting with the device. Data generated on the user device, such as the results of user interaction, can be sent from the device and received by the server.

[0130] This specification includes many specific implementation details, which should not be interpreted as limitations on the scope of any claims or assertions, but rather as descriptions of features that may be specific to particular embodiments. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable partial combination. Furthermore, with respect to features, it may be stated above that they function in certain combinations, and even initially asserted as such, but for example, one or more features from the asserted combination may be removed from this combination. The asserted combination may be a partial combination or a variation of a partial combination.

[0131] The operations are depicted in the drawings in a specific order and enumerated in the claims. This should not be interpreted as meaning that such operations must be performed in a specific order or sequence as shown, or that all the exemplified operations must be performed to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be interpreted as meaning that such separation is necessary in all embodiments. It should be understood that the described program components and systems may be integrated together in a single software product or packaged in multiple software products.

[0132] We have described specific embodiments of the subject matter. Other embodiments are within the scope of the claims. For example, the actions enumerated in the claims may be performed in a different order, and the desired results may still be achieved. As one example, the method shown in the drawings does not necessarily require the specific order or sequence shown to achieve the desired results. For example, multitasking and parallel processing may be advantageous.

[0133] Furthermore, the embodiments described above are merely examples illustrating how to implement this disclosure, and they should not be interpreted as limiting the technical scope of this disclosure. In other words, this disclosure can be implemented in various ways without departing from its technical concept or its main features.

[0134] This embodiment includes the following configurations, methods, and programs. (Composition 1) It has separable convolutional subnetworks, The separable convolutional subnetwork has a plurality of separable convolutional neural network layers arranged in a stack in sequence. Each of the plurality of separable convolutional neural network layers applies a first grouping convolution to the input to the separable convolutional neural network layer. The input to the first grouped convolution includes multiple channels, The neural network system is characterized in that the first grouping convolution is a spatial convolution in which the channels of the input to the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to form the output. (Configuration 2) Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and point-direction convolution separately to the input to the separable convolutional neural network layer. The neural network system according to configuration 1, characterized in that the point-direction convolution is a 1x1 convolution that projects the channels calculated by the first grouping convolution onto a new channel space. (Composition 3) Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and the normal convolution separately to the input to the separable convolutional neural network layer. The neural network system according to configuration 1, characterized in that the conventional convolution is a conventional convolution on the channels calculated by the first grouped convolution. (Composition 4) Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and the second grouping convolution separately to the input to the separable convolutional neural network layer. The neural network system according to configuration 1, characterized in that the second grouping convolution is a spatial convolution in which the channels calculated by the first grouping convolution are divided into groups in the channel direction, convolved, and combined to obtain the output. (Composition 5) Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution, the second grouping convolution, and the third grouping convolution separately to the input to the separable convolutional neural network layer. The second grouping convolution described above is a spatial convolution in which the channels calculated by the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to obtain the output. The neural network system according to configuration 1, characterized in that the third grouping convolution is a spatial convolution in which the channels calculated by the second grouping convolution are divided into groups in the channel direction, convolved, and combined to obtain the output. (Composition 6) The neural network system according to any one of configurations 1 to 5, further comprising an output subnetwork that processes the output of the separable convolutional subnetwork to generate a network output. (Composition 7) A neural network system according to any one of configurations 1 to 6, further comprising an input subnetwork that processes network inputs and generates an initial output. (Composition 8) The neural network system according to configuration 7, characterized in that the input subnetwork has one or more convolutional neural network layers. (Composition 9) The separable convolutional subnetwork has one or more initiation modules, The neural network system according to configuration 7 or 8, characterized in that the start module processes the initial output to generate the output of the start module. (Composition 10) The aforementioned startup module is A first pass-through convolutional layer processes the input of the start module and generates a first pass-through output, A first stack of the separable convolutional neural network layer, comprising a first stack that processes the input of the start module and generates a first stack output, A first max pooling layer that performs max pooling on the first stack output to generate a first max pooled output, A first coupling layer that couples the first pass-through output and the first maximum pooled output to generate the output of the start module, The neural network system according to configuration 9, characterized by having the following: (Composition 11) The neural network system according to configuration 9 or 10, characterized in that the separable convolutional subnetwork has three start modules. (Composition 12) The neural network system according to configuration 10, characterized in that the first pass-through convolutional layer is a 1x1 convolutional layer. (Composition 13) The neural network system according to configuration 10 or 12, characterized in that a first rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the first stack. (Composition 14) The neural network system according to any one of configurations 10, 12, and 14, characterized in that a first batch normalization layer is provided after one or more of the separable convolutional neural network layers in the first stack. (Composition 15) The separable convolutional subnetwork has one or more intermediate modules, The neural network system according to any one of configurations 9 to 14, wherein the intermediate module processes the output of the start module to generate the output of the intermediate module. (Composition 16) The aforementioned intermediate module is A second pass-through convolutional layer processes the input of the aforementioned intermediate module to generate a second pass-through output, A second stack of the separable convolutional neural network layer, comprising a second stack that processes the input of the intermediate module to generate a second stack output, A second coupling layer that couples the second pass-through output and the second stack output to generate the output of the intermediate module. The neural network system according to configuration 15, characterized by having the following: (Composition 17) The neural network system according to configuration 15 or 16, characterized in that the separable convolutional subnetwork has eight intermediate modules. (Composition 18) The neural network system according to configuration 16, characterized in that the second pass-through convolutional layer is a 1x1 convolutional layer. (Composition 19) The neural network system according to configuration 16 or 18, characterized in that a second rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the second stack. (Composition 20) The neural network system according to any one of configurations 16, 18, and 19, characterized in that a second batch normalization layer is provided after one or more of the separable convolutional neural network layers in the second stack. (Composition 21) The separable convolutional subnetwork has a final module, The neural network system according to any one of the configurations 15 to 20, wherein the final module processes the output of the intermediate module to generate the output of the final module. (Composition 22) The aforementioned final module is A third pass-through convolutional layer processes the output of the aforementioned intermediate module to generate a third pass-through output, A third stack of the separable convolutional neural network layer, comprising a third stack that processes the output of the intermediate module to generate a third stack output, A second max pooling layer performs max pooling on the third stack output to generate a second max pooled output, A third coupling layer that couples the third pass-through output and the second maximum pooled output to generate a coupled output. The neural network system according to configuration 21, characterized by having the following. (Composition 23) The neural network system according to configuration 22, characterized in that the third pass-through convolutional layer is a 1x1 convolutional layer. (Composition 24) The neural network system according to configuration 22 or 23, characterized in that a third rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the third stack. (Composition 25) The neural network system according to any one of configurations 22 to 24, characterized in that a third batch normalization layer is provided after one or more of the separable convolutional neural network layers in the third stack. (Composition 26) The aforementioned final module is A fourth stack of the separable convolutional neural network layer, the fourth stack processing the concatenated outputs to generate a fourth stack output, An average pooling layer that performs average pooling on the fourth stack output to generate the output of the final module, A neural network system according to any one of configurations 22 to 25, further comprising the above. (Composition 27) The neural network system according to configuration 26, characterized in that a fourth rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the fourth stack. (Composition 28) The neural network system according to configuration 26 or 27, characterized in that a fourth batch normalization layer is provided after one or more of the separable convolutional neural network layers in the fourth stack. (Composition 29) The neural network system is characterized by receiving an input image and generating a network output of the input image, as described in any one of configurations 1 to 28. (Method 1) A processing method for a neural network system, The aforementioned neural network system has separable convolutional subnetworks, The separable convolutional subnetwork has a plurality of separable convolutional neural network layers arranged in a stack in sequence. Each of the plurality of separable convolutional neural network layers applies a first grouping convolution to the input to the separable convolutional neural network layer. The input to the first grouped convolution includes multiple channels, A processing method for a neural network system, characterized in that the first grouping convolution is a spatial convolution in which the channels of the input to the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to obtain the output. (Program 1) A program to make a computer function as a neural network system as described in one of the configurations 1 through 29. [Explanation of Symbols]

[0135] 100 Neural network system, 102 Input image, 104 Input subnetwork, 106 Start module, 108 Intermediate module, 110 Final module, 112 Output subnetwork, 114 Classification output, 116 Initial output, 118 Start output, 120 Intermediate output, 122 Separable convolutional output, 150 Separable convolutional subnetwork

Claims

1. It has separable convolutional subnetworks, The separable convolutional subnetwork has a plurality of separable convolutional neural network layers arranged in a stack in sequence. Each of the plurality of separable convolutional neural network layers applies a first grouping convolution to the input to the separable convolutional neural network layer. The input to the first grouping convolution includes multiple channels, The first grouping convolution described above is a spatial convolution in which the channels of the input to the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to obtain the output. Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and the second grouping convolution separately to the input to the separable convolutional neural network layer. The neural network system is characterized in that the second grouping convolution is a spatial convolution in which the channels calculated by the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to obtain the output.

2. Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and point-direction convolution separately to the input to the separable convolutional neural network layer. The neural network system according to claim 1, characterized in that the point-direction convolution is a 1x1 convolution that projects the channels calculated by the first grouping convolution onto a new channel space.

3. Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and the normal convolution separately to the input to the separable convolutional neural network layer. The neural network system according to claim 1, characterized in that the conventional convolution is a conventional convolution on the channels calculated by the first grouped convolution.

4. Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution, the second grouping convolution, and the third grouping convolution separately to the input to the separable convolutional neural network layer. The second grouping convolution described above is a spatial convolution in which the channels calculated by the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to obtain the output. The neural network system according to claim 1, characterized in that the third grouping convolution is a spatial convolution in which the channels calculated by the second grouping convolution are divided into groups in the channel direction, convolved, and combined to obtain an output.

5. The neural network system according to claim 1, further comprising an output subnetwork that processes the outputs of the separable convolutional subnetwork to generate a network output.

6. The neural network system according to claim 1, further comprising an input subnetwork that processes network inputs and generates an initial output.

7. The neural network system according to claim 6, characterized in that the input subnetwork has one or more convolutional neural network layers.

8. The separable convolutional subnetwork has one or more start modules, The neural network system according to claim 6, characterized in that the start module processes the initial output to generate the output of the start module.

9. The aforementioned startup module is A first pass-through convolutional layer processes the input of the start module and generates a first pass-through output, A first stack of the separable convolutional neural network layer, comprising a first stack that processes the input of the start module and generates a first stack output, A first max pooling layer that performs max pooling on the first stack output to generate a first max pooled output, A first coupling layer that couples the first pass-through output and the first maximum pooled output to generate the output of the start module. The neural network system according to claim 8, characterized by having the following:

10. The neural network system according to claim 8, characterized in that the separable convolutional subnetwork has three start modules.

11. The neural network system according to claim 9, characterized in that the first pass-through convolutional layer is a 1x1 convolutional layer.

12. The neural network system according to claim 9, characterized in that a first rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the first stack.

13. The neural network system according to claim 9, characterized in that a first batch normalization layer is provided after one or more of the separable convolutional neural network layers in the first stack.

14. The separable convolutional subnetwork has one or more intermediate modules, The neural network system according to claim 8, wherein the intermediate module processes the output of the start module to generate the output of the intermediate module.

15. The aforementioned intermediate module is A second pass-through convolutional layer processes the input of the intermediate module to generate a second pass-through output, A second stack of the separable convolutional neural network layer, comprising a second stack that processes the input of the intermediate module to generate a second stack output, A second coupling layer that couples the second pass-through output and the second stack output to generate the output of the intermediate module. The neural network system according to claim 14, characterized by having the following:

16. The neural network system according to claim 14, characterized in that the separable convolutional subnetwork has eight intermediate modules.

17. The neural network system according to claim 15, characterized in that the second pass-through convolutional layer is a 1x1 convolutional layer.

18. The neural network system according to claim 15, characterized in that a second rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the second stack.

19. The neural network system according to claim 15, characterized in that a second batch normalization layer is provided after one or more of the separable convolutional neural network layers in the second stack.

20. The separable convolutional subnetwork has a final module, The neural network system according to claim 14, wherein the final module processes the output of the intermediate module to generate the output of the final module.

21. The aforementioned final module is A third pass-through convolutional layer processes the output of the intermediate module to generate a third pass-through output, A third stack of the separable convolutional neural network layer, the third stack processing the output of the intermediate module to generate a third stack output, A second max pooling layer performs max pooling on the third stack output to generate a second max pooled output, A third coupling layer that couples the third pass-through output and the second maximum pooled output to generate a coupled output. The neural network system according to claim 20, characterized by having the following:

22. The neural network system according to claim 21, characterized in that the third pass-through convolutional layer is a 1x1 convolutional layer.

23. The neural network system according to claim 21, characterized in that a third rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the third stack.

24. The neural network system according to claim 21, characterized in that a third batch normalization layer is provided after one or more of the separable convolutional neural network layers in the third stack.

25. The aforementioned final module is A fourth stack of the separable convolutional neural network layer, the fourth stack processing the concatenated outputs to generate a fourth stack output, An average pooling layer that performs average pooling on the fourth stack output to generate the output of the final module, The neural network system according to claim 21, further comprising the above.

26. The neural network system according to claim 25, characterized in that a fourth rectified linear unit activation layer is provided before one or more of the separable convolutional neural network layers in the fourth stack.

27. The neural network system according to claim 25, characterized in that a fourth batch normalization layer is provided after one or more of the separable convolutional neural network layers in the fourth stack.

28. The neural network system according to claim 1, characterized in that it takes an input image as input and generates a network output of the input image.

29. A processing method for a neural network system, The aforementioned neural network system has separable convolutional subnetworks, The separable convolutional subnetwork has a plurality of separable convolutional neural network layers arranged in a stack in sequence. Each of the plurality of separable convolutional neural network layers applies a first grouping convolution to the input to the separable convolutional neural network layer. The input to the first grouping convolution includes multiple channels, The first grouping convolution described above is a spatial convolution in which the channels of the input to the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to obtain the output. Each of the plurality of separable convolutional neural network layers generates a layer output by sequentially applying the first grouping convolution and the second grouping convolution separately to the input to the separable convolutional neural network layer. A method for processing a neural network system, characterized in that the second grouping convolution is a spatial convolution in which the channels calculated by the first grouping convolution are divided into groups in the channel direction, convolved, and then combined to obtain an output.

30. A program for causing a computer to function as a neural network system as described in any one of claims 1 to 28.

Images