Neural networks including group convolution layers and point-wise convolution layers

Abstract

A neural network, a method of creating a neural network, and a system for optimizing a neural network are disclosed. The neural network includes a plurality of network modules, each network module including a group convolution layer and a point-wise convolution layer. In one example method, some of the point-wise convolution layers are removed or replaced based on comparing an action score of each point-wise convolution layer to an action score threshold.

Description

Background Technology

[0001] This disclosure relates to neural networks. A neural network is a machine learning model that receives input and processes that input through one or more layers to generate an output, such as classification or decision. The output of each layer in a neural network is used as the input to the next layer. The layers between the input and output layers of a neural network are called hidden layers.

[0002] A convolutional neural network (CNN) is a neural network that includes one or more convolutional layers that perform convolutional functions. CNNs are applied in many fields, including but not limited to visual data processing such as image and video recognition or classification, sound recognition and classification, facial recognition, medical data analysis, natural language processing, user preference prediction, and time series prediction and analysis.

[0003] Convolutional Neural Networks (CNNs) with a large number of layers often offer better performance, but they place high demands on memory and processing resources. Therefore, CNNs are typically implemented on computer or server clusters with powerful graphics processing units (GPUs) or tensor processing units (TPUs) and abundant system memory. However, with the increasing prevalence of machine learning and artificial intelligence applications, there is a growing desire to implement CNNs on resource-constrained devices such as smartphones, cameras, and tablets. Summary of the Invention

[0004] Therefore, a first aspect of this disclosure provides a method for creating a neural network configured to receive input visual data and generate analysis results for the input visual data. The method includes: generating a plurality of network modules by processing resources, wherein each network module includes group convolutional layers and pointwise convolutional layers. The group convolutional layer includes a plurality of filter banks and is configured to receive a plurality of input channels, divide the plurality of input channels into a plurality of channel groups, and convolve each corresponding channel group with a corresponding filter bank, each channel group including one or more channels. The pointwise convolutional layer includes a plurality of pointwise convolutional filters and is configured to receive a plurality of channels, each pointwise convolutional filter being configured to operate on all channels received by the pointwise convolutional layer. The method further includes determining a role score for each pointwise convolutional layer by the processing resources, the role score representing the importance of the pointwise convolutional layer; and removing or replacing some of these pointwise convolutional layers by the processing resources based on a comparison of the role score of each pointwise convolutional layer with a role score threshold. In this context, "some of these pointwise convolutional layers" refers to one or more of these pointwise convolutional layers.

[0005] A second aspect of this disclosure provides a convolutional neural network (CNN) for image processing. The CNN includes multiple first-type network modules, each including a group convolutional layer and a pointwise convolutional layer; and multiple second-type network modules, each including a group convolutional layer and a cyclic channel shifting unit or channel shuffling unit for mixing channels. Each group convolutional layer includes multiple filter banks, wherein each filter bank includes one or more filters. The group convolutional layer is configured to receive multiple input channels, divide the multiple input channels into multiple channel groups, and convolve each corresponding channel group with a corresponding filter bank from the multiple filter banks to produce multiple output channels. Each pointwise convolutional layer includes multiple pointwise convolutional filters and is configured to receive multiple channels, each pointwise convolutional filter being configured to operate on all channels received by the pointwise convolutional layer.

[0006] A third aspect of this disclosure provides an optimization system for a neural network, comprising a layer grouping unit and a layer customization unit. The layer grouping unit is configured to convert an initial model of the neural network into a modified model comprising multiple group convolutional layers and multiple pointwise convolutional layers; wherein each group convolutional layer comprises multiple filter banks and each filter bank is configured to operate on a corresponding channel group of input channels to produce a corresponding convolutional output, each filter bank comprising two or more filters and each channel group comprising two or more input channels. The modified model further comprises a corresponding pointwise convolutional layer associated with each group convolutional layer, the pointwise convolutional layer comprising multiple pointwise convolutional filters, each pointwise convolutional filter being configured to convolve with each of the multiple channels received by the pointwise convolutional layer. The layer customization unit is used to train the modified neural network, determine the role score of each pointwise convolutional layer based on one or more parameters of each pointwise convolutional layer, and remove or replace several pointwise convolutional layers based on a comparison between the role score of the pointwise convolutional layer and a role score threshold, wherein the role score of the pointwise convolutional layer indicates the importance of the pointwise convolutional layer.

[0007] Further features and aspects of this disclosure are provided in the appended claims and the following description. Attached Figure Description

[0008] The example will now be described by way of non-limiting example with reference to the accompanying drawings, in which:

[0009] Figure 1A An example convolutional neural network is shown;

[0010] Figure 1B An example of convolution operation is shown;

[0011] Figure 2A An example of a standard method for convolution using multiple filters is shown;

[0012] Figure 2B An example of depthwise convolution is shown;

[0013] Figure 2C An example of group convolution is shown;

[0014] Figure 3 Examples of hardware utilization of depthwise convolution and group convolution are shown;

[0015] Figure 4 An example method for creating a neural network according to this disclosure is shown;

[0016] Figure 5 Another example of creating a neural network according to this disclosure is shown;

[0017] Figure 6 Another example of creating a neural network according to this disclosure is shown;

[0018] Figure 7A An example method for generating network modules including group convolutional layers and pointwise convolutional layers is shown;

[0019] Figure 7B Another example method for generating network modules that include group convolutional layers and pointwise convolutional layers is shown;

[0020] Figure 8 An example structure of a convolutional neural network (CNN) comprising multiple network modules according to this disclosure is shown;

[0021] Figure 9 This is a graph illustrating an example of the active score of each pointwise convolutional layer in an example neural network according to this disclosure;

[0022] Figure 10A This is a schematic diagram illustrating an example of removing or replacing a pointwise convolutional layer in accordance with this disclosure;

[0023] Figure 10B This disclosure shows the results of the disclosure. Figure 10A Examples of network modules generated by the method;

[0024] Figure 10C This disclosure shows the results of the disclosure. Figure 10A Another example of a network module generated by this method;

[0025] Figure 10D This disclosure shows the results of the disclosure. Figure 10A Another example of a network module generated by this method;

[0026] Figure 11 An example of a channel cyclic shifting process according to this disclosure is shown;

[0027] Figure 12 An example structure of a neural network according to this disclosure is shown;

[0028] Figure 13 An example stage in the development of an example neural network according to this disclosure is shown;

[0029] Figure 14 This is a graph illustrating examples of the action scores and action score thresholds of multiple pointwise convolutional layers according to this disclosure;

[0030] Figure 15 This is a flowchart illustrating example methods for creating multiple versions of neural networks according to this disclosure;

[0031] Figure 16 This is a table comparing the number of pointwise convolutional layers, accuracy, and total number of parameters of various example neural networks according to this disclosure;

[0032] Figure 17 This is a schematic diagram illustrating an example optimization system for neural networks according to this disclosure;

[0033] Figure 18 This is a schematic example illustrating a computational system that can be used to create and / or optimize neural networks according to this disclosure. Detailed Implementation

[0034] For simplicity and illustrative purposes, this disclosure is described primarily by reference to examples thereof. As used herein, the term "includes" means including but not limited to, and the term "including" means including but not limited to. The term "comprises" means including but not limited to, and the term "comprising" means including but not limited to. The term "based on" means at least partially based on. The term "number" means any natural number equal to or greater than one. The terms "a" and "an" are intended to indicate at least one of a particular element.

[0035] Figure 1A An example of a convolutional neural network (CNN) 100 used for image analysis is shown. A feature map 1 representing the image to be analyzed is input to the CNN. The CNN processes the input feature map 1 through multiple layers and outputs an analysis result 180. For example, the analysis result can be a classification; in the example shown, this classification is one of several selectable image categories, such as car, truck, van, etc. In other examples, the analysis result can be used for other purposes, such as object detection, object recognition (including but not limited to face recognition), etc.

[0036] exist Figure 1A In the examples, the input feature map represents visual data, such as a portion of an image or video stream. The feature map includes the value of each of multiple elements and, in some examples, can be represented as a matrix. A CNN can have multiple output nodes. The output of a CNN can be an analysis result, such as a classification corresponding to one of the nodes (e.g., trucks) or a predetermined probability for each of the output nodes (e.g., 95% cars, 3% vans, 2% trucks). For example, the output analysis result could be a classification or decision based on the input feature map.

[0037] The CNN layers between input 1 and output 180 may be invisible to the user and are therefore called hidden layers. Each layer of the CNN receives a feature map from the previous layer and processes the received feature map to produce a feature map as output to the next layer. Thus, the first feature map 1 is input to CNN 100 and processed by the first layer 110 of the CNN to produce a second feature map, which is then input to the second layer 120 of the CNN. The second layer 120 processes the second feature map to produce a third feature map, which is then input to the third layer 130 of the CNN, and so on. CNNs typically consist of multiple convolutional layers, multiple downsampling layers, and one or more fully connected layers.

[0038] exist Figure 1A In the example, layers 110, 130, and 150 are convolutional layers. A convolutional layer is a layer that applies a convolution function to the input feature map. Figure 1B An example of a convolution operation is shown, where the input features Figure 1B Convolution with the filter (sometimes also called the kernel) 110B. Convolution may involve moving the filter over the input feature map and, at each step, calculating the dot product of the filter and the input feature map to produce the value of the output feature map 111B. Therefore, in Figure 1B In the example, the 3×3 filter 110B and the input features Figure 1B The shaded 3×3 regions are multiplied, and the result "15" forms the top left cell of the output feature map 111B. The filter is then shifted to the right (as shown). Figure 1B (As shown at the bottom) and another dot product is taken, this time resulting in a value of "16" in the top right cell of the output feature map 111B. This process continues until the filter moves through each cell of the input feature map and the output feature map is complete. Convolution enables CNNs to recognize features. Since CNNs have many layers, earlier convolutional layers can recognize basic features, such as edges, while later layers can recognize more abstract features, such as the shape or components of an object.

[0039] exist Figure 1AIn the example, layers 120 and 140 are downsampling layers. Downsampling layers reduce the dimensionality of the input feature map. Traditional neural networks perform downsampling using average pooling or max pooling. Downsampling layers reduce the dimensionality of the feature map passed to the next layer while preserving the most important information, keeping the number of nodes in the CNN within a manageable range.

[0040] Traditional CNNs use very large amounts of memory to store feature maps and weights (values) of various convolutional filters and powerful processors to compute the various convolutions. This makes implementing CNNs on resource-constrained devices with limited memory and less powerful processors, especially when the CNN has many layers. Resource-constrained devices can implement CNNs on hardware logic chips, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), but this is challenging because such logic chips may have limited memory and processing power. Therefore, various convolutional methods have been developed to reduce the required memory and / or processing power, such as... Figures 2A to 2C As shown schematically.

[0041] Figure 2A The structure of a standard convolutional layer 200 is shown, where the input feature map 210 is convolved with a standard convolutional filter 220 to produce an output feature map 230. Each standard convolutional filter operates on all input channels, so the depth of the standard convolutional filter is equal to the depth of the input feature map. Furthermore, the depth of the output feature map is equal to the number of standard convolutional filters, since each standard convolutional filter generates channels of the output feature map.

[0042] exist Figure 2A In the example, the width and height of the input feature map 210 are the same as the output feature map. Figure 2A In the example, the input feature map 210 has a width of DF, a height of DF, and a depth of M, meaning it has M channels. There are N standard convolutional filters 220, each with a width of Dk, a height of Dk, and a depth of M. That is, each standard convolutional filter operates on all input channels, and therefore its depth is equal to the number of channels in the input feature map. The output feature map 230 has a width of DF, a height of DF, and a depth of N.

[0043] Figure 2BThe structure of a depthwise separable convolution is shown. A depthwise separable convolution is logically equivalent to a standard convolution, but the operation is divided into two separate layers: a depthwise convolution and a pointwise convolution. In the depthwise convolution, the input feature map 210 is convolved with a depthwise separable filter 222 to produce an intermediate feature map 224. Each depthwise separable filter has a height of Dk, a width of Dk, and a depth of one channel (i.e., each depthwise separable filter includes a single kernel of dimension Dk×Dk). Therefore, each depthwise separable filter operates on only one channel of the input feature map 210. Since there are M channels in the input feature map 210, there are M depthwise separable filters and the intermediate feature map 224 has a depth of M. After the depthwise convolution, a pointwise convolution is performed, where the intermediate feature map 224 is convolved with a pointwise filter 226 (also referred to as a "pointwise convolution filter" in different embodiments, or pointwise layer 226, pointwise convolution layer 226) to generate an output feature map 230. Each pointwise filter is configured to operate on all channels of the output of the deep convolutional layer (i.e., all channels of the intermediate feature map 224). Each pointwise filter has a height and width of 1 (i.e., one point, hence the term "pointwise"), and its depth is equal to the number of channels M in the intermediate feature map 224 output by the deep convolution. Therefore, each pointwise filter comprises M kernels of dimension 1×1. There are N pointwise filters 226, and the height of the output feature map is DF, the width is DF, and the depth N is equal to the number of pointwise filters.

[0044] Compared to standard convolution, depthwise separable convolution reduces computational complexity and the number of required parameters. However, separable depthwise convolution results in inefficient use of hardware resources.

[0045] Figure 2CThe structure of a separable grouped convolution is illustrated. A separable grouped convolution (also called a “grouped convolution”) is logically equivalent to a standard convolution, but the operation is divided into two separate layers: a group convolution layer and a pointwise convolution layer. The group convolution layer includes multiple sets of filters 223 (in different embodiments, filters 223 are also referred to as group convolution layer 223, filter bank 223), hereinafter referred to as a “filter bank”. Each filter bank includes one or more filters. In some examples, at least one filter bank includes more than one filter. In some examples, each filter bank includes two or more filters. The group convolution layer is configured to receive multiple input channels and divide those multiple input channels into multiple channel groups. Each channel group has at least one channel. In some examples, at least one channel group has more than one channel. In some examples, each channel group has at least two channels. The group convolution layer is configured to convolve each corresponding channel group with the corresponding filter bank from the multiple filter banks. Therefore, each input channel is convolved only with a subset of filters in the group convolution layer (i.e., filters from a specific filter bank). Meanwhile, the pointwise layer includes multiple pointwise filters (also known as "pointwise convolutional filters"), each of which is configured to operate on all channels of the group convolutional layer output.

[0046] In the example shown, the input feature map 210 has a height of H, a width of W, and a depth of C1 channels. The C1 input channels are divided into g channel groups (in the example shown, the number of channel groups g equals 3). The number of filter groups can be equal to the number of channel groups (i.e., in the example shown, there are 3 filter groups and 3 channel groups). Each filter group is convolved with a single channel group. That is, each filter in the corresponding filter group is convolved with a channel in the corresponding channel group. Thus, for example, the first channel group is convolved with the first filter group, the second channel group is convolved with the second filter group, and so on. The group convolutions produce an intermediate feature map 224, the depth of which C2 equals the total number of filters in the multiple filter groups.

[0047] In the example shown, each channel group has the same number of channels, C1 / g, but in other examples, different channel groups may have different numbers of channels. Similarly, in the example shown, each filter group has the same number of filters, C2 / g, but in other examples, different filter groups may have different numbers of filters. When each channel group includes C1 / g channels and each filter group includes C2 / g filters, each channel group with C1 / g channels is convolved with the corresponding filter group with C2 / g channels.

[0048] In some examples, the depth (i.e., the number of channels) of the input feature map 210 can be equal to the total number of filters in the plurality of filter banks 223, i.e., C1 can be equal to C2. In this case, the number of filters in each filter bank is equal to the number of channels in the channel bank convolved with it. However, in other examples, the total number of filters in the plurality of filter banks can be greater than or less than the depth of the input feature map.

[0049] Figure 2C The pointwise filter in the image is similar to Figure 2B The pointwise filters in the intermediate feature map 224 are processed, and all channels of the intermediate feature map 224 are operated on. That is, channel C2 of the intermediate feature map 224 is input to the pointwise layer 226, and each pointwise filter in the pointwise layer is convolved with all channels C2 of the intermediate feature map to produce the output feature map 230. The depth C3 of the output feature map is equal to the number of pointwise filters 226.

[0050] Although Figure 2C In the example shown, the group convolutional layer 223 is followed by the pointwise convolutional layer 226. However, in other examples, the order can be reversed, i.e., the pointwise convolutional layer is followed by the group convolutional layer. In this case, the channels of the input feature map 210 are fed into the pointwise layer, and each pointwise layer filter is convolved with all input channels to produce an intermediate feature map. The channels of the intermediate feature map are then fed into the group convolutional layer and divided into multiple channel groups, and each channel group is convolved with the corresponding filter group to generate an output feature map.

[0051] Compared to standard convolution, separable grouped convolution (hereinafter referred to as "grouped convolution") reduces computational complexity and the number of required parameters. This is because in standard convolution, each input channel is convolved with all filters in the standard convolutional layer, while in the grouped convolutional layer, each input channel belongs to a channel group and each channel group is convolved only with a subset of filters in the grouped convolutional layer (filters belonging to a specific filter group).

[0052] The difference between separable depthwise convolution and grouped convolution is that in depthwise convolution, each filter is convolved with only one input channel. In grouped convolution, the input channels are divided into channel groups, and each channel group is convolved with a corresponding group of filters. Therefore, if at least one channel group includes more than one input channel, then at least some filters will be convolved with more than one input channel. Thus, grouped convolution utilizes hardware resources more efficiently than separable depthwise convolution.

[0053] Figure 3 This explains from a hardware perspective how depthwise convolutions are inefficient. Because CNN hardware is designed for... Figure 2AThe standard convolutional design results in a lot of zero padding when the same hardware is used for depthwise convolution. This is because the input feature map 210 will have multiple channels and the output feature map 230 will have multiple channels. Therefore, on the hardware designed for standard convolution, the convolutional logic 222 must have memory capacity to store filters with a depth equal to the number of input channels, and computational capacity sufficient to convolve each input channel with all filters. However, when the same hardware is used for depthwise convolution, each filter has a depth of only one, and the rest of the memory is filled with zeros. Furthermore, each input channel is convolved with only one filter. Therefore, in depthwise convolution, the memory and computational resources used for hardware implementation are not utilized. In contrast, when the same hardware is used for grouped convolution, more of the hardware logic's memory and computational resources are utilized because each group of channels is convolved with the corresponding group of filters. For example, in the example logic of 223, the depth of each filter is C. g And with C g Convolution is performed on each channel, thus utilizing a larger proportion of the available memory and computing resources of the hardware logic.

[0054] Grouped convolution can therefore be seen as a compromise, offering reasonable hardware efficiency in utilizing available hardware resources while reducing the number of parameters and computational complexity compared to standard convolution. Thus, grouped convolution enables more complex neural networks to be implemented on resource-constrained hardware while still efficiently utilizing available resources. However, grouped convolution still uses pointwise convolutional layers, which are computationally complex and use a relatively large number of parameters, thus significantly impacting processing and memory requirements. Therefore, a large portion of the computational and memory requirements of grouped convolution comes from the pointwise convolutional layers.

[0055] Therefore, this disclosure proposes removing or replacing some pointwise convolutional layers to reduce memory and / or computational requirements. In this context, "some pointwise convolutional layers" refers to one or more pointwise convolutional layers. In some examples, multiple pointwise convolutional layers may be replaced or removed. In some examples, a large number of pointwise convolutional layers may be replaced or removed (e.g., more than 10% of the pointwise convolutional layers in a neural network). Specifically, this disclosure proposes determining a role score for each pointwise convolutional layer, where the role score represents the importance of the pointwise convolutional layer; and removing or replacing pointwise convolutional layers based on the role score (e.g., if the role score is below a threshold). In this way, pointwise convolutional layers that do not significantly contribute to the accuracy of the neural network can be removed or replaced with layers of lower computational cost.

[0056] Figure 4An example method is shown for creating a neural network configured to receive an input image and generate analysis results for that input image. This method can be implemented using processing resources, such as one or more processors. Processing resources can include, for example, one or more microprocessors, central processing units, graphics processing units, tensor processing units, or hardware logic chips (such as FPGAs or ASICs).

[0057] At box 410, multiple network modules are generated by processing resources. Each network module includes group convolutional layers and pointwise convolutional layers.

[0058] Group convolutional layers and pointwise convolutional layers can be similar to Figure 2C The layers discussed in the text. Each group convolutional layer includes multiple filter banks. The group convolutional layer is configured to receive multiple input channels, divide those input channels into multiple channel groups, and convolve each corresponding channel group with a corresponding filter bank from those multiple filter banks. Each pointwise layer includes multiple pointwise convolutional filters and is configured to receive multiple channels. Each pointwise convolutional filter is configured to operate on all channels received by the pointwise convolutional layer.

[0059] In some examples, pointwise convolutional layers can follow group convolutional layers. In this case, each pointwise convolutional filter is configured to operate on all channels of the group convolutional layer's output. For example, the group convolutional layer can operate on all channels input to the network module, and the pointwise convolutional layer can take the output of the group convolutional layer as input and operate on all channels of the group convolutional layer's output. In this case, the pointwise convolutional layer blends the channels of the group convolutional layer's output.

[0060] In other examples, pointwise convolutional layers can precede group convolutional layers. In this case, each pointwise filter is configured to operate on all channels input to the network module. The group convolutional layer can then take the output of the pointwise convolutional layer as input and operate on all channels of the pointwise convolutional layer's output. In this case, the pointwise convolutional layer mixes the channels input to the group convolutional layer.

[0061] At box 420, the processing resources determine the role score for each pointwise convolutional layer. The role score represents the importance of the pointwise convolutional layer.

[0062] At box 430, resource processing is based on removing or replacing some pointwise convolutional layers by comparing the action score of each pointwise convolutional layer with an action score threshold.

[0063] In this way, CNNs have reduced memory requirements and a less computationally complex architecture, yet can be implemented in hardware efficiently and achieve reasonable performance and accuracy.

[0064] Figure 5It shows that it can be done Figure 4 The method generates and modifies an example of neural network structure 500 to produce a modified neural network structure 520. Therefore, Figure 5 The neural network structure on the left is 500. Figure 4 The method box 410 can generate an example of a possible network structure. Figure 5 Arrow 515 in the middle indicates Figure 4 The modification process in boxes 420 and 430, and Figure 5 The network structure 520 shown on the right illustrates a possible modified network structure, in which one or more pointwise layers have been based on Figure 4 The method was removed or replaced.

[0065] Therefore, the neural network structure 500 includes multiple network modules, each including group convolutional layers and pointwise convolutional layers. More specifically, in the example shown, the initial neural network includes a convolutional layer 502 with a stride of 2, a first network module 504 including a group convolutional layer 504A with a stride of 1 and a pointwise convolutional layer 504B with a stride of 1, a second network module 506 including a group convolutional layer 506A with a stride of 2 and a pointwise convolutional layer 506B with a stride of 1, a third network module 508 including a group convolutional layer 508A with a stride of 1 and a pointwise convolutional layer 508B with a stride of 1, followed by an average pooling layer 510 and a fully connected layer 512.

[0066] exist Figure 5 In the modified network architecture 520 shown on the right, one or more pointwise layers have been removed or replaced based on a comparison of the effectiveness score of pointwise convolutional layers (also called “pointwise layers”) with a threshold. In one example, if the effectiveness score of a pointwise layer is below the threshold, this can indicate that the pointwise layer is not very important to the accuracy of the neural network and therefore the pointwise layer is removed or replaced. Figure 5 In the example, network module 508 corresponds to network module 528 on the right, but the pointwise layer 508B of network module 508 has been replaced with a cyclic channel shift (CCS). Therefore, network module 528 corresponds to network module 508, but has been simplified and has lower processing and memory requirements. It has been found that removing or replacing pointwise layers with action fractions below a threshold can reduce the computational and memory resources required by neural networks while still maintaining high accuracy.

[0067] It should be understood that Figure 5 This is merely an example, and other implementations may have different numbers of network modules, different strides and / or different layer arrangements and may include further additional layers, such as pooling layers, standard convolutional layers or fully connected layers, etc.

[0068] Figure 6An example method for creating a CNN is shown, which includes... Figure 4 The method and some additional processes.

[0069] In box 610, an initial model of the CNN is provided. This initial model can be designed by an engineer or taken from an existing CNN design. For example, the initial model could be as follows: Figure 2A The model shown uses standard convolutional layers or, as... Figure 2B The model shown uses depthwise separable convolutional layers.

[0070] At box 620, the initial CNN model is modified using processing resources to generate a model with group convolutional layers and pointwise convolutional layers, for example... Figure 2C The type shown. Therefore, box 620 is Figure 4 One possible implementation of the method is block 410, because block 620 generates a neural network model comprising multiple network modules, each network module including group convolutional layers and pointwise convolutional layers.

[0071] At box 630, the processing resource implements and trains the modified neural network model generated by box 620. For example, the processing resource may implement the modified neural network model as an actual modified neural network and then train it. Training the modified neural network may include passing a set of test data (such as, but not limited to, CIFAR-10) through the modified neural network and modifying the filter weights of the neural network to improve the classification or other analysis results of the modified neural network. In some examples, training may include backpropagation of the filter weights and / or minimizing a cost function. Training the modified neural network model may generate a set of filter weights capable of analyzing (e.g., classifying) input images with a desired level of accuracy.

[0072] At box 640, the processing resource customizes the modified neural network model by replacing or removing one or more pointwise convolutional layers. The processing resource determines which pointwise convolutional layers to remove or replace based on a comparison of the activation score of each pointwise convolutional layer with an activation score threshold. Box 640 may correspond to... Figure 4 Boxes 420 and 430.

[0073] At box 650, a custom model of the neural network is trained. The training of this custom model can be performed in the manner described above for training the modified neural network at box 630.

[0074] At box 660, after the custom model has been trained, it is ready to be used. For example, it can be implemented in the cloud, on a server or computer, or transferred to hardware chips (such as ASICs, FPGAs) or other hardware.

[0075] In some examples, blocks 610 to 650 related to the design and training of the neural network can be implemented in the cloud, on servers or computers with abundant memory and processing resources; the model of the neural network 660 can then be implemented on resource-constrained platforms, such as smartphones, cameras, tablets, edge devices, hardware logic chips (such as ASICs, FPGAs), etc. In some examples, the design and training can be implemented by processor-executed software, while the final model can be implemented on a hardware chip.

[0076] It should be noted that, although Figure 6 An example is shown starting with an initial neural network model that has been modified to use group convolutional layers and pointwise convolutional layers. However, in other examples, the neural network model can be designed from the outset to have both group and pointwise convolutional layers; in this case, the method starts with the model in box 620. In still other examples, if a model with both group and pointwise convolutional layers has already been trained, the method can start with box 640.

[0077] Figure 7A An example method for generating network modules including group convolutional layers and pointwise convolutional layers is shown. This method can begin with a standard convolutional layer 700 or a separable depthwise convolutional layer 710.

[0078] In the case of a standard convolutional layer of 700, this layer can be described above regarding... Figure 2A The operation is described in the following manner. A standard convolutional layer operates on the input feature map 702 to generate an output feature map 704. The input feature map 702 includes multiple channels, and the output feature map 704 includes multiple output channels; in some examples, the number of input channels and the number of output channels can be the same. Figure 7A In the example, there are eight input channels and eight output channels, but in other examples, there may be a different number of channels. As indicated by the arrows between the channels of the input feature map 702 and the channels of the output feature map 704, each output channel includes information from each of the input channels. This is because each standard convolutional filter operates on all input channels, and each standard convolutional filter generates a corresponding output channel. Standard convolution has been found to produce high accuracy in many cases, but it requires significant memory and processing resources because each filter operates on every input channel.

[0079] In the case of a separable depthwise convolutional layer 710, this layer can be described above regarding... Figure 2BThe operation is described in the following manner. Layer 710 includes a deep convolutional layer 712 and a pointwise convolutional layer 714. The deep convolutional layer operates on the input feature map 712A to generate an intermediate feature map 712B. The input feature map 712A includes multiple channels and the output feature map 712B includes multiple output channels; in some examples, the number of input channels and the number of output channels can be the same. In the example of Figure 7, there are eight input channels and eight output channels, but in other examples, there can be different numbers of channels.

[0080] Compared to the standard convolutional layer 700, in the depthwise convolutional layer 712, each channel of the output feature map 712B is based on information from only one channel of the input feature map 712A. Therefore, the arrow between the input feature map 712A and the output feature map 712B indicates that each output channel is based on only one input channel. This is because in depthwise convolution, each depth filter operates on only one input channel to generate the corresponding output channel.

[0081] The output from the deep convolutional layer 712 is input to the pointwise convolutional layer 714. Therefore, the feature map 712B output from the deep convolutional layer 712 can serve as the input feature map 714A of the pointwise convolutional layer 714. Thus, feature maps 712B or 714A can be referred to as intermediate feature maps. In the pointwise convolutional layer 714, as indicated by the arrows in the figure, each input channel of feature map 714A contributes to the output feature map 714B. This is because each pointwise convolutional filter operates on each of the input channels to generate the corresponding output channel. Therefore, while the deep convolutional layer 712 does not mix information between channels, the pointwise convolutional layer 714 mixes information between channels. Although the mixing is less than that of standard convolution, this method has been found to produce results with a reasonable level of accuracy while requiring significantly less memory and processing resources than standard convolution.

[0082] Figure 7A The right side (generally indicated by reference numeral 720) shows a network module equivalent to a standard convolutional layer 700 or a separable deep convolutional layer 710. However, network module 720 includes group convolutional layers 722 and pointwise convolutional layers 724. The group convolutional layer 722 includes multiple filter banks that operate on the channels of the input feature map 722A to generate the output feature map 722B of the group convolutional layer.

[0083] The input feature map 722 includes multiple channels, and these channels are divided into multiple channel groups. Each channel group includes one or more channels. At least one of these channel groups may include more than one channel. In some examples, each channel group includes two or more channels. In some examples, for each group convolutional layer, all channel groups in the group convolutional layer have the same number of channels. Figure 7AIn the example, there are two channel groups, and each channel group includes four channels.

[0084] Each channel group is convolved with a corresponding filter group. Each filter group includes one or more filters, and each filter generates a corresponding channel of the output feature map. In some examples, each filter group includes two or more filters. In some examples, the filter groups can be discrete, such that each filter belongs to only one filter group. In other examples, the filter groups can overlap, such that some filters belong to more than one filter group. In some examples, each channel group is convolved with different filter groups, while in other examples, some or all channel groups are convolved with the same filter group. In some examples, each filter group can operate on all channels of its corresponding channel group, but not on channels in other channel groups.

[0085] exist Figure 7A In the example, the group convolutional layer comprises two non-overlapping filter banks, and each filter bank contains four filters. Each filter bank operates on all channels in one of the channel banks, but not on channels in the other channel banks. Therefore, the output channels of feature map 722B are based on information from multiple input channels 722A within the same channel bank, but not on information from other input channel banks. Figure 7A In the example, the first filter bank has four filters, which are applied to the input channels of the first channel group and generate the four output channels of the first channel group. Similarly, the second filter bank has four filters, which are applied to the input channels of the second channel group and generate the four output channels of the second channel group.

[0086] As explained above, each filter in a filter bank is convolved with all the input channels of the corresponding channel group with which the filter bank is convolved. Therefore, each filter can have the same number of kernels as the number of input channels in its corresponding channel group. The number of output channels in the filter bank is equal to the number of filters in the filter bank. A filter bank can have the same number of input and output channels; in this case, the number of filters in the filter bank will be equal to the number of input channels in its corresponding channel group.

[0087] Therefore, the number of channels in the first channel group can be equal to the number of filters in the first filter bank, the number of channels in the second channel group can be equal to the number of filters in the second filter bank, and so on. Each filter in the filter bank can have multiple kernels, allowing it to convolve with multiple channels of the corresponding channel group. Each filter in the filter bank can have the same number of kernels as the number of input channels in the corresponding channel group.

[0088] The output of the group convolutional layer 722 is fed into the pointwise convolutional layer 724. Therefore, the feature map output 722B can serve as the input feature map 724A of the pointwise convolutional layer 724. The pointwise convolutional layer operates in the same manner as the aforementioned pointwise convolutional layer 714.

[0089] It can be seen that, regarding channel mixing, if network module 720 has at least one channel group with more than one channel and at least one filter group with more than one filter, then this falls between the standard convolution method 700 (which has a large amount of mixing) and the separable depthwise convolution method 710 (which has less mixing). It has been found that the grouped convolution of network module 720 can provide a good level of accuracy in at least some applications. However, grouped convolution requires significantly less memory and processing resources than standard convolution because each filter in the grouped convolutional layer does not have to be applied to every input channel. Furthermore, compared to separable depthwise convolution, accuracy can be better due to the greater mixing, and the process utilizes hardware resources more efficiently because there are fewer empty cells (as referenced above). Figure 3 (As explained).

[0090] In some examples, the method for forming a neural network according to this disclosure can begin with an existing or previously designed neural network structure (or model) including standard convolutional layers 700. In this case, the method can modify these structures to replace some or all of the standard convolutional layers 700 with corresponding network modules 720 including group convolutional layers and pointwise convolutional layers. In other examples, the method can begin with an existing or previously designed neural network structure including separable depthwise convolutional layers 710. In this case, the method can modify these structures to replace some or all of the separable depthwise convolutional layers 710 with corresponding network modules 720 including group convolutional layers and pointwise convolutional layers. In such cases, the neural network including network module 720 can be referred to as a modified neural network; layers 700 or 710 correspond to... Figure 6 The initial model layer in box 610, and the arrow pointing to 720 corresponds to... Figure 6 Modification box 620. In other examples, the neural network can be designed from the outset as a grouped convolutional network with network module 720, rather than transforming a previous network model.

[0091] Figure 7B Similar to Figure 7A And the same reference numerals denote the same parts. However, although Figure 7A An example of a network module is shown, where a group convolutional layer 722 is followed by a pointwise convolutional layer 724, but... Figure 7B In the example, the order is reversed and the pointwise convolutional layer 724 comes first, followed by the group convolutional layer 722.

[0092] Figure 8 An example architecture of a convolutional neural network (CNN) comprising multiple network modules is shown. That is, each network module includes group convolutional layers and pointwise convolutional layers, as shown, for example, at 720 in Figure 7. While Figure 7 shows a single network module, convolutional neural networks include multiple such network modules and may also include additional layers in some cases. Figure 8 In the example, the CNN includes a standard convolutional layer 801, multiple network modules 802 to 813, a pooling layer 814, a fully connected layer 815, and a softmax layer 816. The softmax layer is a layer that performs the softmax function, used to normalize the network's output to a probability distribution over the output class. Each network module includes group convolutional layers and pointwise convolutional layers. While these diagrams show network modules where the group convolutional layers precede the pointwise convolutional layers, in other examples, the order may be reversed, i.e., the pointwise convolutional layers precede the group convolutional layers. Figure 8 In other similar figures disclosed herein, the symbol “conv” indicates a standard convolutional layer, while the symbol “G.conv” indicates a group convolutional layer and the symbol “conv_pw” indicates a pointwise convolutional layer. Further, the symbol / s2 indicates a stride of 2, and the symbol s1 indicates a stride of 1. Figure 8 The dimensions on the right side of the middle frame 801 to 816 indicate the dimension of the feature map output for each layer in terms of height × width × depth, where the depth is the number of output channels.

[0093] Network modules 807 to 811 are illustrated once and accompanied by the symbol ×5 because there are five consecutive network modules with the same structure (i.e., in this case, a group convolution with a stride of 1 and an output of 2×2×512 and a pointwise convolution with a stride of 1 and an output of 2×2×512). Although the structure of network modules 807 to 811 is the same, the actual values ​​of the filters in the constituent layers may differ for each network module.

[0094] Understandable. Figure 8 This is merely an example, and other implementations may have different numbers of layers, different strides and / or output sizes, and different layer arrangements or orders. The general principle is to have multiple network modules, each comprising group convolutional layers and pointwise convolutional layers.

[0095] The group convolutional layers and pointwise convolutional layers in each network module may include additional features or procedures, such as batch normalization and corrected linear activation (ReLU). Figure 8A more detailed breakdown of one of the network modules 802 is shown, wherein the group convolutional layer 802A includes a filter bank having 3×3 group convolutional layers 802A-1, followed by a batch normalization process 802A-2 and a modified linear activation process 802A-3. Similarly, the pointwise convolutional layer 802B includes a 1×1 pointwise convolutional layer 802B-1, followed by a batch normalization process 802B-2 and a modified linear activation process 802B-3.

[0096] It should be understood that in the CNN according to this disclosure, within each network module, each group convolutional layer may include a batch normalization process and a modified linear activation function, and each pointwise convolutional layer may include a batch normalization process and a modified linear activation function. In some examples, group convolutional layers and / or pointwise convolutional layers may include a batch normalization process without a modified linear activation function. Further, although Figure 8 The example shows group convolution preceding pointwise convolution, but in other examples, the order can be reversed.

[0097] Figure 8 The CNN shown corresponds to Figure 4 Box 410 and Figure 6 Boxes 620 and 630. See above for reference. Figure 4 Boxes 420 and 430 and Figure 6 Box 640 allows for the customization of the neural network by removing or replacing one or more pointwise convolutional layers based on their effectiveness scores. This is because, referring to Figure 7, it will be noted that the most computationally expensive part of the network module is the pointwise convolutional layer 724. Figure 8 As shown, a CNN designed to implement separable group convolutions can include a large number of pointwise convolutional layers. If some of these layers can be removed or replaced without significantly reducing the accuracy of the neural network, the required processing and memory resources can be reduced, thus enabling CNNs to be implemented on a wider range of end devices, especially those with limited resources.

[0098] It has been found that by determining the role score of each pointwise layer and removing or replacing pointwise layers based on a comparison with a threshold, CNNs can be simplified and processing and memory resources saved while maintaining a relatively high level of accuracy. The role score can represent the importance of a layer in contributing to the CNN's analytical results (e.g., classification). Therefore, for example, if a high role score indicates high importance, pointwise convolutional layers with role scores below a threshold can be removed or replaced.

[0099] Figure 9This is a graph illustrating an example of the role score for each pointwise convolutional layer in a CNN comprising thirteen pointwise convolutional layers (layers 0 to 12). In this example, the role score is set between 0 and 1, and the threshold τ is set to 0.4. Therefore, layers 7 to 11, which have role scores below the threshold, are removed or replaced.

[0100] It should be understood that Figure 9 This is just an example using a threshold. In other examples, there could be more or fewer layers, different ranges of action scores, and / or the threshold could be set at different levels. Furthermore, although in Figure 9 In the example, multiple adjacent pointwise convolutional layers (layers 7 to 11) are below the threshold, but in other examples, pointwise convolutional layers below the threshold do not need to be consecutive. Furthermore, although in Figure 9 In this context, a low action score indicates that a pointwise convolutional layer has low importance and can be removed or replaced. However, in other examples, the action score can be calculated differently, such that a low action score indicates high importance. In this case, pointwise convolutional layers with action scores above a threshold can be removed or replaced.

[0101] In some examples, the processing resources determine the contribution score of a pointwise convolutional layer based on one or more parameters of each pointwise layer. These parameters can be based, for example, on filter weights or other parameters established during CNN training. In some examples, the contribution score of a pointwise convolutional layer is based on a statistical measure of the weight distribution within that layer. A larger distribution of weight values ​​compared to a pointwise convolutional layer where weight values ​​are tightly clustered can indicate that the pointwise convolutional layer contributes relatively more significantly to the accuracy of the neural network.

[0102] In some examples, the impact score of each pointwise convolutional layer is determined based on the scaling parameters of the batch normalization process following the convolution of that pointwise convolutional layer. Batch normalization recenters and rescales the output of the pointwise convolutional layer in order to normalize the input to the next layer.

[0103] The scaling parameter of batch normalization provides a statistical measure of the weight distribution in the previous layer and can therefore be used to generate action scores. Using the batch normalization scaling parameter is a convenient approach because batch normalization is frequently used in CNNs, and therefore in many cases, existing batch normalization processes can be utilized to provide action scores without adding complex additional processes. For example, batch normalization can be performed as follows: Figure 8 As shown, this is performed after pointwise convolution, and the effective score of the pointwise convolutional layer can be generated using batch-normalized scaling parameters.

[0104] In some examples, the role score for each pointwise layer is determined based on the scaling parameters of the batch normalization process and the number of pointwise filters in that pointwise convolutional layer. This may allow the role score to account for the contribution of the pointwise layer's filters to the CNN's analytical results. For example, if the statistical distributions of the weights of each pointwise filter in a layer are summed, the total can be divided by the number of filters to give an indication of the layer's relative importance. For instance, if a layer has a large number of pointwise filters, but most of them have null values, then the layer's importance and therefore its role score may be low.

[0105] In one example, the role score α1 of the pointwise layer is calculated based on the weight scaling parameters of layer l determined by training the neural network (refer to other diagrams). In one example, the role score α1 is calculated as follows:

[0106]

[0107] Where l is the layer number, F1 is the number of filters in layer l, and γ i It is the scaling parameter of the i-th filter.

[0108] The action fraction can be normalized to a predetermined range. In the example of Equation 1, the denominator uses the same equation as the numerator, but the number of filters in the layer is set to the number of filters in the layer with the most filters, in order to normalize the action fraction to the range of 0 to 1.

[0109] The action score threshold can be set as a cutoff value when determining the importance of pointwise convolutional layers. The ratio between network modules with pointwise convolutional layers and those without varies depending on the level of the action score threshold (because the action score threshold determines whether a particular pointwise convolutional layer is removed or replaced, so for example, if the threshold is low, more pointwise convolutional layers can be removed or replaced).

[0110] In some examples, the effective score threshold is based on the size of the channel group in the group convolutional layer associated with the pointwise convolutional layer. If the group convolutional layer is in the same network module, then the group convolutional layer is associated with the pointwise convolutional layer (e.g., Figure 8 In this context, group convolutional layers 802A-1 are associated with pointwise convolutional layers 802B-1. For example, group convolutional layers can be associated with subsequent pointwise convolutional layers (such as in...). Figure 7A and Figure 8 (in the middle), or it can be associated with the preceding pointwise convolutional layer (such as in the middle). Figure 7B middle).

[0111] In some examples, the size of a channel group can be defined as the number of channels in each channel group of the corresponding convolutional layer (C0). gThis approach is effective when each channel group has the same number of channels. For example, in the example network module 720 of Figure 7, each channel group has four channels, and therefore the size of the channel group is four.

[0112] In other examples, when the number of channels in each channel group is different, the size of the channel group can be defined as the number of channels in the channel group with the fewest channels (C). g(min) For example, this might be applicable to situations where each channel group within a group convolutional layer has the same number of channels, but different group convolutional layers have channel groups with different numbers of channels. For example, the first network module might include pointwise convolutional layers and layers with a size of C. g The second network module may include pointwise convolutional layers and corresponding group convolutional layers with a channel group of size C = 4 (i.e., each channel group has four channels); while the second network module may include pointwise convolutional layers and convolutional layers with a channel group of size C = 4. g =3 channel groups (i.e., each channel group has three channels) corresponding to the group convolutional layers; in this case, the channel group with the fewest channels in the CNN has C g(min) = 3 channels.

[0113] As can be seen from Figure 7, the size of the channel group is related to the amount of mixing that occurs during group convolution. Therefore, in Example 720 of Figure 7, N channels are divided into groups of size C. g = 4 channel groups, and therefore there is a relatively high degree of mixing, because half of the channels in each channel group are mixed. However, if the channel group size is small, such as C g If the value is 2, there will be four channel groups, and only two channels in each channel group (1 / 4 of the total) will be mixed. The less mixing in the network nodes, the more important the pointwise convolutional layers become in order to maintain the accuracy of the CNN.

[0114] The more mixing in a group of convolutional layers, the less important the pointwise convolutional layers become. It's useful to consider extreme cases as thought experiments to illustrate how the channel group size interacts with pointwise convolution. If each channel group has only one channel C... g =1, which is equivalent to depthwise convolution, as shown in 7.12 of Figure 7, which does not involve any channel blending and therefore would typically require subsequent pointwise convolutional layers to introduce blending. However, if at least one channel group has more than one channel, there will be at least some degree of blending in the group convolutional layer. If each channel group has at least two channels, there will be even more blending. In contrast, if C = 1, then this is equivalent to depthwise convolution, as shown in Figure 7.12, which does not involve any channel blending and therefore would typically require subsequent pointwise convolutional layers to introduce blending. g = N (where N is the number of input channels), then all channels remain together, which is the standard convolution shown in Figure 700, rather than a group convolution because it is not divided into multiple groups. Because the standard convolution has a very high degree of channel blending, subsequent pointwise convolutional layers are usually not needed.

[0115] In some examples, the action score threshold is based on the size of the channel group and the number of channels operated on by the pointwise convolutional layer in the neural network that operates on the maximum number of channels (C). max The effect score threshold is determined. In these examples, for instance, the effect score threshold can be the channel group size (C). g Or C g(min) The function and C max The ratio of functions.

[0116] exist Figure 9 In the example, C g =16 and C max =1024; that is, each of the thirteen pointwise convolutional layers is associated with a group convolutional layer, where the group size is 16 channels and the layer with the most channels has 1024 channels (and therefore 1024 / 16 = 64 group convolutions). This is understandable. Figure 9 This is just an example, and other CNNs can have different numbers of layers and different CL values. g and C max value.

[0117] Based on the above principle, in one example, the threshold τ can be calculated as follows:

[0118]

[0119] For example, Equation 2 is one approach that can be used when the channel group size is the same in each network module of a CNN.

[0120] In other examples, the threshold τ can be calculated as follows:

[0121]

[0122] For example, Equation 3 is one approach that can be used when the channel group size differs between different network modules in a CNN.

[0123] So far, we have discussed how to remove or replace pointwise convolutional layers based on their role score and a role score threshold. For example, a pointwise layer can be removed or replaced in response to a comparison of its role score with a role score threshold matching a predetermined criterion. For instance, the predetermined criterion could be a role score below a threshold (if a high role score indicates high importance for the layer). In other examples, the predetermined criterion could be a role score above a threshold (if a high role score indicates low importance for the layer).

[0124] Figure 10A This is a schematic diagram illustrating an example of removing or replacing a pointwise convolutional layer. Figure 10AThe left side shows network module 1000, which includes group convolutional layers 1010 and pointwise convolutional layers 1020. The group convolutional layer includes group convolution 1012, batch normalization (BN) 1014, and modified linear activation function (ReLU) 1016. The pointwise convolutional layer 1020 includes pointwise convolution 1022, batch normalization (BN) 1024, and modified linear activation function (ReLU) 1026. The group convolutional layer 1010 and the pointwise convolutional layer 1020 can have the above examples (including but not limited to...). Figure 2C And any features described in Figure 7). Network module 1000 can be a CNN comprising multiple network modules (e.g., as described in the reference above). Figure 4 and Figure 8 (as described). Network module 1000 may, for example, correspond to... Figure 4 The network module in the CNN generated at box 410 of the method. Figure 5 One of the network modules on the left, or Figure 6 The network modules in the modified neural network generated in box 620.

[0125] Figure 10A The right side shows network module 1030 after the method disclosed herein has been applied to a CNN. For example, in the execution of... Figure 4 The method follows the network module after blocks 420 and 430. Figure 5 The network model on the right, or Figure 6 The custom group convolutional model is located in box 640. Therefore, network module 1030 can be referred to as a custom group convolutional network module.

[0126] Network module 1030 includes a group convolutional layer 1010, which is identical to the group convolutional layer 1010 of the original network module 1000. This group convolutional layer includes group convolution 1012, batch normalization 1014, and a modified linear activation function 1016. However, as shown in box 1040, the pointwise convolutional layer 1020 of the original network module can be removed or modified. As shown in box 1042, processing resources can determine whether the activation score αl of the pointwise convolutional layer matches a predetermined criterion. The predetermined criterion can be based on a comparison of the activation score with a threshold. In the example shown, the predetermined criterion is that the activation score is greater than the threshold τ. In other examples, the predetermined criterion could be that the activation score is less than the threshold τ.

[0127] If the effective fraction of the pointwise convolutional layer 1020 does not meet the predetermined criteria ( Figure 10A If the condition is "No" in the text, then the pointwise convolutional layer is preserved as shown in 1020. However, if a predetermined criterion is met ( Figure 10A If the "Yes" option is selected, then the pointwise convolutional layer will be replaced or removed as shown in box 1050.

[0128] Figure 10B It shows the result of Figure 10A An example of a network module generated by box 1050. In this case, the pointwise convolutional layer 1020 has been removed from the network module in response to the comparison of the action score of the pointwise layer with the action score threshold matching a predetermined criterion. Therefore, the resulting network module 1050B includes group convolutional layers, which in this example include group convolution 1012, batch normalization 1014, and a modified linear activation function 1016.

[0129] Figure 10C It shows the result of Figure 10A An example of the network module generated by box 1050. In this case, the pointwise convolutional layer 1020 has been replaced in response to the comparison of the action score of the pointwise layer with the action score threshold matching a predetermined criterion. Specifically, the pointwise convolutional layer 1020 has been replaced with a channel cyclic shifting process. Thus, the resulting network module 1050B includes group convolutional layers (which in this example include group convolution 1012, batch normalization 1014, and modified linear activation function 1016) and channel cyclic shifting 1050C.

[0130] Figure 10D It shows the result of Figure 10A An example of the network module generated by box 1050. In this case, the pointwise convolutional layer 1020 has been replaced in response to the comparison of the action score of the pointwise layer with the action score threshold matching a predetermined criterion. Specifically, the pointwise convolutional layer 1020 has been replaced with channel shuffle 1050D. Thus, the resulting network module 1050B includes group convolutional layers (which in this example include group convolution 1012, batch normalization 1014, and modified linear activation function 1016) and channel shuffle 1050D.

[0131] Figure 11 An example of a channel cyclic shifting process is shown. Multiple input features 1100 enter the channel cyclic shifting process. The input features 1100 are divided into multiple input channels. These input features and channels may correspond to the channels output from the preceding group convolutional layers. The output of the channel cyclic shifting process is divided into multiple output features 1120 with multiple channels. The output features 1120 may correspond to the input features, but are shifted in position.

[0132] That is, cyclic shifting maps each input channel to a different output channel. Furthermore, during cyclic shifting, channels are mapped in such a way that their positions are shifted while maintaining their relative order. Therefore, in Figure 11As can be seen, the input channels are numbered 1, 2, 3, 4, 5, 6, 7, 8 from top to bottom, while the output channels are numbered 3, 4, 5, 6, 7, 8, 1, 2. This means that each channel's position has been shifted upwards by two positions, and because the channels are processed in a cyclic manner, the first two channels have already been shifted to the bottom, with channel 1 following channel 8. In the cyclic shift, channels can shift forward or backward. For example, in another example, channels 1, 2, 3, 4, 5, 6, 7, 8 can be shifted to 7, 8, 1, 2, 3, 4, 5, 6.

[0133] In the example shown, channels are shifted in pairs and shifted two positions. However, in other examples, one channel or a larger group (such as a triplet) can be shifted at a time, and the shifts can occur in one position or three or more positions. Figure 11 In the example, the input channels are received from a group convolutional layer, which divides the channels into two channel groups, each containing C. g One channel. In this example, C g =4. Because cyclic shifting causes channels to shift in pairs, channel shifting can be called C. g A 2-channel shift occurs because the number of channels remaining together during the shift is equal to half the size of the channel group. Therefore, if the output channels are divided into channel groups of the same number and size, half of the channels in each output channel group will originate from different input channel groups. Thus, a considerable degree of mixing can be observed.

[0134] As can be understood, in the example shown, there are a total of eight channels, two channel groups, and C g =4. However, in other examples, there may be different numbers of channels, different numbers of channel groups, and / or different channel sizes.

[0135] Channel shuffling is similar to cyclic channel shifting because some or all input channels are mapped to different output channels, somewhat like shuffling cards. However, channel shuffling is not limited to cyclic shifting; any input channel can be mapped to any output channel. The mapping between input and output channels can be stored, for example, in a lookup table. Compared to channel shuffling, cyclic channel shifting is simpler to implement and uses fewer processing and memory resources because simple hardware structures (such as simple logic circuits or memory structures) are easily used to perform cyclic channel shifting.

[0136] In some examples, each network module from which a pointwise layer has been removed or replaced is configured to output the same number of channels as the number of channels input to that network module. For example, in Figure 11As can be seen, the circular channel shifter has the same number of output channels as the input channels to the circular channel shifter. Therefore, if the convolutional layer outputs the same number of channels as the input channels to the convolutional layer, the network module will output the same number of channels as the input channels to the network module.

[0137] Figure 12 An example architecture of a CNN 1200 according to this disclosure is shown. CNNs can be configured for image processing.

[0138] CNN 1200 includes multiple first-type network modules 1210 and 1230, each of which includes group convolutional layers 1210A and 1230A, followed by pointwise convolutional layers 1210B and 1230B. CNN 1200 further includes multiple second-type network modules 1220 and 1240, each of which includes group convolutional layers 1220A and 1240A, followed by channel shuffling units or cyclic channel shifting units 1220B and 1240B to mix the channels output by the group convolutional layers.

[0139] CNN 1200 can be seen from Figure 1 to... Figure 11 The network can be generated using any of the methods described above. Therefore, it can be understood that the second type of network modules 1220 and 1240 can be network modules that originally had pointwise convolutions, but for these network modules, the pointwise convolutional layers are customized during the process (e.g., ...). Figure 4 Boxes 420 and 430 Figure 6 The frame 640 and Figure 9 In the customized process shown in the figure, it is replaced by a channel shuffling unit or a cyclic channel shifting unit.

[0140] In CNN 1200, each group convolutional layer includes multiple filter banks and is configured to receive multiple input channels, divide these input channels into multiple channel groups, and convolve each corresponding channel group with a corresponding filter bank from the multiple filter banks to produce multiple output channels. Further, each pointwise convolutional layer includes multiple pointwise convolutional filters, each of which is configured to operate on all channels input to that pointwise convolutional layer. For example, if a group convolutional layer precedes a pointwise convolutional layer, each pointwise convolutional filter operates on all channels output from the preceding group convolutional layer.

[0141] The multiple first-type network modules and the multiple second-type network modules can be arranged in any order. A particular network module can be determined as first-type or second-type based on the role scores of pointwise convolutional layers present in an earlier model of the neural network prior to customization.

[0142] In some examples, each pointwise convolutional layer has a role score determined based on statistical differences between the weights in that pointwise convolutional layer, and the role score of each pointwise convolutional layer exceeds a role score threshold defined based on the number of channels in one of the channel groups.

[0143] In some examples, network modules can be arranged such that multiple first-type network modules are stacked one after another, followed by multiple second-type network modules stacked one after another, and the multiple second-type network modules are followed by at least one first-type network module. Figure 13 The rightmost example shows a network with this structure.

[0144] Figure 13 This shows the stages in the development of CNN. Figure 13 The leftmost image shows an example architecture 1300 from MobileNet, which is a CNN using separable depthwise convolutions. For example, this could correspond to... Figure 6 The original CNN design in 610. Figure 13 The middle section shows an example of the modified CNN architecture 1310, where separable depthwise convolutions have been replaced with separable group convolutions. That is, depthwise convolutional layers have been replaced with group convolutional layers. For example, this could correspond to... Figure 6 The modified CNN design in 620. Figure 13 The far right shows a custom CNN architecture 1320, which has been customized from the modified architecture 1310 by removing or replacing pointwise convolutional layers based on a comparison of the effectiveness score of each pointwise convolutional layer with a threshold. Therefore, it can be seen that the five 1330 and 1340 in the network module include group convolutional layers and recurrent channel shifters. That is, the pointwise layers of these network modules have been replaced with recurrent channel shifters.

[0145] Although Figures 10A to 13 An example is shown where, in each network module, the group convolutional layer precedes the pointwise convolutional layer (or a replacement layer, such as channel cyclic shift or channel shuffling). However, it is understood that in other examples, the relative positions may be reversed, i.e., the pointwise convolutional layer (or a replacement layer, such as channel cyclic shift or channel shuffling) precedes the group convolutional layer.

[0146] In the example above, a threshold is used to determine which pointwise convolutional layers to replace or remove. The simplest approach is to remove or replace pointwise convolutional layers that meet predetermined criteria, such as having a role score higher or lower than the threshold. In this way, the baseline number of pointwise convolutional layers is determined by removing or replacing pointwise convolutional layers with role scores that meet predetermined criteria compared to the role score threshold. While this is generally found to provide the best results, in some cases, even better results can be achieved by modifying this baseline number of pointwise convolutional layers.

[0147] Figure 14 This is a graph illustrating an example of the impact scores and impact score threshold τ for multiple pointwise convolutional layers. If a threshold is applied to determine the baseline, pointwise convolutional layers 0 through 6 and 12 are retained, and other pointwise convolutional layers (PWCs) are removed or replaced. This can be referred to as the first version of the CNN. However, this structure of the CNN can be modified to generate a second version of the CNN with additional PWC layers by adding back PWCs (e.g., the layer with the next highest impact score, which is layer 7 in this example) or a third version of the CNN with one less PWC by removing PWC layers (e.g., the remaining layer with the lowest impact score, which is layer 6 in this example). The various versions can then be tested to determine which version provides the best performance in one or more aspects of accuracy, processing and memory requirements, and / or hardware computational efficiency. In other examples, the number of PWC layers can be increased or decreased by more than one or indirectly modified by changing the level of the impact score threshold.

[0148] Figure 15 This is a flowchart illustrating method 1500, which can be implemented by processing resources to create multiple versions of the CNN as described above and determine the version with the best performance.

[0149] At box 1510, the baseline number of pointwise convolutional layers is determined by removing or replacing pointwise convolutional layers that have an action score that meets a predetermined criterion compared to the action score threshold.

[0150] At box 1520, create and train the first version of a neural network with a baseline number of pointwise convolutional layers.

[0151] At box 1530, a second version of the neural network with a larger number of pointwise convolutional layers than the baseline is created and trained.

[0152] At box 1540, a third version of the neural network with fewer pointwise convolutional layers than the baseline is created and trained.

[0153] At box 1550, compare the performance of the first, second, and third versions of the neural network and select the version with the best performance.

[0154] The selected version can then be implemented, for example, on a logic chip such as an ASIC or FPGA.

[0155] In some examples, a second version of the neural network is created by adding at least one pointwise layer back into the neural network or by adjusting the action fraction threshold to increase the number of pointwise layers by at least one.

[0156] In some examples, a third version of the neural network is created by removing at least one additional pointwise layer from the neural network or by adjusting the action fraction threshold to reduce the number of pointwise layers by at least one.

[0157] Figure 16 This is a table comparing the number of pointwise convolutional layers, accuracy, and total number of parameters for various CNNs that use separable depthwise convolutions or group convolutions.

[0158] The CNN in the first row, 1610, is MobileNet, which uses depthwise separable convolutions. For example, it corresponds to... Figure 13 The CNN in the middle row, number 1300, is a modified form of MobileNet. It uses group convolutions and includes network modules with group convolutional layers and subsequent pointwise convolutional layers. For example, it corresponds to 720 in Figure 7. Figure 5 500 Figure 6 620 and 640 in Figure 8 ,as well as Figure 13 The CNN in the third row, 1310, is of the type shown in this disclosure. The third row, 1630, is a CNN according to this disclosure, where some of the pointwise convolutional layers have been removed or replaced. For example, it corresponds to 730 in Figure 7. Figure 5 520 in Figure 6 640 to 660 and Figure 13 In the third row, 1320. The fourth row, 1640, is similar to the third row but has an additional pointwise convolutional layer; for example, it corresponds to... Figure 15 The second version of the method is in box 1530. The fifth row, 1650, is similar to the third row, but without a single pointwise convolutional layer; for example, it corresponds to... Figure 15 The third version of the method in box 1540.

[0159] In these types of CNNs, most of the processing demand comes from pointwise convolutional layers. Therefore, CNNs with fewer pointwise convolutional layers will generally have lower processing demands and are suitable for implementation on devices with less powerful processors. The total number of parameters affects the memory required to implement a CNN, as more parameters require more memory. Therefore, CNNs with fewer parameters will be suitable for implementation on devices with limited memory. Finally, the accuracy of the CNN's analysis results (e.g., classification) is also a consideration; if it drops too low, the CNN's utility may be limited.

[0160] from Figure 16 As can be seen, while the modified CNN1620 exhibits better performance than Mobilenet1610, it has a significantly larger number of parameters and therefore higher memory requirements. In contrast, CNNs 1630, 1640, and 1650, based on the methods and structures taught in this disclosure, have fewer pointwise convolutional layers and significantly fewer parameters compared to Mobilenet1610 or group convolutional CNN 1620, and thus lower processing and memory requirements. However, despite the lower memory and processing requirements, CNNs 1630, 1640, and 1650 achieve higher accuracy than CNNs 1610 and 1620. In this particular example, CNN1630 was found to offer the best performance in terms of the trade-off between accuracy, memory, and processing requirements. However, this is sensitive to the training data used and the type of analysis performed by the CNN, and therefore, in other examples, CNNs of the 1640 or 1650 type may offer better performance. Thus, in some cases, Figure 15 The methods shown may help in identifying and designing CNNs with even better performance.

[0161] Figure 17 This is a schematic diagram illustrating an example optimization system for a neural network. The system includes a layer grouping unit 1710 and a layer customization unit 1720. This system can be used to implement, for example... Figures 4 to 16 The methods and CNNs shown are illustrated.

[0162] The layer grouping unit 1710 is configured to transform an initial model of the neural network into a modified model comprising multiple group convolutional layers and multiple pointwise convolutional layers. Each group convolutional layer comprises multiple filter banks, and each filter bank is configured to operate on a corresponding channel group of input channels to produce a corresponding convolutional output. Each channel group comprises one or more channels. In some examples, each channel group comprises at least two input channels.

[0163] The layer grouping unit 1710 is configured to generate a modified model, wherein each group convolutional layer is followed by a pointwise convolutional layer. The pointwise convolutional layer is a layer configured to mix the convolutional outputs of multiple filter banks to generate multiple output channels from the pointwise convolutional layer output.

[0164] The layer customization unit 1720 is configured to train the modified neural network, determine the role score of each pointwise convolutional layer based on one or more parameters of that pointwise convolutional layer, and remove or replace several pointwise convolutional layers based on a comparison between the role score of each pointwise convolutional layer and a role score threshold. The role score of a pointwise convolutional layer indicates the importance of that pointwise convolutional layer.

[0165] In some examples, the performance score of each pointwise convolutional layer is determined based on the differences between parameter values ​​in the trained pointwise convolutional layers.

[0166] In some examples, the effective score threshold is determined based on the ratio between the number of input channels in one of the channel groups and the number of input channels corresponding to the layer with the most input channels in the neural network.

[0167] Figure 18 An example of a computing system 1800 is shown, which can be used to create neural networks according to the methods described herein and / or as a system for optimizing neural networks as described herein. The computing system includes processing resources 1810, a non-transitory storage medium 1820 for storing data and / or instructions executable by the processing resources, and an I / O interface 1830.

[0168] Processing resource 1810 may include, for example, one or more microprocessors, central processing units, graphics processing units, tensor processing units, or hardware logic chips (such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)). Storage medium 1820 may include, for example, any random access memory (RAM) device, any read-only memory (ROM) device, flash memory, or combinations thereof, or any other type of volatile or non-volatile data storage device that can store data and instructions for execution by processing resource 1810. In some examples, storage medium may include storage devices for persistent storage of electronic data (e.g., hard disk drives), including, for example, magnetically, optically, or circuit-based data recording media and any associated circuitry and physical components for reading and writing data to / from the recording media. I / O interface 1830 may include one or more conventional data communication devices for receiving: (i) input data representing the weights of a neural network; (ii) configuration data representing one or more parameters for the operation of the control system 100A; and / or (iii) instruction data representing instruction codes for the operation of the control system. I / O interface 1830 may include any interface for inputting and / or outputting data or instructions, such as display interface, keyboard interface, wired or wireless network interface, etc.

[0169] Storage medium 1820 may store instructions that, when executed by processing resource 1810, cause any of the methods described herein to be implemented. Computing system 1800 can therefore be used to create and / or optimize neural networks. For example, the computing system may be a general-purpose computer, a server for a cloud computing service. The methods and neural networks described herein can be implemented on a general-purpose computer or server, or a cloud computing service including a processor, or on a dedicated hardware logic chip such as an ASIC or FPGA. In some examples, the CNN is trained and customized on a general-purpose computer or server or a cloud computing service, and then implemented on a hardware logic chip. Implementing the CNN on a hardware logic chip makes it possible to implement the CNN on resource-constrained devices such as smartphones, cameras, tablets, or embedded devices (including logic chips embedded in drones, smart glasses, cars or other vehicles, watches, or home appliances).

[0170] Resource-constrained devices may include physical sensors and processors or logic chips for implementing CNNs as described in any of the examples above. For example, the logic chip may be an FPGA or an ASIC. For example, the device may be a portable device, such as, but not limited to, smartphones, tablets, cameras, drones, watches, wearable devices, etc. Physical sensors may be configured to collect physical data, and processors or logic chips may be configured to classify the data according to the methods described above. For example, a physical sensor may be a camera used to generate visual data such as image and / or video data, and processors or logic chips may be configured to convert the visual data into feature maps for classification or other analysis by the CNN.

[0171] The above embodiments are described by way of example only. Many variations are possible without departing from the scope of this disclosure as defined in the appended claims.

[0172] For clarity, in some instances, this technology is presented as comprising various functional blocks, which include devices, device components, steps or routines in a method embodied in software or a combination of hardware and software.

[0173] The methods described in the examples above can be implemented using computer-executable instructions stored in or otherwise made available from a computer-readable medium. These instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a specific function or group of functions. Some of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include read-only memory, random access memory, magnetic disks or optical disks, flash memory, etc.

[0174] Devices implementing the methods disclosed herein may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include laptop computers, smartphones, small form factor personal computers, personal digital assistants, logic chips, etc. As further examples, this functionality may also be implemented on circuit boards of different chips, or on different processes that can be performed in a single device.

[0175] Instructions, media for transmitting such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are means for providing the functionality described in these disclosures.

[0176] All features disclosed in this specification (including any appended claims, abstract, and drawings) and / or all steps of any method or process so disclosed may be combined in any combination unless at least some of such features and / or steps in the combination are mutually exclusive.

[0177] Unless otherwise expressly stated, each feature disclosed in this specification (including any appended claims, abstract, and drawings) can be used as a substitute feature for the same, equivalent, or similar purpose. Therefore, unless otherwise expressly stated, each disclosed feature is merely one example of a general series of equivalent or similar features.

[0178] Although various examples and other information are used to interpret aspects within the scope of the appended claims, no limitation on the claims should be implied based on specific features or arrangements in such examples, as those skilled in the art will be able to derive a wide variety of implementations from these examples. Furthermore, and although a subject matter may have been described in language specific to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or actions. For example, such functionality may be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claims

1. A method of creating a neural network configured to receive input visual data and generate analysis results for the input visual data, the method comprising: generating, by a processing resource, a plurality of network modules, each network module comprising a group convolution layer and a point-wise convolution layer; wherein the group convolution layer comprises a plurality of filter groups and is configured to receive a plurality of input channels, divide the plurality of input channels into a plurality of channel groups, and convolve each respective channel group with a respective filter group, wherein each channel group comprises one or more channels; wherein the point-wise convolution layer comprises a plurality of point-wise convolution filters and is configured to receive a plurality of channels, each point-wise convolution filter configured to operate on all channels received by the point-wise convolution layer; determining, by the processing resource, an importance score for each point-wise convolution layer, the importance score representing an importance of the point-wise convolution layer; and removing or replacing, by the processing resource, some of the point-wise convolution layers based on comparing the importance score of each point-wise convolution layer to an importance score threshold; wherein the number of point-wise convolution layers in the neural network is adjusted to a baseline number of point-wise convolution layers by removing or replacing some of the point-wise convolution layers based on comparing the importance score of each point-wise convolution layer to the importance score threshold, and wherein the method further comprises: creating and training a first version of the neural network having the baseline number of point-wise convolution layers; creating and training a second version of the neural network having a greater number of point-wise convolution layers than the baseline number; creating and training a third version of the neural network having a lesser number of point-wise convolution layers than the baseline number; and comparing the performance of the first, second, and third versions of the neural network and selecting the version having the best performance; wherein the second version of the neural network is created by adding at least one point-wise convolution layer back into the neural network or by adjusting the importance score threshold to increase the number of point-wise convolution layers in the neural network by at least one; and wherein the third version of the neural network is created by removing at least one other point-wise convolution layer from the neural network or by adjusting the importance score threshold to decrease the number of point-wise convolution layers in the neural network by at least one.

2. The method of claim 1, wherein, For each group convolution layer, all channel groups in the group convolution layer have the same number of channels.

3. The method of claim 1, comprising determining the importance score of each point-wise convolution layer based on one or more parameters of the point-wise convolution layer.

4. The method of claim 1, wherein, The importance score of a point-wise convolution layer is based on a statistical measure of a distribution of weights in the point-wise convolution layer.

5. The method of claim 1, wherein, The importance score of each point-wise convolution layer is determined based on a scaling parameter of a batch normalization process following convolution of the point-wise convolution layer.

6. The method of claim 5, wherein, The importance score of each point-wise convolution layer is determined based on a scaling parameter of the batch normalization process and a number of point-wise filters in the point-wise convolution layer.

7. The method of claim 1, wherein, Removing a point-wise convolution layer from a network module in response to the comparison of the importance score of the point-wise convolution layer to the importance score threshold matching a predetermined criterion.

8. The method of claim 1, wherein, Replacing a point-wise convolution layer with a channel circular shift process or a channel shuffle process in response to the comparison of the importance score of the point-wise convolution layer to the importance score threshold matching a predetermined criterion.

9. The method of claim 1, wherein, The action score threshold is determined based on a size of the channel groups.

10. The method of claim 9, wherein, When the number of channels in each channel group is the same, the size of the channel groups is defined as the number of channels in each channel group (Cg).

11. The method of claim 9, wherein, When the number of channels in each channel group is different between groups of convolutional layers, the size of the channel groups is defined as the number of channels in the channel group with the least number of channels (Cg(min)).

12. The method of claim 9, wherein, The action score threshold is determined based on the size of the channel groups and a number of channels operated on by a point-wise convolutional layer in the neural network that operates on the largest number of channels (Cmax).

13. A convolutional neural network for image processing, the convolutional neural network comprising: a plurality of first type network modules, each first type network module comprising a group convolutional layer and a point-wise convolutional layer; and a plurality of second type network modules, the plurality of second type network modules comprising a group convolutional layer and a circular channel shift unit or a channel shuffle unit for mixing channels; wherein each group convolutional layer comprises a plurality of filter groups, wherein each filter group comprises one or more filters, and wherein the group convolutional layer is configured to receive a plurality of input channels, divide the plurality of input channels into a plurality of channel groups, and convolve each respective channel group with a respective filter group of the plurality of filter groups to produce a plurality of output channels; and wherein each point-wise convolutional layer comprises a plurality of point-wise convolutional filters and is configured to receive a plurality of channels, each point-wise convolutional filter being configured to operate on all channels received by the point-wise convolutional layer; the convolutional neural network removes or replaces some of the point-wise convolutional layers based on comparing an action score of each point-wise convolutional layer to an action score threshold, adjusts a number of point-wise convolutional layers in the neural network to a baseline number of point-wise convolutional layers, and wherein the convolutional neural network further comprises: creating and training a first version of the neural network with the baseline number of point-wise convolutional layers; creating and training a second version of the neural network with a greater number of point-wise convolutional layers than the baseline number; creating and training a third version of the neural network with a lesser number of point-wise convolutional layers than the baseline number; and comparing performance of the first, second, and third versions of the neural network and selecting the version with the best performance; wherein the second version of the neural network is created by adding at least one point-wise convolutional layer back into the neural network or by adjusting the action score threshold to add at least one to the number of point-wise convolutional layers in the neural network; and wherein the third version of the neural network is created by removing at least one other point-wise convolutional layer from the neural network or by adjusting the action score threshold to subtract at least one from the number of point-wise convolutional layers in the neural network.

14. The convolutional neural network of claim 13, wherein, The network modules are configured such that the plurality of first type network modules are stacked one after another, followed by the plurality of second type network modules stacked one after another, and the plurality of second type network modules are followed by at least one first type network module. The action score threshold is determined based on a size of the channel groups. When the number of channels in each channel group is the same, the size of the channel groups is defined as the number of channels in each channel group (Cg). When the number of channels in each channel group is different between groups of convolutional layers, the size of the channel groups is defined as the number of channels in the channel group with the least number of channels (Cg(min)). The action score threshold is determined based on the size of the channel groups and a number of channels operated on by a point-wise convolutional layer in the neural network that operates on the largest number of channels (Cmax).

13. A convolutional neural network for image processing, the convolutional neural network comprising: a plurality of first type network modules, each first type network module comprising a group convolutional layer and a point-wise convolutional layer; and a plurality of second type network modules, the plurality of second type network modules comprising a group convolutional layer and a circular channel shift unit or a channel shuffle unit for mixing channels; wherein each group convolutional layer comprises a plurality of filter groups, wherein each filter group comprises one or more filters, and wherein the group convolutional layer is configured to receive a plurality of input channels, divide the plurality of input channels into a plurality of channel groups, and convolve each respective channel group with a respective filter group of the plurality of filter groups to produce a plurality of output channels; and wherein each point-wise convolutional layer comprises a plurality of point-wise convolutional filters and is configured to receive a plurality of channels, each point-wise convolutional filter being configured to operate on all channels received by the point-wise convolutional layer; the convolutional neural network removes or replaces some of the point-wise convolutional layers based on comparing an action score of each point-wise convolutional layer to an action score threshold, adjusts a number of point-wise convolutional layers in the neural network to a baseline number of point-wise convolutional layers, and wherein the convolutional neural network further comprises: creating and training a first version of the neural network with the baseline number of point-wise convolutional layers; creating and training a second version of the neural network with a greater number of point-wise convolutional layers than the baseline number; creating and training a third version of the neural network with a lesser number of point-wise convolutional layers than the baseline number; and comparing performance of the first, second, and third versions of the neural network and selecting the version with the best performance; wherein the second version of the neural network is created by adding at least one point-wise convolutional layer back into the neural network or by adjusting the action score threshold to add at least one to the number of point-wise convolutional layers in the neural network; and wherein the third version of the neural network is created by removing at least one other point-wise convolutional layer from the neural network or by adjusting the action score threshold to subtract at least one from the number of point-wise convolutional layers in the neural network. The network modules are configured such that the plurality of first type network modules are stacked one after another, followed by the plurality of second type network modules stacked one after another, and the plurality of second type network modules are followed by at least one first type network module.

15. The convolutional neural network of claim 13, wherein, Each point-wise convolution layer has an action score determined based on a difference statistic between weights in the point-wise convolution layer, and the action score of each point-wise convolution layer exceeds an action score threshold defined based on a number of channels of one of the plurality of channel groups.

Images