Methods and systems for training convolutional neural networks using built-in attention

Abstract

Methods and systems for updating weights of a set of convolutional kernels in a convolutional layer of a neural network are described. Based on features of the weights, an attention mechanism is used to generate a set of convolutional kernels with attention-injected weights. For example, a set of position-based attention multipliers is applied to the weights in the set of convolutional kernels, a size-based attention function is applied to the weights in the set of convolutional kernels, or both. An output activation map is generated using the set of convolutional kernels with attention-injected weights. The loss of the neural network is calculated, and the gradients are backpropagated to update the attention-injected weights of the convolutional kernels.

Description

[0001] Cross-application of related applications

[0002] The present invention claims the benefit of prior application priority to U.S. Provisional Patent Application No. 62 / 934,744, filed on November 13, 2019, entitled “Methods and Systems for Training Convolutional Neural Networks Using Built-in Attention,” and also claims the benefit of prior application priority to U.S. Provisional Patent Application No. 17 / 095,257, filed on November 11, 2020, entitled “Methods and Systems for Training Convolutional Neural Networks Using Built-in Attention,” the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates to an artificial neural network, including a convolutional neural network and its training. Background Art

[0004] Convolutional neural networks (CNNs) are based on convolution operations performed using kernels in convolutional layers. The kernels consist of a set of weights, and training a CNN involves learning appropriate weight values for different weight positions throughout the CNN. If the weights are not trained properly (for example, high-value weights are misplaced during training), the accuracy of the trained CNN will decrease.

[0005] For deep CNNs, there may be many layers, and each layer may have many kernels to learn, which means there may be a large number of weights to learn. During training, it can be assumed that at least some weights will be misplaced. Due to issues such as overfitting and the high computational power required, optimizing the placement of all weights is not appropriate. Therefore, how to focus CNN training on the more important weights becomes a problem. Summary of the Invention

[0006] In various examples, the present invention describes methods and systems that help improve the accuracy of trained CNNs by identifying important weights and focusing CNN training on these important weights. The disclosed examples can be integrated into the training of various existing convolutional neural network architectures.

[0007] In some aspects, the present invention describes a method for updating the weights of a set of convolution kernels of a convolutional layer of a convolutional neural network during neural network training. The method includes: obtaining the set of convolution kernels of the convolutional layer. The method also includes: generating a set of convolution kernels, the set of convolution kernels having attention injection weights by performing at least one of the following: applying a set of position-based attention multipliers to the weights in the set of convolution kernels; or applying a size-based attention function to the weights in the set of convolution kernels. The method also includes: performing convolution on an input activation map using the set of convolution kernels with attention injection weights to generate an output activation map; and updating the attention injection weights in the set of convolution kernels using the back-propagated gradients of the loss calculated for the neural network.

[0008] In any of the examples, the set of position-based attention multipliers can be applied to the weights in the set of convolution kernels to obtain a set of position-excitation weights, wherein the size-based attention function can be applied to the set of position-excitation weights.

[0009] In any of the examples, the method may further include applying a channel-based attention function to the output activation map before computing the loss of the neural network.

[0010] In any of the examples, applying the set of position-based attention multipliers can include learning the set of position-based attention multipliers.

[0011] In any of the examples, learning the set of position-based attention multipliers may include: performing average pooling to obtain an average weight for each convolution kernel; feeding the average weight of the convolution kernel through one or more fully connected layers to learn the attention multiplier for each convolution kernel; and expanding the attention multiplier to all weights in each corresponding convolution kernel to obtain the set of position-based attention multipliers.

[0012] In any of the examples, feeding the average weight of the convolution kernel through the one or more fully connected layers may include: feeding the average weight of the convolution kernel through a first fully connected layer; applying a first activation function to the output of the first fully connected layer; feeding the output of the first activation function to a second fully connected layer; and applying a second activation function to the output of the second fully connected layer.

[0013] In either example, the size-based attention function can apply more attention to larger weights and less attention to smaller weights.

[0014] In any of the examples, the size-based attention function may be: Among them, w m is the weight of the convolution kernel, w A is the weight after applying size-based attention, M A =(1+∈ A )*M, M is all w in the convolution layer m The maximum value of ∈ A is a hyperparameter with a chosen small value.

[0015] In any of the examples, the method may include normalizing the weights in the set of convolution kernels prior to applying the set of position-based attention multipliers or the size-based attention function.

[0016] In some aspects, the present disclosure describes a processing system comprising a processing device and a memory storing instructions that, when executed by the processing device, cause the system to perform any of the methods described herein.

[0017] In some aspects, the present invention describes a computer-readable medium having instructions tangibly stored thereon that, when executed by a processing device of a processing system, cause the processing system to perform any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Reference will now be made, by way of example, to the accompanying drawings which illustrate example embodiments of the present application, in which:

[0019] Figure 1 shows a block diagram of an exemplary processing system that may be used to implement the examples described herein;

[0020] Figure 2 A schematic diagram showing an exemplary architecture of a neural network such as a CNN;

[0021] Figure 3A shows exemplary operations that may be performed by a squeeze and excitation block;

[0022] Figure 3B shows exemplary operations that can be performed by a convolution block of a CNN according to the present invention;

[0023] Figure 4 A flowchart of an exemplary method for training a neural network having convolutional blocks according to the present invention is shown;

[0024] Figure 5 A flowchart illustrating an exemplary method for learning a position-based attention multiplier, which may be Figure 4 A portion of the method shown;

[0025] Figure 6A graph showing an exemplary function for applying size-based attention to weights, which can be used Figure 4 The method shown.

[0026] Similar reference numbers may be used in different drawings to identify similar components. DETAILED DESCRIPTION

[0027] In the examples described herein, the performance of a convolutional neural network (CNN), or other neural network including one or more convolutional blocks, trained according to the present invention can be improved with little or no additional computational cost.

[0028] A neural network such as a CNN comprising one or more convolutional blocks is trained according to the examples described herein. The convolutional blocks of the present invention can be implemented in any neural network such as a CNN. For simplicity, the present invention will refer to a convolutional block alone, however, it should be understood that the one or more convolutional blocks may be part of a larger neural network such as a CNN, and the training of the neural network includes optimizing the weights of a set of convolutional kernels for each convolutional block in the neural network. In the context of the present invention, a convolutional block includes a convolutional layer and may have additional layers such as pooling layers and activation layers. In addition, the present invention may use the term CNN to include deep CNNs.

[0029] The examples described herein can be applied to training CNNs to perform a variety of tasks, including object classification, object detection, semantic segmentation, gesture recognition, action recognition, and other applications, such as image processing or computer vision applications, where CNNs can be used.

[0030] In various examples, the present invention describes identifying important weights in a set of convolution kernels of a convolution block of a neural network, such as a CNN, based on specific characteristics including the size of the weights in the set of convolution kernels and / or the location of the weights in the set of convolution kernels. In some examples, the location of a weight can refer to the layer in the neural network where the convolution kernel containing the weight is found. Based on the fact that some weights in a set of convolution kernels of a convolution block are more important than other weights, the present invention also describes an exemplary method for focusing on or attending to the more important weights in the set of convolution kernels during training of a neural network that includes one or more convolution blocks of the present invention. After the neural network has been trained for a specific task and appropriate weights for the neural network have been learned, including the weights for the set of convolution kernels in each convolution block in the neural network, the learned weights can be fixed and the trained neural network can be deployed and used to perform the specific task for which the neural network was trained based on new data (e.g., a new digital image or point cloud).

[0031] This disclosure describes examples for training a neural network, such as a CNN, comprising one or more convolutional blocks by applying more attention to weights of a set of convolutional kernels within each convolutional block that are considered more important due to their position (e.g., depending on the channel in which the weights of the set of convolutional kernels are found). Furthermore, this disclosure describes examples for training a neural network, such as a CNN, by applying more attention to weights of the set of convolutional kernels within each convolutional block that are considered more important due to their size (e.g., applying more attention to larger weights). Position-based attention and size-based attention can be applied separately or in combination. Furthermore, attention can also be applied based on other characteristics. For example, more attention can be applied to weights associated with larger gradients (e.g., calculated based on a loss function and used to optimize weights during backpropagation). In another example, more attention can be applied to the defining graph of the set of convolutional kernel weights (e.g., applying more attention to weights within a set of convolutional kernels that match the defining graph in a convolutional layer of a convolutional block in the neural network). It should be understood that the present disclosure is not necessarily limited to position-based and / or size-based attention mechanisms.

[0032] Neural networks, such as CNNs, that include one or more convolutional blocks and are trained according to examples of the present invention have been found to improve baseline performance on many computer vision tasks, such as image classification and semantic segmentation, compared to neural networks that include one or more convolutional blocks trained using some existing methods for training neural networks. Such improvements may occur at little or no additional computational cost or memory usage during inference.

[0033] As will be discussed further below, examples of the methods disclosed herein apply attention to the weights of a set of convolution kernels of a convolutional layer and can be used with existing methods that apply attention to the convolution outputs generated by the convolutional layer (e.g., applying attention to a portion of the output activations that may be generated by the convolutional layer, as done by a squeeze and excitation block).

[0034] Existing methods for training neural networks have not attempted to identify the important weights of a set of convolutional kernels of a convolutional block for optimization during backpropagation, and have not attempted to focus training on reducing the misalignment (or mislearning) of the more important weights of a set of convolutional kernels of a convolutional layer.

[0035] Some existing methods for training neural networks (e.g., see Siyuan Qiao et al., "Weight Normalization," arXiv preprint arXiv:1903.10520, 2019; Tim Salimans et al., "Weight Normalization: A Simple Reparameterization for Accelerating Deep Neural Network Training," in Advances in Neural Information Processing Systems, pp. 901–909, 2016; Takeru Miyato et al., "Spectral Normalization for Generative Adversarial Networks," arXiv preprint arXiv:1802.05957, 2018) include weight reparameterization techniques designed to make it easier for optimization algorithms to converge to an optimal solution (i.e., to better optimize the network's performance for a particular task). For example, weight normalization reparameterizes the weights in a way that reduces the Lipschitz constant of the loss and gradient, resulting in a smoother loss function (i.e., continuous with slowly varying derivatives) and a more stable optimal solution. With a more stable optimal solution, the weight values are less likely to be significantly misaligned, allowing neural networks such as CNNs to be trained to an optimal solution with a well-defined minimum. However, such methods for training neural networks such as CNNs do not attempt to identify important weights in a set of convolutional kernels of a convolutional layer, nor do they focus on reducing the misalignment of important weights in a set of convolutional kernels of a convolutional layer.

[0036] In some examples, the disclosed methods and systems for identifying more important weights in a set of convolutional kernels of a convolutional layer can be used to improve future techniques for weight reparameterization.

[0037] Other existing approaches for training CNNs comprising one or more convolutional layers involve attention mechanisms that apply attention to specific parts of the output activation maps generated by the convolutional layers of the CNN (e.g., see Jie Hu et al., “Squeeze and Excitation Networks,” in IEEE Transactions on Computer Vision and Pattern Recognition, pp. 7132–7141, 2018; Irwa Bello et al., “Attention-augmented Convolutional Networks,” arXiv preprint arXiv:1904.09925, 2019; Jongchan Park et al., “BAM: Bottleneck Attention Module,” arXiv preprint arXiv:1807.06514, 2018; Sanghuyn Woo et al., “CBAM: Convolutional Block Attention Module,” in Transactions on the European Conference on Computer Vision (ECCV), pp. 3–19, 2018). Such existing methods typically do not have much control over focusing on specific weights in the convolution kernel - for example, in the squeeze and excitation block of a CNN, the excitation channel of the output activation map causes attention to be applied to all weights of that channel that contribute to generating the output activation map. In addition, such activation attention providing methods typically require additional feature memory, additional computational cost, and / or changes to the CNN architecture during runtime.

[0038] In various examples, the present disclosure describes a mechanism for applying attention (also known as "weight excitation") to the weights in a set of convolutional kernels of a convolutional layer, directly targeting the weights in the set of convolutional kernels during training of a neural network (e.g., a CNN) that includes a convolutional layer. At runtime, little or no additional computational cost or architectural changes may be required. Furthermore, the attention mechanism described herein can be relatively easily added to existing convolutional layers of a neural network by modifying the convolutional layers in the neural network.

[0039] In the present invention, the term "weight incentive" may be used to refer to a process of emphasizing the learning (i.e., optimization) of the weights in a set of convolution kernels of a convolution layer or applying more attention to them during the training of a neural network such as a CNN that includes a convolution layer. A "weight incentive mechanism" may be any mechanism designed to apply more attention (or incentive) to the weights in a set of convolution kernels of a convolution layer. In some cases, "attention" and "attention mechanism" may be terms that can be used to replace "incentive" and "incentive mechanism".

[0040] A system that may be used in the examples disclosed herein is first described.

[0041] Figure 1A block diagram of an exemplary simplified processing system 100 is shown, which may be part of a system for performing operations of a convolutional layer of a neural network according to the examples disclosed herein, including convolution operations. Other processing systems suitable for implementing the embodiments described herein may be used, which may include components different from those discussed below. Although Figure 1 A single instance of each component is shown, but there may be multiple instances of each component in the processing system 100 .

[0042] The processing system 100 may include one or more processing devices 102, such as a processor, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, or a combination thereof. In addition, the processing system 100 may also include one or more optional input / output (I / O) interfaces 104 that may enable connection to one or more optional input devices 114 and / or optional output devices 116.

[0043] In the illustrated example, input devices 114 (e.g., a keyboard, mouse, microphone, touch screen, and / or keys) and output devices 116 (e.g., a display, speakers, and / or printer) are shown as optional external devices for processing system 100. In other examples, one or more of input devices 114 and / or output devices 116 may be included as components of processing system 100. In other examples, no input devices 114 and output devices 116 may be present, in which case I / O interface 104 may not be required.

[0044] The processing system 100 may include one or more optional network interfaces 106 for wired or wireless communication with a network (e.g., an intranet, the Internet, a P2P network, a WAN, and / or a LAN) or other nodes. The network interface 106 may include a wired link (e.g., an Ethernet cable) and / or a wireless link (e.g., one or more antennas) for communication within and / or between networks.

[0045] The processing system 100 may also include one or more storage units 108, which may include mass storage units such as solid-state disks, hard disk drives, magnetic disk drives, and / or optical disk drives. The processing system 100 may include one or more memories 110, which may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The non-transitory memory 110 may store instructions for execution by the processing device 102, such as for executing the examples described herein. The memory 110 may include other software instructions, such as software instructions for implementing an operating system and other applications / functionality. In some examples, the memory 110 may include software instructions executed by the processing device 102 to train a neural network and / or implement a trained neural network, as disclosed herein.

[0046] In some other examples, one or more data sets and / or modules may be provided by an external memory (e.g., an external drive in wired or wireless communication with the processing system 100), or may be provided by a transient or non-transitory computer-readable medium. Examples of non-transitory computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory.

[0047] A bus 112 may be present that provides communications between components of the processing system 100, including the processing device 102, the optional I / O interface 104, the optional network interface 106, the storage unit 108, and / or the memory 110. The bus 112 may be any suitable bus architecture, including, for example, a memory bus, a peripheral bus, or a video bus.

[0048] Although Figure 1 A processing system 100 is shown for performing operations on a convolutional layer including built-in attention during training of a neural network. Any physical machine (e.g., a physical computer, including a server or a distributed computer) can perform operations on a convolutional layer including built-in attention of the present invention. In addition, one or more virtual machines (e.g., virtual machines provided by a cloud computing provider) can perform operations on a convolutional layer including built-in attention of the present invention.

[0049] Figure 2A block diagram of an exemplary neural network 200 including a CNN 204 and a fully connected network (such as a classifier 206) is shown. The neural network 200 in this example is used to perform an object classification task. The neural network 200 has been simplified and is not intended to be limiting, and is only used for illustrative purposes. For example, the input data of the neural network 200 can be image data or video data. The neural network 200 includes a pre-processing block 202, which can perform various operations (e.g., normalization) on the input data to generate pre-processed data (e.g., input activation map) for input to the CNN 204. The CNN 204 includes one or more convolutional layers (not shown). The CNN 204 receives the input activation map (e.g., pre-processed data), and the convolutional layer performs a convolution operation using a set of convolution kernels (each convolution kernel is used for convolution of a corresponding channel in the corresponding convolution layer) to generate an output activation map. As will be discussed further below, the convolution kernel (also referred to as a filter kernel or simply a filter) includes weights. Neural network 200 is trained using a training dataset comprising labeled input data samples in order to learn (i.e., optimize) the weights of neural network 200, including the weights in each convolutional kernel of each convolutional layer in CNN 204. The output activation map is provided to a fully connected neural network, referred to in this example as classifier 206, which then outputs an indication of the class (i.e., category) of the object represented by the input data. Classifier 206 selects an indication of a class from a set of classes predicted by the fully connected network (e.g., classifier 206) based on the predicted probability values for each class. The indication of the class output by classifier 206 is often referred to as a label. Neural network 200 can be trained end-to-end. For example, sampled training data can be received by preprocessing block 202, and the resulting activation map is forward propagated through the layers of CNN 204 and ultimately through the layers of the fully connected network (e.g., classifier 206) to generate an output label. A loss function is used to calculate a loss based on the output, and the gradient of the loss is calculated. The gradients are used to update (i.e., optimize) the weights of each layer of the neural network 200 using backpropagation. This end-to-end training can be performed in multiple iterations until the desired convergence condition is reached (e.g., the weights have converged to an optimal solution). The weights can then be fixed, and the trained neural network 200 can be deployed to perform the task for which the neural network was trained (e.g., a classification task).

[0050] Although Figure 2 CNN 204 is shown as part of neural network 200 with classifier 206, but it should be understood that the present invention can be applied to the training of any neural network having a convolutional layer with a convolution kernel. For example, CNN 204 can be trained (e.g., as a standalone neural network) to perform any machine learning task using the methods described herein.

[0051] The above discussion provides examples that illustrate how CNN 204 can be used as part of a trained neural network for performing specific tasks during inference. In general, the input data can have one, two, or three (or more) dimensions, and the output can have any suitable format, depending on the application.

[0052] In order for neural network 200 to perform a specific task with a desired accuracy, the method used to train neural network 200 is crucial.

[0053] Figure 3A A flow chart of an exemplary method 300 performed by a convolutional layer of a squeeze and excitation block is shown. Method 300 simplifies operations performed by a convolutional layer of the squeeze and excitation block, wherein attention is applied to a convolution output (e.g., an output attention map) generated by the convolution operation performed by the convolutional layer of the squeeze and excitation block. Generally, an input activation map is received by a convolutional layer of the squeeze and excitation block. At 302, a convolution operation is performed by the convolutional layer of the squeeze and excitation block. The weights in the convolution kernel of the convolutional layer of the squeeze and excitation block are used in the convolution operation without paying attention to specific more important weights in the convolution kernel. The convolution operation receives a first number of input channels of the input activation map and outputs a second number of output channels of the output activation map. At 304, attention is applied to the convolution output generated by the convolution operation performed by the convolutional layer of the squeeze and excitation block. In the convolutional layer of the squeeze and excitation block, attention is applied by applying different weights to the output channels of the output activation map using a channel-based attention function, thereby scaling each output channel of the output activation map based on the relative importance of each output channel of the output activation map relative to each other output channel of the output activation map. It is worth noting that attention is not applied directly to the weights in the convolution kernel. The output activation map to which the channel-based attention is applied can be further processed by a fully connected layer to generate an output (e.g., a classification label). The output can be compared with the ground truth label to calculate a loss using a loss function. At 306, the weights of the convolutional layer of the squeeze and excitation block are updated by backpropagating the gradient of the calculated loss.

[0054] Figure 3B A flow chart of an exemplary method 350 performed by a convolutional layer of the present invention during training of a CNN including the convolutional layer of the present invention is shown. According to the present invention, the convolutional layer of the present invention includes a built-in attention mechanism that applies attention to the weights in a set of convolution kernels of the convolutional layer. In combination with the above Figure 3A Unlike the described method 300 which is performed by a convolutional layer of a squeeze and excitation block, the method 350 of the present invention applies attention directly to the weights in a set of convolutional kernels of a convolutional layer of a CNN.

[0055] Similar to method 300, the convolution layer can be a layer in any convolution block of any neural network (e.g., CNN), and the input activation map input into the convolution layer can be, for example, the output of the previous layer of the neural network (e.g., the activation map generated by the preprocessing layer, the activation map generated by the pooling layer, the activation map generated by the activation layer).

[0056] At 352, a convolution operation is performed using a convolution layer that includes built-in attention. Because attention is applied to the weights of the convolution kernels of the convolution layer (rather than to the convolution output (e.g., the output activation map generated by the convolution layer)), this approach can be referred to as "built-in" attention. In the present invention, different attention mechanisms are described (described in further detail below) to enable more attention to be applied to weights in a set of convolution kernels that are considered to be more important. The more important weights in the convolution kernels of the convolution layer are weights that are expected to contribute more to the performance of the neural network, and therefore are weights that should be further optimized during training. In contrast, the less important weights in a set of convolution kernels of the convolution layer are weights that are expected to contribute less to the performance of the neural network, and therefore do not need to be fully learned. The weights in a set of convolution kernels that use built-in attention in this way can be referred to as attention-injected weights.

[0057] At 354, attention can optionally also be applied to the convolution output (e.g., the output activation map generated using the convolution operation). Similar to the attention applied at 304 above, the attention applied at 354 can be applied using a channel-based attention function, such as the attention function used in squeeze and excitation neural networks. Thus, the built-in attention described in the present invention can be used with and complement existing methods of attention-based learning that apply attention to the convolution output (e.g., the output activation map) of a convolutional layer.

[0058] The resulting output activation map can then be further processed by a fully connected layer (e.g., a classifier) to generate an output (e.g., a classification label). The output can be compared to the ground truth label to calculate a loss using a loss function. At 356, the attention injection weights of the convolutional layer of the neural network are updated by backpropagating the gradient of the calculated loss using gradient descent or gradient ascent (e.g., using an error backpropagation algorithm). It is worth noting that since attention has been applied directly to the more important weights of the convolution kernels of the convolutional layer, the loss calculated using the loss function and backpropagation will focus more on optimizing those more important weights.

[0059] After a neural network (e.g., a CNN) has been trained and its weights have been learned (i.e., optimized) to achieve a desired accuracy for the CNN trained for a specific task, the neural network can be deployed and used to perform the specific task for which it was trained based on new input data (e.g., a new image, point cloud, etc.).

[0060] Based on the weight size, the weight of the convolution kernel of the convolution layer of a neural network (e.g., CNN) can be considered as a more important weight (compared to other weights in the CNN). In general, the baseline convolution operation performed by the convolution layer in a CNN can be expressed as:

[0061]

[0062] Among them, y i is the i-th output channel of the convolutional layer, x is the input (e.g., a 1D, 2D, or 3D (or higher dimensional) activation map), is the convolution operator, W i is the i-th convolution kernel. W i With dimensions In×h×w, where In is the number of input channels in the input activation map, h and w are the height and width of the convolution kernel respectively. Assuming x is non-zero, find the same as W i Compared with setting the minimum weight to zero, i The maximum weight of zero will result in y i A greater change occurs (mathematically expressed as ). This means that W i The larger weights in contribute more to the convolution output of the convolution operation. i Compared with the smaller weight in W i Larger weights in may have a greater impact on the performance (e.g., accuracy) of the trained CNN. i Compared with the smaller weight in W i The larger weights in are considered to be more important weights.

[0063] Another feature that may cause a weight to be considered more important is the location of the weight in the CNN (e.g., the specific convolutional layer and convolutional kernel to which the weight belongs). Previous research on the squeeze and excitation block (Jie Hu et al., 2018, cited above) found that some convolutional layers in squeeze and excitation-based CNNs (e.g., CNNs including the squeeze and excitation block) tend to apply attention to some output channels of the output activation maps generated by the squeeze and excitation convolutional layers, rather than to other channels of the output activation maps, almost independently of the input activation maps of those convolutional layers. This finding suggests that during training of squeeze and excitation-based CNNs, some output channels of the output activation maps (and therefore all weights in the convolutional layers responsible for generating the output channels of the output activation maps) become more important than other output channels. Accordingly, weights can have different degrees of importance depending on the location or input channel to which the weight belongs.

[0064] Other features can be used to determine whether some weights in a set of convolution kernels should be considered more important than other weights. For example, weights associated with larger gradients during training can be considered more important than other weights associated with smaller gradients. In another example, weights found in defined locations or that match defined patterns in a convolutional layer of a CNN can be considered more important.

[0065] The present invention describes a weight excitation mechanism that is able to apply more attention to specific weights in a set of convolution kernels of a convolutional layer of a CNN based on some features associated with their importance. Since it has been found that the importance of a weight in some cases depends on its size and / or position features, the present invention describes a weight excitation mechanism based on each of these two features. One weight excitation mechanism is referred to herein as location-based weight excitation (LWE), and the other weight excitation mechanism is referred to herein as magnitude-based weight excitation (MWE). In general, in order to incentivize important weights w j , compared to the size gain provided to other weights, the effect of j Apply a relatively large size gain G j . Since it is used to update the weight w j The back-propagation gradient is also affected by the gain G j Therefore, it is necessary to properly optimize the weight w j Apply more attention.

[0066] Figure 4A flow chart of an exemplary method 400 for applying attention to weights in a set of convolution kernels of a convolutional layer based on defining characteristics of the weights during training of a CNN comprising one or more convolutional layers is shown. In particular, Figure 4 An exemplary method 400 for applying position-based attention and size-based attention to the weights in a set of convolution kernels of a convolution layer during forward propagation training of a CNN is shown. Since attention is applied to the weights in the convolution kernels of the convolution layer within the CNN (rather than to the convolution outputs of the convolution layer of the CNN), this method can be referred to as "built-in" attention. Compared to the forward propagation method 300 described above, the weight excitation mechanism described here results in applying attention directly to the weights in the set of convolution kernels themselves, rather than to the convolution outputs.

[0067] The input activation map of the convolutional layer is a 4D weight tensor (W(Out, In, h, w)). It should be understood that the dimensions may be different depending on the dimensions of the input activation map of the convolutional layer. Optionally, similar to normalizing the input activation map before feeding it to the convolutional layer of a CNN, the weights W can be normalized on each i-th channel of the input activation map. The normalization of the weights in the convolution kernel of the convolutional layer can be calculated as:

[0068] W n,i =(W i -μ i ) / σ i

[0069] Among them, W n,i is the normalized weight of the i-th output channel of the output activation map of the convolutional layer, μ i and σ i are the mean and standard deviation of the weights of the i-th output channel, respectively. The result of normalization is a normalized mean of 0 and a normalized deviation of 1. Such normalization can be performed to help simplify the learning of weights in the convolution kernels of the convolution layers of the CNN. In some examples, normalization may not be performed (e.g., in some cases, when normalization of weights in the convolution kernels of the convolution layers is omitted, the performance of the trained CNN can be improved).

[0070] At 402, a position-based attention multiplier m is learned. Figure 5 This section discusses the details of the subnetwork used to learn the position-based attention multiplier m. The position-based attention multiplier m in this example is an array of different multiplier values applied to the corresponding convolutional kernels in the convolutional layer. The size of the position-based attention multiplier m is an array of dimensions Out × In. Generally, the larger the multiplier, the more attention is applied to the weights in the convolutional kernels of the convolutional layer.

[0071] At 404, an array of position-based attention multipliers m is applied to the weights in the convolution kernels of the convolution layer. For each channel W n,i The normalized weights in the convolution kernel of are applied individually to each multiplier m in the multiplier array. i In this example, a multiplier can be applied using Hadamard multiplication so that

[0072]

[0073] in, represents Hadamard multiplication, W m,i is the weight in the convolution kernel of the i-th output channel after applying the position-based attention multiplier. For simplicity, W m,i It can also be called the position excitation weight in the convolution kernel of the i-th output channel.

[0074] As will be discussed further below, the position-based attention multiplier can apply an independent multiplier to each convolution kernel in a set of convolution kernels in a convolution layer. i The principle is that each of these convolutional kernels is located in a different part of the convolutional layer, and the weights in the convolutional kernel have different importance and should therefore receive different levels of attention.

[0075] At 406, the position incentive weight W m Apply size-based attention. The size-based weight incentive mechanism applies more attention to larger weights in a set of convolution kernels. This involves steps 408 and 410.

[0076] At 408 , the maximum value M of the positional excitation weight is calculated.

[0077] At 410, a magnitude incentive weight is calculated. An attention function is applied to the magnitude-based incentive, as will be discussed further below.

[0078] The result of the position-based and size-based incentives is a set of attention injection weights W A , where more important weights (determined by location and size characteristics) are more motivated than less important weights. Figure 3B As discussed, during CNN training, attention is injected into the weights W in the convolution operation. A .

[0079] It should be noted that the position-based and size-based weight incentive mechanisms can only be applied during training. After CNN training is completed, the position-based and size-based weight incentive mechanisms are no longer applicable. During inference runtime, the disclosed weight incentive mechanism is not required. This may result in little or no additional computational cost, memory usage, and structural changes in the overall network architecture.

[0080] Although Figure 4 The application of position-based attention and size-based attention is shown, but it should be understood that position-based attention and size-based attention are separate attention mechanisms that can be used independently of each other. For example, only the position-based attention mechanism can be used, in which case step 406 (including step 408 and step 410) can be omitted, and the last set of attention injection weights are position incentive weights. In another example, only the size-based attention mechanism can be used, in which case steps 402 and 404 can be omitted. In examples where both the position-based attention mechanism and the size-based attention mechanism are used, the order in which the attention mechanisms are applied can be different (for example, position-based attention can be applied first, as in Figure 4 As shown, size-based attention can also be applied first).

[0081] Now combine Figure 5 Discusses details on how to learn position-based attention multipliers. Figure 5 The illustrated method 500 can be used in step 402 discussed above. Compared to the forward propagation method 300 described above, the position-based weight excitation mechanism described herein results in applying attention directly to the weights of a set of convolution kernels of a convolutional layer, rather than applying attention to the convolution outputs generated by the convolutional layer (e.g., the output channels of the output activation map generated by the convolutional layer).

[0082] The overall operation of method 500 can be expressed as

[0083] m i =A2(FC2(A1(FC1(Avg(W n,i ))))) (1)

[0084] Among them, W n,i is the weight in the convolution kernel of the i-th input channel, which can be normalized; Avg represents the average pooling operation; FC1 and FC2 are the first and second fully connected layers, respectively; A1 and A2 are the first and second activation functions (e.g., rectified linear unit (ReLU) and sigmoid function, respectively); m i is the position-based attention multiplier for the ith channel.

[0085] Optionally, at 502, as described above, the weights in each convolution kernel (ie, for each input channel) are normalized to obtain normalized weights W n,i (wherein the subscript i represents the i-th input channel).

[0086] At 504, an average pooling operation Avg is performed. Average pooling is an operation that averages all weights (or normalized weights) in the convolution kernel of the i-th input channel into an average value, thereby generating a tensor of size In. It should be understood that the average pooling operation can be performed on convolution kernels of any dimension (e.g., 1D, 2D, 3D, or higher). When a 1D convolution kernel of size w is used to perform convolution (i.e., for 1D convolution), the average pooling operation averages all w weights in the convolution kernel into a single average value. Similarly, when a 2D convolution kernel of size h×w is used to perform convolution (i.e., for 2D convolution), the average pooling operation averages all weights h×w in the convolution kernel into a single average value; when a 3D convolution kernel of size t×h×w is used to perform convolution (i.e., for 3D convolution), the average pooling operation averages all weights t×h×w in the convolution kernel into a single average value. The average pooling operation can be performed in a dimensionality-reduced form. This can help reduce the number of calculations, help improve computational efficiency, and help simplify learning the weights of the convolutional layers of the CNN. Other types of dimensionality reduction operations can be performed instead.

[0087] The average weight is fed to the first fully connected layer FC1 at 506. The output of the first fully connected layer FC1 is processed by a first activation function A1 at 507. For example, the first activation function A1 may be a rectification function performed using ReLU.

[0088] The output of the first activation function A1 is processed by the second fully connected layer FC2 at 508. The output of the second fully connected layer FC2 is processed by the second activation function A2 at 509. For example, the second activation function A2 may be a sigmoid function.

[0089] The output of the second activation function A2 is another tensor of size In. The use of the first fully connected layer FC1 and the second fully connected layer FC2 (as well as the first activation function A1 and the second activation function A2) enables learning the relative importance of the weights in the convolution kernel of the i-th input channel. Therefore, the tensor of size In can be used as an attention multiplier for the In convolution kernel.

[0090] It is worth noting that FC1 and FC2 of all outputs of a convolutional layer can have shared weights.

[0091] Although Figure 5Two fully connected layers are shown in , but in some examples there may be one fully connected layer, or three (or more) fully connected layers. Each fully connected layer may be followed by a corresponding activation function. Although ReLU and sigmoid functions have been described above, it should be understood that any activation function may be selected independently for each fully connected layer. For example, the activation function may be a sigmoid function, a linear rectifier function, a leaky linear rectifier function, or any other suitable activation function. The activation function after the fully connected layer may be used to normalize or regularize the attention multiplier to be used by the position-based weight incentive mechanism. In general, using two fully connected layers (with or without an activation function after them) may be common for learning classification tasks, and may enable learning more complex (e.g., nonlinear) relationships between channels.

[0092] At 510, the tensor of size In for each output channel is expanded by value copying according to the dimension of the corresponding convolution kernel (i.e., for the i-th convolution kernel). For example, in the case where the kernel is a 2D kernel of size h×w, the tensor of size In can be expanded to a tensor m of size In×h×w. i , to form the multiplier array m.

[0093] It is worth noting that the above process (expressed by equation (1)) is for each output channel W n,i Executed, and finally generated In different attention multipliers m i (wherein, i=1, ..., In).

[0094] This array of multipliers can then be used to apply position-based attention to the weights (e.g., using Hadamard multiplication) at 404 discussed above.

[0095] Figure 6 An exemplary size-based attention function f that can be used to apply size-based attention is shown. A (w m ). In some contexts, the attention function can also be called an activation function. The attention function accepts a single weight value w m , and provides a relatively high gain G to weights that are relatively large compared to other weights. In this example, the attention function is:

[0096]

[0097] Among them, M A =(1+∈ A )*M, M is all w in the convolutional layer m The maximum value of ∈ A is a small value (e.g., 0<∈ A <0.2). For smaller values of w m(i.e., smaller weights), the attention function f A Approximately the identification line (ie, w A =w m ). Since the gradient of the identification line is 1, when applying f A Afterwards, w m Backpropagation gradient of small value Not affected. For larger values of w m (i.e., larger weights), the gradient gain gradually increases, and at the same time, due to M A w m The normalization of keeps ∈ bounded (see Eq. (2)). It should be noted that one can choose the hyperparameter ∈ A The value of ∈ can be used to control the performance of the training (for example, to increase the convergence speed, depending on the sensitivity of the optimizer, etc.). A This results in the size-based attention function being close to the identity line (i.e., the influence of size-based attention is small), while smaller values of ∈ A This leads to a larger effect of size-based attention. In some simulations, it is found that ∈ A =0.1 can achieve good performance.

[0098] Other attention functions can be used (e.g. etc.). Usually, the attention function f A (w m ) should be a larger value of w m Provides large size gains, should be differentiable, and avoid vanishing and exploding gradient problems.

[0099] In the present invention, weight incentives can be performed using a weight incentive mechanism based on position and a weight incentive mechanism based on size. These two incentive mechanisms can be used independently. For example, Figure 4 In the context of , steps 402 and 404 can be omitted, and only the size-based weight incentive mechanism is used to obtain the attention injection weight. Conversely, steps 406 to 410 can be omitted, and only the position-based weight incentive mechanism is used to obtain the attention injection weight. Variants of method 400 were considered in the exemplary study, and it was found that position-based weight incentive may have a greater impact on improving accuracy than size-based weight incentive. However, since the position-based weight incentive mechanism can be calculated on a per-channel basis, the additional use of size-based weight incentive may be useful for shift-based convolution. In some examples, in addition to or instead of position-based weight incentive and / or size-based weight incentive, weight incentive may also be performed based on other weight features described above in the present invention.

[0100] In various examples, a method for training CNNs using built-in attention applied directly to weights is described. This method has been found to improve the performance (e.g., accuracy) of CNNs for specific tasks during inference. At the same time, there is little or no increase in computational effort during inference because no mechanism for applying attention to weights is required during inference.

[0101] Furthermore, since the fully connected layers in CNN can also be expressed as convolution operations, the built-in attention mechanism disclosed in this paper can also be applied to various other applications using fully connected layers.

[0102] Although the present invention describes methods and processes with steps arranged in a certain order, one or more steps of the methods and processes may be omitted or changed as appropriate. One or more steps may be performed in an order different from the order described in the present invention.

[0103] Although the present invention has been described at least in part in terms of methods, it will be understood by those skilled in the art that the present invention is also directed to various components for performing at least some aspects and features of the methods, whether through hardware components, software, or any combination thereof. Accordingly, the technical solutions of the present invention can be embodied in the form of software products. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or non-transient computer-readable media, such as DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored thereon, which enable a processing device (e.g., a personal computer, a server, or a network device) to perform examples of the methods disclosed herein.

[0104] The present invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are intended to be illustrative in all respects and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations are understood to be within the scope of the present invention.

[0105] All values and subranges within the disclosed ranges are also disclosed herein. In addition, although the system, equipment, and process disclosed and illustrated herein may include a specific number of elements / assemblies, the system, equipment, and assembly may be modified to include more or less such elements / assemblies. For example, although any disclosed element / assembly may be cited as a singular, the embodiments disclosed herein may be modified to include a plurality of such elements / assemblies. The subject matter described herein is intended to encompass and include all appropriate changes in technology.

Claims

1. A method for updating the weights of a set of convolution kernels of a convolutional layer of a neural network during training of the neural network, characterized in that The method comprises: Obtaining the set of convolution kernels of the convolution layer; Generate a set of convolution kernels with attention-injected weights by performing at least one of the following: Applying a set of position-based attention multipliers to the weights in the set of convolutional kernels; or Applying a size-based attention function to the weights in the set of convolutional kernels; performing convolution on the input activation map using the set of convolution kernels with the attention injection weights to generate an output activation map; The attention injection weights in the set of convolutional kernels are updated using back-propagated gradients of a loss calculated for the neural network, wherein the neural network is used to perform image processing or computer vision applications and the input data of the neural network is image data or video data.

2. The method according to claim 1, characterized in that The set of position-based attention multipliers is applied to the weights in the set of convolution kernels to obtain a set of position-excitation weights, wherein the size-based attention function is applied to the set of position-excitation weights.

3. The method according to claim 1, characterized in that Also includes: A channel-based attention function is applied to the output activation map before computing the loss of the neural network.

4. The method according to any one of claims 1 to 3, characterized in that Applying the set of position-based attention multipliers also includes: The set of position-based attention multipliers is learned.

5. The method according to claim 4, characterized in that Learning the set of position-based attention multipliers includes: Perform average pooling to obtain the average weight of each convolution kernel; Feeding the average weights of the convolution kernels through one or more fully connected layers to learn the attention multiplier for each convolution kernel; The attention multiplier is extended to all weights in each corresponding convolution kernel to obtain the set of position-based attention multipliers.

6. The method according to claim 5, characterized in that Feeding the average weight of the convolution kernel through the one or more fully connected layers includes: Feeding the average weight of the convolution kernel through a first fully connected layer; Applying a first activation function to the output of the first fully connected layer; Feeding the output of the first activation function to a second fully connected layer; A second activation function is applied to the output of the second fully connected layer.

7. The method according to any one of claims 1 to 3, characterized in that The size-based attention function applies more attention to larger weights and less attention to smaller weights.

8. The method according to claim 7, characterized in that The size-based attention function is Among them, w m is the weight of the convolution kernel, w A is the weight after applying size-based attention, M A =(1+∈ A )*M, M is all w in the convolution layer m The maximum value of ∈ A is a hyperparameter with a chosen small value.

9. The method according to any one of claims 1 to 3, characterized in that Also includes: The weights in the set of convolution kernels are normalized before applying the set of position-based attention multipliers or the size-based attention function.

10. A processing system, characterized in that A system comprising a processing device and a memory storing instructions that, when executed by the processing device, cause the processing system to update weights of a set of convolutional kernels of a convolutional layer of a convolutional neural network during neural network training by: Obtaining the set of convolution kernels of the convolution layer; Generate a set of convolution kernels with attention-injected weights by performing at least one of the following: Applying a set of position-based attention multipliers to the weights in the set of convolutional kernels; or Applying a size-based attention function to the weights in the set of convolutional kernels; performing convolution on the input activation map using the set of convolution kernels with the attention injection weights to generate an output activation map; The attention injection weights in the set of convolutional kernels are updated using back-propagated gradients of a loss calculated for the neural network, wherein the neural network is used to perform image processing or computer vision applications and the input data of the neural network is image data or video data.

11. The processing system according to claim 10, characterized in that The set of position-based attention multipliers is applied to the weights in the set of convolution kernels to obtain a set of position-excitation weights, wherein the size-based attention function is applied to the set of position-excitation weights.

12. The processing system according to claim 10, wherein: The instructions further cause the processing system to: A channel-based attention function is applied to the output activation map before computing the loss of the neural network.

13. The processing system according to any one of claims 10 to 12, characterized in that The instructions further cause the processing system to apply the set of location-based attention multipliers by: The set of position-based attention multipliers is learned.

14. The processing system according to claim 13, characterized in that The instructions further cause the processing system to learn the set of position-based attention multipliers by: Perform average pooling to obtain the average weight of each convolution kernel; Feeding the average weights of the convolution kernels through one or more fully connected layers to learn the attention multiplier for each convolution kernel; The attention multiplier is extended to all weights in each corresponding convolution kernel to obtain the set of position-based attention multipliers.

15. The processing system according to claim 14, characterized in that The instructions further cause the processing system to feed the average weights of the convolutional kernels through the one or more fully connected layers by: Feeding the average weight of the convolution kernel through a first fully connected layer; Applying a first activation function to the output of the first fully connected layer; Feeding the output of the first activation function to a second fully connected layer; A second activation function is applied to the output of the second fully connected layer.

16. The processing system according to any one of claims 10 to 12, characterized in that The size-based attention function applies more attention to larger weights and less attention to smaller weights.

17. The processing system according to claim 16, characterized in that The size-based attention function is Among them, w m is the weight of the convolution kernel, w A is the weight after applying size-based attention, M A =(1+∈ A )*M, M is all w in the convolution layer m The maximum value of ∈ A is a hyperparameter with a chosen small value.

18. The processing system according to any one of claims 10 to 12, characterized in that The instructions further cause the processing system to: The weights in the set of convolution kernels are normalized before applying the set of position-based attention multipliers or the size-based attention function.

19. A computer-readable medium having instructions tangibly stored thereon, characterized in that The instructions, when executed by a processing device of a processing system, cause the processing system to perform the method according to any one of claims 1 to 9.

20. A computer program product comprising instructions tangibly stored thereon, characterized in that The instructions, when executed by a processing device of a processing system, cause the processing system to perform the method according to any one of claims 1 to 9.

Images