Multi-dimensional attention for dynamic convolution kernels
By introducing dynamic convolution filters into the convolution layer of computer vision model and adjusting the convolution kernel using attention weights, the problem of insufficient flexibility of convolution layer in the existing technology is solved, and more efficient feature extraction and detection is achieved Improved accuracy.
Patent Information
- Application Number
- CN202280094895.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2024-12-03
AI Technical Summary
The convolutional layers in existing computer vision models lack flexibility in processing input data and cannot effectively utilize the information in the input data, especially the relative position information about important information in the input data, resulting in insufficient detection accuracy and feature extraction capabilities. .
Using dynamic convolution filters, the attention weight is determined at each convolution layer according to the input feature map, the dynamic convolution kernel is generated, and the weight of the convolution kernel is dynamically adjusted to achieve a more refined and adaptive input data. deal with.
It significantly improves the accuracy and feature extraction capabilities of object detection, can improve detection performance without adding additional computing costs, and is suitable for a variety of image processing tasks.
Smart Images

Figure CN119072689A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to computer modeling utilizing convolutional layers, and in particular to models having convolutional layers with dynamic convolution kernels based on inputs to the convolutional layers. Background Art
[0002] Convolutional layers in computer models can be used in many types of image processing to extract features and characterize information about images. Such models can be used in many types of computer vision, such as object detection (e.g., identifying classifications and bounding boxes of objects), image segmentation, object tracking, environmental perception, and so on. For many of these models, convolution is a common and basic operation. In a typical implementation, the convolution kernel applied to the input feature map is determined during training and is static during application. However, this inflexibility can neglect to benefit from information within the input data, which can signal the relative location of important information within the input data, which can be better characterized by different convolution kernels. Although various methods have been used to improve specific types of models, these methods typically modify the model structure, how the input is represented or encoded, and so on. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] The embodiments will be easily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, similar reference numerals refer to similar structural elements. In the figures of the accompanying drawings, the embodiments are illustrated by way of example and not by way of limitation.
[0004] Figure 1A-1B Illustration of convolutional filters used in convolutional layers of various neural networks.
[0005] Figure 2 An architecture for generating dynamic convolution filters according to one embodiment is shown.
[0006] Figure 3 and Figure 4 Illustrated is the application of attention weights and weights for generating weights for dynamic convolution filters according to one embodiment.
[0007] Figure 5 An example of components of an attention model for generating a set of attention weights according to one embodiment is shown.
[0008] Figure 6 An example of an attention model with layers for implementing the attention model is shown in accordance with various embodiments.
[0009] Figure 7 An example structure of an object detector that may include components using convolutional layers that may use dynamic convolutional filters is provided according to various embodiments.
[0010] Figures 8A-8B Shown is a visual comparison of object detection results using a dynamic convolution filter, according to one embodiment.
[0011] Fig. 9 Example computer model inference and computer model training are shown.
[0012] Fig.10 An example neural network architecture is illustrated.
[0013] Fig.11 is a block diagram of an example computing device that may include one or more components for training, analyzing, or implementing a computer model according to any of the embodiments disclosed herein. DETAILED DESCRIPTION
[0014] Overview
[0015] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein.The details of one or more implementations of the subject matter described in this specification are set forth in the following description and drawings.
[0016] The present disclosure provides a convolution layer that determines a dynamic convolution filter based on an input feature map, and applies the dynamic convolution filter as a convolution layer. Convolution layers with dynamic convolution kernels can be used in many types of applications instead of static convolution filters, including object detection and other image processing methods. For example, in object detection, this convolution layer design can be inserted into any component of an object detector with a convolution layer (e.g., the trunk, head, or neck of a model) to significantly improve detection performance at negligible additional computational costs. Unlike current solutions for convolution layer design, this approach can be used to implement dynamic convolution filters for convolution operations at each layer, thereby providing improved object detection accuracy. Similarly, the dynamic convolution layers discussed herein can be used to perform convolutions in any other suitable type of model.
[0017] Specifically, at each convolution layer, the input feature vector is used to determine the attention weights along several dimensions of the convolution filter space. In particular, the attention weights can represent values across spatial size, input channels, output channels and relative to each convolution filter in several convolution filters (e.g., weighting the combination of convolution filters). These types of attention weights complement each other and provide different means for modifying the resulting dynamic convolution filter across various dimensions. Like this, multiple directions in which the attention weights can affect the convolution filter can be referred to as "omnidirectional". In one embodiment, based on the corresponding attention weights of each filter, a static convolution filter set is combined to generate a dynamic convolution filter. The corresponding attention weights along each dimension are generated and applied to generate a dynamic convolution filter. As a means for applying attention weights, attention weights can be applied using element-by-element multiplication operations and subsequent linear summaries to obtain dynamic convolution filters. This approach can significantly enhance the feature extraction capabilities of the basic convolution operation of the detector in a unified manner. As a "drop-in" alternative to a static convolution operation, a dynamic convolution filter can be used as an alternative in many types of models. Additional examples are discussed below, showing example benefits in an object detection model. As such, this approach can also be applied to different object detection solutions and many downstream tasks such as instance segmentation, object tracking, and image captioning. Finally, the use of “omnidirectional” attention allows attention weight scalars to be dynamically learned and applied in a filter / kernel adaptive, sample adaptive, channel-by-channel, or spatially aware manner.
[0018] For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to those skilled in the art that the present disclosure may be implemented without the specific details, or / and the present disclosure may be implemented using only some of the described aspects. In other instances, well-known features are omitted or simplified so as not to obscure the illustrative implementations.
[0019] In the following detailed description, reference is made to the accompanying drawings which form a part of the detailed description, and in which embodiments that may be implemented are shown by way of illustration. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be construed in a restrictive sense.
[0020] Various operations can be described as multiple discrete actions or operations in a manner that is most helpful for understanding the claimed subject matter. However, the order of description should not be interpreted as implying that these operations are necessarily sequence-dependent. In particular, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed, and / or the described operations may be omitted.
[0021] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). When used with reference to a measurement range, the term "between" includes the endpoints of the measurement range. The meanings of "a," "an," and "the" include plural references. The meaning of "in" includes "in" and "on."
[0022] The description uses the phrases "in one embodiment" or "in an embodiment," which may each refer to one or more of the same or different embodiments. In addition, the terms "including," "comprising," "having," and the like as used with respect to the embodiments of the present disclosure are synonymous. The present disclosure may use perspective-based descriptions, such as "above," "below," "top," "bottom," and "side"; such descriptions are used to facilitate discussion and are not intended to limit the application of the disclosed embodiments. The drawings are not necessarily drawn to scale. The terms "substantially," "close," "approximately," "near," and "approximately" generally refer to within + / -20% of a target value. Unless otherwise stated, the use of the ordinal adjectives "first," "second," and "third," etc., describing common objects, merely indicates that different instances of similar objects are referenced, and is not intended to imply that the objects so described must be in a given order in time, space, order, or in any other way.
[0023] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.
[0024] Dynamic convolution kernel
[0025] Figure 1A-B illustrates convolution filters used in convolution layers of various neural networks. Convolution layers are common and fundamental operations of many types of networks, including components of deep object detection models, which are discussed in more detail in the examples below. In general, the convolution operation plays a key role in modeling contextual cues for object detection by processing the input feature map 100 into the output feature map 120 by applying the convolution filter 110. To perform the convolution, the weights of the convolution filter 110 are applied to a sliding window of the input feature map 100 to generate the output feature map 120. As described below with reference to Figure 9-10 As discussed, computer models typically include parameters for processing inputs to predict outputs. Such computer models can be iteratively trained to learn parameters, including weights, for predicting various types of outputs based on input data. Fig.10 As further discussed in , individual layers in a neural network can receive input activations and process the input activations to generate output activations for the layer. Embodiments of convolutional layers including dynamic convolutional filters can be included in many types of computer models, such as those related to Figure 7 -8 discusses the example object detection model.
[0026] Figure 1B 1 shows additional details of the convolution filter 110 as applied to an input portion of the input feature map 100 to generate a corresponding portion of the output feature map 120. The input feature map 100 has a plurality of spatial dimensions, each of which includes a plurality of input channels c in .exist Figure 1B In the example of , the input feature map has two spatial dimensions corresponding to the height H and width W of the input feature map. The spatial dimensions can correspond to spatial information or regions processed by the corresponding computer model to, for example, analyze a received camera image with a resolution (e.g., multiple color channels), although input feature maps 100 with other spatial dimensions can also be used (e.g., for volumetric regions with three dimensions (height, width, and depth)). The convolution filter 110 can be applied to a k×k (e.g., 1×1, 3×3, 5×5, etc.) portion of the spatial dimensions of the input feature map 100 to generate a plurality of output channels c out To process the entire input feature map 100 and generate the entire output feature map 120, the sliding window moves across the input feature map 100 to select a portion of the input feature map 100, applies the convolution filter 110, and generates a corresponding portion of the output feature map 120. Figure 1BIn the example shown in , the input feature map 100 and the output feature map 120 have the same height H and W, which can be generated by applying a convolution filter 110 centered at each spatial location in the input feature map 100. In other embodiments, the output feature map 120 can have a different size than the input feature map 100, such as by modifying which specific k×k portions of the input feature map 100 are selected by the sliding window.
[0027] The convolutional filters 110 each include a k×k portion for the input feature map 100 and a plurality of input channels c in The number of convolution kernels 115 corresponds to the number of output channels c in the output feature map 120. out Each convolution kernel 115 may include weights of a k×k portion of the input from the input feature map 100 and the corresponding input channel c in As such, each convolution kernel 115 in the convolution filter 110 may include a k×k×c in The convolution operation generates the corresponding values of the output channels of the output feature map 120. out The set of convolution kernels 115, the convolution filter 110 may thus include k×k×c in ×c out The weight sets of individual weights, because each spatial location, each input channel, and each output channel (e.g., corresponding to each convolution kernel) can have separately defined weights. The outputs of the convolution kernels 115A-N can then be concatenated to generate an output channel c corresponding to a portion of the output feature map 120 corresponding to a portion of the input feature map 100. out As such, as used herein, convolution kernel 115 refers to a weight for performing a convolution operation with a portion of input feature map 100 to determine a value of an output channel in output feature map 120, and convolution filter 110 refers to a set of convolution kernels 115A-N, which is a weight for an output channel c in output feature map 120 for a portion of input feature map 100. out The corresponding set generates values.
[0028] Figure 2 FIG. 2 shows an architecture for generating a dynamic convolution filter 230 according to one embodiment. The dynamic convolution filter 230 is designated as and is determined based on a combination of static convolution filters 220A-K (designated as ). Similar to Figure 1B, each of the static convolution filters 220A-K includes a set of convolution kernels describing weights for generating corresponding output channels. To determine the dynamic convolution filter 230, the static convolution filters 220 are modified by the attention weight set and then combined to determine the dynamic convolution filter 230.
[0029] The set of weights applied by the convolution operation (e.g., the weights of the dynamic convolution kernel 230) is therefore dynamic and is a function of the input feature map input to the convolution layer. In order to determine the weights of the dynamic convolution kernel 230 to be applied in the convolution operation of the convolution layer (e.g., Figure 1A -B), the input feature map is input to the attention model 200, which determines the attention weights (as the attention weight set 205) based on the input feature map for determining the weights of the dynamic convolution filter 230 (i.e., the corresponding output filter The attention model 200 determines a set of attention weights based on a particular distribution of features in the input feature map. That is, the way in which a particular data sample (i.e., an input feature map) assigns its values (e.g., which channels of this input feature map contain more or less information) can be used to determine a set of attention weights 205 for determining the dynamic convolution filter 230. The attention weight set 205 describes corresponding attention weights across multiple dimensions, which may include a set of spatial positions, input channels, output channels, and convolution kernels. The attention weight set 205 may therefore include a set of spatial attention weights Att describing weights for multiple spatial positions s , the input channel attention weight set Att describing the attention weights of multiple input channels c , a core attention weight set Att describing the corresponding weights of each static convolution filter 220 f , and the output channel attention weight set Att describing the attention weights of multiple output channels w . Each type of these attention weights can be different for each static convolution filter 220, so that each static convolution filter 220 can vary across the corresponding dimension for combination in the dynamic convolution filter 230. As such, each corresponding attention weight type can have a set of attention weights for the corresponding static convolution filter 220A-K, as designated by the corresponding subscripts. For example, the first static convolution filter 220A can be represented by att s1 ,att c1 ,att f1 and att w1 Modified, and the final static convolution filter 220K is composed of att sK ,att cK ,attfK and att wK Modifications. In some embodiments, a single static convolution filter 220 may be used, in which case there may be no kernel attention weights.
[0030] The following table highlights example characteristics of various types of attention weights that may be used to generate the dynamic convolution kernel 230.
[0031]
[0032] Table 1
[0033] Figure 3 and Figure 4 The application of weights and attention weights for generating a dynamic convolution filter 350 according to one embodiment is illustrated. Each type of attention weight can provide a corresponding set of values for modifying each convolution filter across a corresponding dimension, which can include a spatial position, an input channel, an output channel, and as a corresponding filter for combining with other filters to generate a dynamic convolution filter 350 (e.g., by combining corresponding convolution kernels). As such, each weight type of each weight set can include a weight for each possible position of the corresponding dimension of a static convolution filter, as shown in Table 1. Figure 3 shows that for the first static convolution filter Example application of corresponding attention weights to the example static convolution filter 300. Figure 3 As shown in and discussed above, the convolution filter 300 Contains a collection of convolution kernels, each corresponding to an output channel c out (e.g., 1-n).
[0034] As shown in Table 1, the spatial attention weights 310 of the static convolution filter 300 include a set of s(k×k) individual weights for the spatial locations of the convolution kernel. For example, the k×k (shown here as 3×3) spatial region processed by the kernel is represented by the k×k region corresponding to the first static convolution filter. att s1 The spatial attention weights 310 are described by s attention weights (here 9 attention weights). Each of the spatial attention weights 310 is applied to the input channel of the static convolution filter 300 and the corresponding position of each static convolution kernel in the static convolution kernel. Like this, the spatial attention weights 310 can similarly modify the attention weights across spatial dimensions, across different input channels of the filter, and across different output channels.
[0035] Similarly, the corresponding input channel attention weight 320 (att c1 ) includes corresponding to the input channel c inAs with the spatial attention weights 310, the input channel attention weights 320 are applied to the corresponding input channels of the kernels across the static convolution filter 300. The input channel attention weights 320 can thus modify the weights on the input channels across spatial locations and convolution kernels (e.g., output channels).
[0036] The output channel attention weights 330 can specify specific attention weights for modifying the entire individual convolution kernels (each corresponding to the output channel of the filter) of the static convolution filter 300. Thus, the output attention weights provide a set of weights for modifying the relative contribution of individual kernels (i.e., for the final dynamic output channel) without affecting the contribution of the corresponding individual convolution filter as a whole. The contribution of the static convolution filter as a whole can be specified by the core attention weights 340, which describe the corresponding contributions of the kernels from the filter when combined to the individual kernels of the dynamic convolution filter 350. Like this, after applying the attention weights to each static convolution filter, the attention-modified kernels can be combined to generate the dynamic convolution filter 350. In one embodiment, each of the attention weights can be applied multiplicatively according to the corresponding attention weight dimension of each static convolution filter, and then summed to produce the dynamic convolution filter 350. The dynamic convolution filter 350 in this embodiment can be formally described as:
[0037]
[0038] In equation 1, is the dynamic convolution kernel,
[0039] yes The static convolution kernel in the static convolution kernel, and
[0040] ⊙ is an element-wise multiplication (e.g., multiplication of the corresponding attention dimension).
[0041] Like this, the overall convolution operation of the convolutional layer can be defined as:
[0042]
[0043] in is the output feature map, x is the input feature map, is the dynamic convolution filter, and the attention weight set att is the input feature map Function, such as Figure 5-6 Further discussed in .
[0044] As another way to express the effect of applying attention weights across spatial locations, input channels, output channels, and convolutional filters, consider the weights w, which have the ability to be expressed relative to the description s×c in ×c out Static convolution filter The weight of the concentrated s, c in and c out To determine the weight w' of the attention modification, the corresponding weight of w relative to the position of each attention weight dimension in the attention weight dimension is applied:
[0045]
[0046] Figure 4 An example application of corresponding attention weights according to one embodiment is shown. Figure 4 The illustration of shows the corresponding dimensions of the attention weights with example shading. As in the equation above, “⊙” represents an element-wise multiplication across the dimensions of the corresponding static convolutional filter. Figure 4 As shown in the example of , the static convolution filter 400 may include K static convolution filters, which may include corresponding to c out After applying the attention weights, a dynamic convolution kernel 410 can be generated by combining (e.g., summing) the weights from the corresponding convolution kernels.
[0047] To apply the attention weights, the static convolution layer 400 may multiply the corresponding attention weights element-wise. Figure 4 4 shows that the corresponding attention weights are applied to the corresponding parts of the static convolution filter 400. The spatial attention weights 420 can be defined by k×k(s) weights with a depth of 1, which are applied to the positions s in each convolution kernel of the filter. Similarly, the input channel attention weights 430 can be weighted at w As 1×1×c in The weight set is applied to the corresponding k×k×c in The output channel attention weight 440 is calculated by affecting the k×k×c in Finally, the kernel attention weights 450 represent the respective contributions of the individual static convolution filters and can provide attention weights to influence the entire corresponding filter. Each single value of the 1×1×K kernel attention weight 450 affects the entire static convolution filter By applying various dimensions of attention weights in an element-by-element manner, several dimensions of attention weights can be applied to affect the dynamic convolution kernel 410, and the static convolution filter 400 is adaptively combined using attention scalars across multiple dimensions. The additional computation to determine the dynamic convolution kernel 410 can be relatively small relative to the computation of applying the convolution kernel to the input feature map, as further discussed below.
[0048] Furthermore, while the weights shown here may be applicable to (and associated with) an individual static convolution filter 220, alternative embodiments of the various types of weights may be applied in different manners, such as according to different individual convolution kernels, or by applying the same set of attention weights of one type to more than one static convolution filter 220 (e.g., applying the same set of spatial attention weights or input channel attention weights to multiple static convolution filters 220).
[0049] Figure 5 An example of components of an attention model for generating a set of attention weights 540 is shown according to one embodiment. Figure 5 and Figure 6 The attention model shown in is one that can be used for Figure 2. In this example, the input feature map 500 is processed by the input representation layer 510 to generate a dense input representation 520 of the input features. The specific data of the dense input representation 520 may be different in different embodiments, and may include fewer channels and compressed spatial regions relative to the input feature map 500. In one embodiment, the dense input representation 520 has fewer spatial locations and fewer channels than the input feature map 500, so that the channels are compressed by the input representation layer 510, and relevant channel and spatial information is generated. In one embodiment, the W×H spatial locations of the input feature map 500 are compressed into a one-dimensional array of channel information, so that the dense input representation 520 represents the relative positions of activations across the channels of the input feature map 500. As another benefit, this can allow the input representation layer 510 to receive input feature maps 500 of different sizes, so that relevant channel information can be pooled across the variable width and height of the input feature map. As such, the input representation layer 510 may include a feature aggregation layer that compresses the dimensions of the input feature map 500, and one or more excitation / activation layers to convert into a dense input representation 520. In this example, the attention model uses an input representation layer 510 to generate a joint representation, which is then used by a set of attention output layers 530A-D to generate corresponding sets of attention weights 540 for spatial attention weights 540A, input channel attention weights 540B, output channel attention weights 540C, and core attention weights 540D. Although weights are shown to be generated for each static convolution filter 220A-K in this figure, in other embodiments, the various attention weights for different dimensions may be represented differently.
[0050] Each of the corresponding attention dimensions may produce a resulting weight set of different dimensions from the attention output layer 530. For example, the kernel attention weight describes a value for each of the K convolutional filters, such that the kernel attention weight set is described as a 1×K array. Similarly, the spatial attention weight set att s 540A can generate s weights for each of the K filters, resulting in s×K spatial attention weights; input channel attention weight set att c 540B can generate c for each of the K filters in weights, generating c in ×K input channel attention weights; and output channel attention weight set att f 540C can generate c for each of the K filters out weights, generating c out ×K output channel attention weights.
[0051] The attention output layers 530A-D can utilize various types of processing layers to generate corresponding types of attention weights 540A-D. In general, a dense input representation can be interpreted by a layer and scaled to the dimensions of the corresponding type of attention weights. The layers of the attention model (e.g., in the input representation layer 510 and the attention output layers 530A-D) can therefore include various parameters that can be learned during the training of the convolutional layers. The learned parameters can be used to generate an effective dense input representation 520 from the input feature map 500 and convert the dense input representation 520 into each of the corresponding attention weights 540A-D.
[0052] Figure 6 An example of an attention model with layers for implementing the attention model according to various embodiments is shown. Similar to the previous example, Figure 6 The convolution layer of provides the generation of a dynamic convolution filter 630, which is used for the convolution operation of converting the input feature map into the output feature map. The attention model receives the input feature map to generate multi-dimensional attention weights across several dimensions, which can be used to dynamically modify and combine the convolution filters for the dynamic convolution filter 630.
[0053] In this example, the attention model includes a channel pooling layer 600 that receives an input feature map to generate a channel descriptor. The channel pooling layer 600 pools the values across the spatial dimensions of the input feature map to obtain a representative value in the channel descriptor. In this example, the channel pooling layer 600 may perform global average pooling or global maximum pooling to determine a value for each channel of the input feature map that represents the channel of the feature map as a whole. This may reduce the dimension of the input feature map from H×W×c to 1 in the channel descriptor. in Reduce to 1×1×c in Next, to obtain a dense input representation, additional layers are performed to reduce the number of channels to a dense input representation through a channel squeeze and excitation layer 610.
[0054] The channel squeeze and excitation layer 610 receives the channel descriptors and generates a dense input representation that describes the relative portions of interest across channels. In one embodiment, the channel squeeze and excitation layer 610 is implemented with a fully connected layer, such as a multilayer perceptron (MLP), followed by a batch normalization layer (BN) and an activation layer, such as a rectified linear unit (ReLU) layer. In order to apply the dense input representation and generate corresponding attention weights, in this example, the attention output layer 620A-D includes a fully connected layer (e.g., MLP) and a SoftMax layer to scale (via the connection layer) and smooth (via SoftMax) the dense input representation to the corresponding size of the corresponding attention weight type. Like this, the attention model can be considered to compress / encode the input feature value into a dense input representation, which is then expanded / decoded into the corresponding size of the attention weight by the attention output layer 620A-D. In this example, the dense input representation can be jointly determined for each attention output layer in the attention output layer; in other embodiments, the input feature map can be processed differently with more or fewer joint layers between different types of attention weight output dimensions.
[0055] Also as discussed above, each convolution filter across various dimensions The corresponding weight (For example, att sn ,att cn ,att fn and att wn ) can be multiplied with each other and with the corresponding filter The convolution kernels are multiplied and then summed to generate the corresponding convolution kernel of the dynamic convolution filter 630. This is more formally given by Equation 1 discussed above. As such, the various attention weights provide scalars along multiple dimensions for modifying the filter (s) to be combined for the dynamic convolution filter 630.
[0056] The various types of attention weights across multiple dimensions allow the attention mechanism to more precisely modify the convolution operations of the layer with dynamic convolution filters 630 based on different characteristics of the input feature maps. Convolutional layers with dynamic convolution filters can be used with many types of computer models, particularly those that process images or videos (e.g., as a sequence of images or as individual images). Because the attention model can be included within the layer, the parameters of the attention model of the dynamic convolution filter 630 can be learned relative to a specific convolution layer of the model and can be "plugged in" for previous convolution layers using static or other convolution layers. Like this, convolutional layers can be before or after other types of layers, as in existing model architectures, because the convolution operations can use dynamic convolution kernels, but otherwise leave the input feature maps and output feature maps the same as existing convolutional layers. During training of the computer model and convolutional layer, the convolutional layer can learn a static convolution filter. , as well as the parameters of the attention model itself, such as the parameters of the fully connected, normalized, and activation layers discussed above.
[0057] Object Detection Using Dynamic Convolutional Kernels
[0058] Figure 7 An example structure of an object detector 710 is provided that may include components using convolutional layers that may use dynamic convolutional filters according to various embodiments. Figure 7 An example object detector 710 is shown that receives an image 700 and can generate an output 730 describing an object detected in the input image 700. The object detector 710 includes various components, such as a backbone 705A, a neck 705B, and a head 705C, which are generally composed of several types of layers, including corresponding convolutional layers 720A-C. In this example architecture, the backbone 705A extracts core features from the image 700, which may include, for example, features that can be applicable to various image processing tasks, while the neck 705B can generate more complex features that can be used by one or more "head" models for specific purposes, such as separate types of objects. In various embodiments, the object detector 710 can also be considered a one-stage or two-stage detector. In a one-stage detector, object classification and corresponding boundaries of the object can be performed jointly; in a two-stage detector, the first stage (e.g., the first part) of the model can generate a set of candidate bounding boxes (e.g., regions of interest in the image), and the second stage (e.g., the second part) of the model predicts possible classifications based on specific candidate bounding boxes. As a specific example, a dynamic convolution filter may be used in an object detector as any (or all) of the convolutional layers 720 in any of the components 705A-C.
[0059] Object Detection-Experimental Results
[0060] The experimental results prove that Figure 6 The benefits of the dynamic convolution filter shown in and according to Equation 1 are disclosed, and in particular for object detectors with a general and efficient design that can be used in many additional contexts. As verified by extensive experiments on the large-scale object detection dataset MS-COCO, such a dynamic convolution filter achieves a better trade-off between accuracy and efficiency than existing convolution designs. For example, on mainstream object detectors using both lightweight and large CNN (convolutional neural network) architectures as backbones, the disclosed method brings a 4.5-6.3% mAP margin, while the additional computational cost on the MS-COCO dataset is negligible, see detailed results in Table 1 and Figure 6 Visual comparison in .
[0061] The dynamic convolution filter is used as an "insert" for the convolutional layer in mainstream object detectors, including "Fast R-CNN" and "Mask R-CNN" using ResNet-50 and MobileNetV2 as the backbone, using the object detection benchmark MSCOCO as the dataset for evaluation. As shown in Table 2 below, the dynamic convolution filter (called "OAConv") brings promising accuracy improvements to various backbone models and results in a significantly smaller increase in model complexity compared to existing dynamic convolution filters.
[0062]
[0063] Table 2
[0064] Table 2 shows the performance comparison of “OAConv” with different numbers of static convolutional filters (K=1 and K=4), each of which outperforms other state-of-the-art models when used as a backbone for object detection.
[0065] Fig. 8A -B shows a visual comparison of detection results on the MS-COCO dataset from the above experiments. Fig. 8A Shown are object detection results using Fast R-CNN and ResNet-50, the baseline detectors. Figure 8B The detection results using a detector equipped with “OAConv” as the backbone feature detector as discussed above are shown, and the data is shown in Table 2. As shown, the performance of the detector using dynamic convolution filters as the backbone outperforms the baseline detector due to the improved feature extraction capability (e.g., the effectiveness of multi-dimensional attention weights) and utilizes more accurate bounding boxes to locate objects.
[0066] As a further performance discussion, the number of additional parameters for applying dynamic convolutional filters does not significantly increase processing. In one embodiment, the additional parameters of the attention model can be represented as K×c in ×c inr +K×c inr ×(c out +c in +s). When using K = 4 kernels, the squeeze ratio r = 16 (i.e., reducing the channels to a dense input representation), and when taking c in =256,c out =512 and s=3×3=9, the number of additional parameters introduced into a single static convolutional layer is approximately the same as the original static kernel (c out ×c in ×s), which is a lightweight design, especially given the performance improvements shown in Table 2 above.
[0067] Example Computer Modeling
[0068] Fig. 9 Example computer model inference and computer model training are shown. Computer model inference refers to applying a computer model 910 to a set of input data 900 to generate an output or model output 920. The computer model 910 determines the model output 920 based on the parameters of the model (also referred to as model parameters). The parameters of the model can be determined based on a training process that finds an optimization of the model parameters, typically using training data and the expected output of the model for the corresponding training data, as discussed below. The output of the computer model can be referred to as an "inference" because it is a predicted value based on the input data 900 and based on previous example data used in model training.
[0069] The input data 900 and the model output 920 vary depending on the specific use case. For example, for computer vision and image analysis, the input data 900 may be an image with a specific resolution, such as 75×75 pixels, or a point cloud describing a volume. In other applications, the input data 900 may include a vector, such as a sparse vector, representing information about an object. For example, in a recommendation system, such a vector may represent a user-object interaction, such that the sparse vector indicates a separate item that is positively rated by the user. In addition, the input data 900 may be a processed version of another type of input object, such as representing various features of the input object or representing preprocessing of the input object before inputting the object into the computer model 910. As an example, a 1024×1024 resolution image may be processed and subdivided into 64×64 individual image portions, which are the input data 900 processed by the computer model 910. As another example, an input object, such as the sparse vector discussed above, may be processed to determine an embedding or another compact representation of the input object, which may be used to represent the object as input data 900 in the computer model 910. Such additional processing of the input objects may itself be a learned representation of the data, such that another computer model processes the input objects to generate an output that serves as input data 900 for computer model 910. Although not further discussed herein, such additional computer models may be trained independently or in conjunction with computer model 910.
[0070] As described above, model output 920 may depend on the specific application of computer model 910 and represent recommendation systems, computer vision systems, classification systems, tagging systems, weather predictions, autonomous control, and any other type of modeling output / prediction.
[0071] As described above, the computer model 910 includes various model parameters that describe the characteristics and functions of generating the model output 920 from the input data 900. In particular, the model parameters may include the model structure, model weights, and the model execution environment. The model structure may include, for example, a specific type of computer model 910 and its structure and organization. For example, the model structure may specify a neural network, which may include multiple layers, and the model parameters may describe the separate types of layers included in the neural network and the connections between the layers (e.g., which layers' outputs constitute the inputs to which other layers). Such a network may include, for example, feature extraction layers, convolutional layers, pooling / dimensionality reduction layers, activation layers, output / prediction layers, and the like. While in some instances, the model structure may be determined by the designer of the computer model, in other examples, the model structure itself may be learned via a training process, and thus may form certain "model parameters" of the model.
[0072] The model weights may represent the values of the input data 900 that the computer model 910 uses to process the model output 920. Each portion or layer of the computer model 910 may have such weights. For example, the weights may be used to determine the values used to process the inputs to determine the output at a particular portion of the model. In other words, for example, the model weights may describe how to combine or manipulate the values of the input data 900 or the thresholds used to determine activations as the output of the model. As an example, a convolutional layer typically includes a collection of convolution "weights" that together form a convolution filter to be applied to a set of inputs to the layer in a convolution operation. These are then combined, typically with a "bias" parameter and weights for other transformations to generate the output of the convolutional layer.
[0073] Model execution parameters represent parameters describing model execution conditions. In particular, aspects of the model can be implemented on various types of hardware or circuit modules for executing computer models. For example, parts of the model can be implemented in various types of circuit modules, such as general-purpose circuit modules (e.g., general-purpose CPUs), circuit modules dedicated to certain computer model functions (e.g., GPUs or programmable multiplication and accumulation circuits), or circuit modules specially designed for specific computer model applications. In some configurations, different parts of the computer model 910 can be implemented on different types of circuit modules. As discussed below, the training of the model can include optimizing certain types of hardware (e.g., co-training) for certain aspects of the computer model, or can be determined after determining other parameters of the computer model, without considering the configuration of the execution model. In another example, the execution parameters can also determine or limit the types of processes or functions available at different parts of the model, such as the range of values available at certain points in the process, the operations that can be used to perform tasks, etc.
[0074] Thus, computer model training can be used to determine or "train" the values of model parameters of the computer model 940. During training, the model parameters are optimized to "learn" the values of the model parameters (such as individual weights, activation values, model execution environment, etc.), which are based on an optimization function that seeks to improve a cost function (sometimes also referred to as a loss function) to improve the model parameters. Prior to training, the computer model 940 has model parameters that have initial values that can be selected in a variety of ways, such as by randomized initialization, initial values selected based on other or similar computer models, or by other means. During training, the model parameters are modified based on the optimization function to improve the cost / loss function relative to the previous model parameters.
[0075] In many applications, the training data 930 includes a data set used to train the computer model 940. The data set varies depending on the specific application and purpose of the computer model 940. In supervised learning tasks, the training data typically includes a set of training data labels that describe the training data and the desired output of the model relative to the training data. For example, for an object classification task, the training data may include individual images where individual portions, regions, or pixels in the image are labeled with the classification of the object. For this task, the training data may include training data images depicting dogs and humans and training data labels that label regions of the image that include dogs and humans, such that the computer model is intended to learn to also label the same portions of the image as dogs and humans, respectively.
[0076] To train the computer model, a training module (not shown) applies training data 930 to the computer model 940 to determine the output predicted by the model given the training data 930. Although not shown, the training module is a computing module for performing computer model training by executing the computer model according to the input and output of the computer model by given model parameters and modifying the model parameters based on the results. The training module can apply the actual execution environment of the computer model 940, or can simulate the results of the execution environment, for example to estimate the performance, running time, memory or circuit area of the computer model (for example, if dedicated hardware is used). The training module and the training data and model evaluation can be performed by, for example, Fig.11 One or more processing devices of the example computing device 1100 shown in is instantiated with software and / or hardware. In various examples, the training process can also be performed by multiple computing systems in conjunction with each other, such as a distributed / cloud computing system.
[0077] After processing the training input according to the current model parameters of the computer model 940, the predicted output of the model is evaluated 950, and the computer model is evaluated relative to the cost function, and the computer model is optimized using the optimization function of the training model. Depending on the optimization function, the specific training process and training parameters after the model evaluation are updated to improve the optimization function of the computer model. In supervised training (i.e., training data labels are available), the cost function can evaluate the predicted output of the model relative to the training data labels, and evaluate the relative cost or loss of the prediction relative to the "known" labels of the data. This provides a measure of the frequency of correct predictions of the computer model, and can be measured in various ways, such as precision (the frequency of false positives) and recall (the frequency of false negatives). The cost function may also evaluate other characteristics of the model in some cases, such as model complexity, processing speed, memory requirements, physical circuit characteristics (e.g., power requirements, circuit throughput) and other characteristics of the computer model structure and execution environment (e.g., to evaluate or modify these model parameters).
[0078] After determining the result of the cost function, the optimization function determines the modification of the model parameters to improve the cost function of the training data. Many such optimization functions are known to those skilled in the art. Many such methods distinguish the cost function relative to the parameters of the model and determine the modification of the model parameters, which thus improves the cost function. The parameters of the optimization function (including the algorithm for modifying the model parameters) are the training parameters of the optimization function. For example, the optimization algorithm can use gradient descent (or its variant), momentum-based optimization or other optimization methods used in the art, and is suitable for the specific purpose of the model. The optimization algorithm therefore determines the parameter update to the model parameters. In some implementations, the training data is batched, and the parameter update is iteratively applied to the batched training data. For example, the model parameters can be initialized and then applied to the first batch of data to determine the first modification to the model parameters. Then, the second batch of data can be evaluated with the modified model parameters to determine the second modification to the model parameters, and so on, until the stopping point, usually based on or the amount of available training data or the incremental improvement of the model parameters is lower than a threshold (for example, the additional training data no longer continues to improve the model parameters). Additional training parameters can describe the batch size of the training data, the portion of the training data used as validation data, the step size for parameter updates, the learning rate of the model, etc. Additional techniques can also be used to determine the global optimum or deal with nondifferentiable model parameter space.
[0079] Fig.10 An example neural network architecture is illustrated. Typically, a neural network includes an input layer 1010, one or more hidden layers 1020, and an output layer 1030. The data values in each layer of the network are generally determined based on one or more previous layers of the network. Each layer of the network generates a set of values called "activations," which represent the output values of that layer of the network and can be the input to the next layer of the network. For the input layer 1010, the activations are typically the values of the input data, although the input layer 1010 may represent the input data modified by one or more transformations to generate a representation of the input data. For example, in a recommendation system, interactions between users and objects may be represented as a sparse matrix. Individual users or objects may then be represented as input layer 1010 as transformations of the data in the sparse matrix associated with that user or object. The neural network may also receive the output of another (or several) computer model as its input layer 1010, such that Fig.10The input layer 1010 of the neural network shown in is the output of another computer model. Thus, each layer may receive a set of inputs (also referred to as "input activations") representing the activations of one or more previous layers of the network, and generate a set of outputs (also referred to as "output activations") representing the activations of that layer of the network. In other words, the output activations of one layer become the input activations of another layer of the network (except for the final output layer 1030 of the network).
[0080] Each layer of a neural network typically represents its output activations (i.e., also called its outputs) in a matrix, which may be 1, 2, 3, or n-dimensional, depending on the specific structure of the network. Fig.10 As shown in , the dimensions of each layer can be different depending on the design of each layer. The dimensions of the output layer 1030 depend on the characteristics of the predictions made by the model. For example, a computer model for multi-object classification can generate an output layer 1030 with a one-dimensional array, where each position in the array represents the possibility of a different classification of the input layer 1010. In another example of classifying portions of an image, the input layer 1010 can be an image with a resolution such as 512×512, and the output layer can be a 512×512×n matrix, where the output layer 1030 provides n classification predictions for each of the input pixels, so that the corresponding position of each pixel in the input layer 1010 in the output layer 1030 is an n-dimensional array corresponding to the classification prediction for that pixel.
[0081] Hidden layer 1020 provides output activations that differently characterize input layer 1010 in various ways that help efficiently generate output layer 1030. Thus, the hidden layer can be considered to provide additional features or characteristics of input layer 1010. Fig.10 Two hidden layers are shown in FIG, but in practice any number of hidden layers may be provided in various neural network structures.
[0082] Each layer generally determines the output activation value of its position in the activation matrix based on the output activation of one or more previous layers of the neural network (which can be considered as the input activation of the evaluated layer). Each layer applies a function to the input activation to generate its activation. Such layers can include fully connected layers (e.g., each input is connected to each output of the layer), convolutional layers, deconvolutional layers, pooling layers, and recursive layers. Various types of functions can be applied by the layer, including linear combinations, convolution kernels, activation functions, pooling, etc. The parameters of the function of the layer are used to determine the output activation of the layer based on the activation input of the layer, and are usually modified during the model training process. Parameters that describe the contribution of a specific part of the previous layer are generally referred to as weights. For example, in some layers, the function is the product of each input and the corresponding weight to determine the activation of the layer. For neural networks, the parameters of the model as a whole can therefore include the parameters of each layer in the individual layers, and in large-scale networks can include hundreds of thousands, millions, or more different parameters.
[0083] As an example of training a neural network, a cost function is evaluated at the output layer 1030. To determine the modification of the parameters of each layer, the parameters of each previous layer can be evaluated to determine the corresponding modification. In one example, the cost function (or "error") is back-propagated so that the parameters are evaluated by the optimization algorithm for each layer in sequence until the input layer 1010 is reached.
[0084] Example Device
[0085] Fig.11 1 is a block diagram of an example computing device 1100 that may include one or more components for applying or training a computer model according to any of the embodiments disclosed herein. For example, the computing device 1100 may include a model inference module or a training module for applying and training various models using the techniques discussed herein.
[0086] Multiple components in Fig.11 1100, but any one or more of these components may be omitted or duplicated as appropriate for the application. In some embodiments, some or all of the components included in the computing device 1100 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured on a single system-on-chip (SoC) die.
[0087] Additionally, in various embodiments, the computing device 1100 may not include Fig.111100, but the computing device 1100 may include an interface circuit module for coupling to one or more components. For example, the computing device 1100 may not include the display device 1106, but may include a display device interface circuit module (e.g., a connector and a driver circuit module) to which the display device 1106 may be coupled. In another set of examples, the computing device 1100 may not include the audio input device 1118 or the audio output device 1108, but may include an audio input or output device interface circuit module (e.g., a connector and a supporting circuit module) to which the audio input device 1118 or the audio output device 1108 may be coupled.
[0088] The computing device 1100 may include a processing device 1102 (e.g., one or more processing devices). As used herein, the term "processing device" or "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. The processing device 1102 may include one or more digital signal processors (DSPs), application specific ICs (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (dedicated processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices. The computing device 1100 may include a memory 1104, which itself may include one or more memory devices, such as volatile memory (e.g., dynamic random access memory (DRAM)), non-volatile memory (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or hard drive). The memory 1104 may include instructions that can be executed by the processing device for performing methods and functions as discussed herein. Such instructions may be instantiated in various types of memories, which may include non-volatile memories and are stored on one or more non-temporary media. In some embodiments, memory 1104 may include memory that shares a die with processing device 1102. This memory may be used as cache memory and may include embedded dynamic random access memory (eDRAM) or spin transfer torque magnetic random access memory (STT-MRAM).
[0089] In some embodiments, computing device 1100 may include a communication chip 1112 (e.g., one or more communication chips). For example, communication chip 1112 may be configured to manage wireless communications for transmitting data to and from computing device 1100. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. Data may be transferred using modulated electromagnetic radiation passing through a non-solid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they may not contain any wires.
[0090] The communication chip 1112 may implement any of a plurality of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including: Wi-Fi (IEEE 802.11 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects together with any amendments, updates and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as "3GPP2"), etc.). Broadband wireless access (BWA) networks compatible with IEEE 802.16 are generally referred to as WiMAX networks (an acronym for Worldwide Interoperability for Microwave Access), which is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The communication chip 1112 may operate according to a Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication chip 1112 may operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1112 may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution Data Optimized (EV-DO) and its derivatives, and any other wireless protocols designated as 3G, 4G, 5G and higher. In other embodiments, the communication chip 1112 may operate according to other wireless protocols. The computing device 1100 may include an antenna 1122 to facilitate wireless communications and / or receive other wireless communications (such as AM or FM radio transmissions).
[0091] In some embodiments, the communication chip 1112 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1112 may include multiple communication chips. For example, the first communication chip 1112 may be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, while the second communication chip 1112 may be dedicated to long-range wireless communications such as global positioning systems (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first communication chip 1112 may be dedicated to wireless communications, while the second communication chip 1112 may be dedicated to wired communications.
[0092] Computing device 1100 may include battery / power circuit module 1114. Battery / power circuit module 1114 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuit modules for coupling components of computing device 1100 to an energy source (e.g., AC line power) separate from computing device 1100.
[0093] Computing device 1100 may include a display device 1106 (or a corresponding interface circuit module, as discussed above). Display device 1106 may include any visual indicator, such as, for example, a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.
[0094] Computing device 1100 may include an audio output device 1108 (or a corresponding interface circuit module, as discussed above). Audio output device 1108 may include any device that generates an audible indicator, such as, for example, a speaker, headphones, or earphones.
[0095] The computing device 1100 may include an audio input device 1118 (or a corresponding interface circuit module, as discussed above). The audio input device 1118 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital instrument (e.g., an instrument with a Musical Instrument Digital Interface (MIDI) output).
[0096] Computing device 1100 may include GPS device 1116 (or a corresponding interface circuit module, as discussed above). GPS device 1116 may communicate with a satellite-based system and may receive the location of computing device 1100, as is known in the art.
[0097] The computing device 1100 may include other output devices 1110 (or corresponding interface circuit modules, as discussed above). Examples of other output devices 1110 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.
[0098] The computing device 1100 may include other input devices 1120 (or corresponding interface circuit modules, as discussed above). Examples of other input devices 1120 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a quick response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.
[0099] The computing device 1100 may have any desired form factor, such as a handheld or mobile computing device (e.g., a cell phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, etc.), a desktop computing device, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computing device. In some embodiments, the computing device 1100 may be any other electronic device that processes data.
[0100] Selected Examples
[0101] The following paragraphs provide various examples of the embodiments disclosed herein.
[0102] Example 1 provides a method, including: receiving an input feature map of a convolutional layer of a neural network for implementing a dynamic convolution kernel; determining multiple attention weight sets based on the input feature map, the multiple attention weight sets including spatial attention weights describing weights of multiple spatial positions, input channel attention weights describing attention weights of multiple input channels, and output channel attention weights describing attention weights of multiple output channels; determining a convolution weight set for the dynamic convolution kernel, each convolution weight in the convolution weight set being determined by applying a corresponding spatial attention weight in a spatial attention weight set, an input channel attention weight in an input channel attention weight set, and an output channel attention weight in an output channel attention weight set; and applying the convolution weight set to the input feature map to generate an output feature map.
[0103] Example 2 provides the method of Example 1, wherein the multiple attention weight sets are determined by the input feature map applied to the attention computer model.
[0104] Example 3 provides the method of Example 2, wherein the attention computer model pools the values of the input feature map within each channel of the input feature map.
[0105] Example 4 provides the method of Example 3, wherein after pooling the values, the attention computer model reduces the number of channels of the input feature map.
[0106] Example 5 provides the method of Example 2, wherein the attention computer model generates each set of attention weights using one or more parallel neural network layers.
[0107] Example 6 provides the method of any one of Examples 1-5, wherein the multiple attention weight sets include core attention weight sets for multiple convolution kernels, and the convolution weight sets are further based on the core attention weight sets applied to the multiple convolution kernels.
[0108] Example 7 provides the method of Example 2, wherein the multiple attention weight sets are determined by a computer model, and the parameters of the computer model are jointly trained with the multiple convolution kernels.
[0109] Example 8 provides the method of any one of Examples 1-7, wherein the convolution weight set for the dynamic convolution kernel includes multiple output channel filters, and each output channel filter is determined based on the corresponding spatial attention weights, input channel weights, and output channel weights in the multiple attention weight sets.
[0110] Example 9 provides a system comprising: a processor; and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium containing a computer program code executed by the processor, for: receiving an input feature map of a convolutional layer of a neural network for implementing a dynamic convolution kernel; determining a plurality of attention weight sets based on the input feature map, the plurality of attention weight sets comprising spatial attention weights describing weights of a plurality of spatial positions, input channel attention weights describing attention weights of a plurality of input channels, and output channel attention weights describing attention weights of a plurality of output channels; determining a convolution weight set for the dynamic convolution kernel, each convolution weight in the convolution weight set being determined by applying a corresponding spatial attention weight in a spatial attention weight set, an input channel attention weight in an input channel attention weight set, and an output channel attention weight in an output channel attention weight set; and applying the convolution weight set to the input feature map to generate an output feature map.
[0111] Example 10 provides the system of Example 9, wherein the plurality of attention weight sets are determined by the input feature map applied to an attention computer model.
[0112] Example 11 provides the system of Example 10, wherein the attention computer model pools the values of the input feature map within each channel of the input feature map.
[0113] Example 12 provides the system of Example 11, wherein after pooling the values, the attention computer model reduces the number of channels of the input feature map.
[0114] Example 13 provides the system of Example 10, wherein the attention computer model generates each set of attention weights using one or more parallel neural network layers.
[0115] Example 14 provides the system of any of Examples 9-13, wherein the multiple attention weight sets include core attention weight sets for multiple convolution kernels, and the convolution weight sets are further based on the core attention weight sets applied to the multiple convolution kernels.
[0116] Example 15 provides the system of Example 10, wherein the multiple attention weight sets are determined by a computer model, and the parameters of the computer model are jointly trained with the multiple convolution kernels.
[0117] Example 16 provides a system of any of Examples 9-15, wherein the convolution weight set for the dynamic convolution kernel includes multiple output channel filters, and each output channel filter is determined based on corresponding spatial attention weights, input channel weights, and output channel weights in the multiple attention weight sets.
[0118] Example 17 provides a non-temporary computer-readable storage medium containing instructions executable by a processor for: receiving an input feature map of a convolutional layer of a neural network for implementing a dynamic convolution kernel; determining multiple attention weight sets based on the input feature map, the multiple attention weight sets including spatial attention weights describing weights of multiple spatial positions, input channel attention weights describing attention weights of multiple input channels, and output channel attention weights describing attention weights of multiple output channels; determining a convolution weight set for the dynamic convolution kernel, each convolution weight in the convolution weight set being determined by applying a corresponding spatial attention weight in a spatial attention weight set, an input channel attention weight in an input channel attention weight set, and an output channel attention weight in an output channel attention weight set; and applying the convolution weight set to the input feature map to generate an output feature map.
[0119] Example 18 provides the non-transitory computer-readable storage medium of Example 17, wherein the plurality of attention weight sets are determined by the input feature map applied to an attention computer model.
[0120] Example 19 provides the non-transitory computer-readable storage medium of Example 18, wherein the attention computer model pools the values of the input feature map within each channel of the input feature map.
[0121] Example 20 provides the non-transitory computer-readable storage medium of Example 19, wherein after pooling the values, the attention computer model reduces the number of channels of the input feature map.
[0122] Example 21 provides the non-transitory computer-readable storage medium of Example 18, wherein the attention computer model generates each set of attention weights using one or more parallel neural network layers.
[0123] Example 22 provides a non-transitory computer-readable storage medium of any of Examples 17-21, wherein the multiple attention weight sets include core attention weight sets for multiple convolution kernels, and the convolution weight sets are further based on the core attention weight sets applied to the multiple convolution kernels.
[0124] Example 23 provides the non-transitory computer-readable storage medium of Example 18, wherein the multiple attention weight sets are determined by a computer model, and the parameters of the computer model are jointly trained with the multiple convolution kernels.
[0125] Example 24 provides a non-transitory computer-readable storage medium of any one of Examples 17-23, wherein the convolution weight set for the dynamic convolution kernel includes multiple output channel filters, and each output channel filter is determined based on the corresponding spatial attention weights, input channel weights, and output channel weights in the multiple attention weight sets.
[0126] The above description of the illustrated implementation of the present disclosure, including the content described in the abstract, is not intended to be exhaustive or limit the present disclosure to the precise form disclosed. Although specific implementations and examples of the present disclosure are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. In view of the above detailed description, these modifications may be made to the present disclosure.
Claims
1. A method comprising: Receiving an input feature map of a convolutional layer of a neural network for implementing a dynamic convolutional kernel; Determining a plurality of attention weight sets based on the input feature map, the plurality of attention weight sets comprising spatial attention weights describing weights of a plurality of spatial positions, input channel attention weights describing attention weights of a plurality of input channels, and output channel attention weights describing attention weights of a plurality of output channels; Determine a convolution weight set for the dynamic convolution kernel, each convolution weight in the convolution weight set being determined by applying a corresponding spatial attention weight in a spatial attention weight set, an input channel attention weight in an input channel attention weight set, and an output channel attention weight in an output channel attention weight set; as well as The set of convolution weights is applied to the input feature map to generate an output feature map.
2. The method of claim 1, wherein: The plurality of attention weight sets are determined by applying the input feature map to an attention computer model.
3. The method of claim 2, wherein: The attention computer model pools the values of the input feature map within each channel of the input feature map.
4. The method of claim 3, wherein: After pooling the values, the attention computer model reduces the number of channels of the input feature map.
5. The method of claim 2, wherein: The attention computer model generates each set of attention weights using one or more parallel neural network layers.
6. The method of claim 1, wherein: The multiple attention weight sets include core attention weight sets for multiple convolution kernels, and the convolution weight sets are further based on the core attention weight sets applied to the multiple convolution kernels.
7. The method of claim 2, wherein: The multiple attention weight sets are determined by a computer model, and the parameters of the computer model are jointly trained with the multiple convolution kernels.
8. The method of claim 1, wherein: The convolution weight set for the dynamic convolution kernel includes multiple output channel filters, and each output channel filter is determined based on the corresponding spatial attention weight, input channel weight and output channel weight in the multiple attention weight sets.
9. A system comprising: processor; as well as A non-transitory computer readable storage medium containing computer program code executed by the processor for: Receiving an input feature map of a convolutional layer of a neural network for implementing a dynamic convolutional kernel; Determining a plurality of attention weight sets based on the input feature map, the plurality of attention weight sets comprising spatial attention weights describing weights of a plurality of spatial positions, input channel attention weights describing attention weights of a plurality of input channels, and output channel attention weights describing attention weights of a plurality of output channels; Determine a convolution weight set for the dynamic convolution kernel, each convolution weight in the convolution weight set being determined by applying a corresponding spatial attention weight in a spatial attention weight set, an input channel attention weight in an input channel attention weight set, and an output channel attention weight in an output channel attention weight set; as well as The set of convolution weights is applied to the input feature map to generate an output feature map.
10. The system of claim 9, wherein: The plurality of attention weight sets are determined by applying the input feature map to an attention computer model.
11. The system of claim 10, wherein: The attention computer model pools the values of the input feature map within each channel of the input feature map.
12. The system of claim 11, wherein: After pooling the values, the attention computer model reduces the number of channels of the input feature map.
13. The system of claim 10, wherein: The attention computer model generates each set of attention weights using one or more parallel neural network layers.
14. The system of claim 9, wherein: The multiple attention weight sets include core attention weight sets for multiple convolution kernels, and the convolution weight sets are further based on the core attention weight sets applied to the multiple convolution kernels.
15. The system of claim 10, wherein: The multiple attention weight sets are determined by a computer model, and the parameters of the computer model are jointly trained with the multiple convolution kernels.
16. The system of claim 9, wherein: The convolution weight set for the dynamic convolution kernel includes multiple output channel filters, and each output channel filter is determined based on the corresponding spatial attention weight, input channel weight and output channel weight in the multiple attention weight sets.
17. A non-transitory computer-readable storage medium comprising instructions executable by a processor for: Receiving an input feature map of a convolutional layer of a neural network for implementing a dynamic convolutional kernel; Determining a plurality of attention weight sets based on the input feature map, the plurality of attention weight sets comprising spatial attention weights describing weights of a plurality of spatial positions, input channel attention weights describing attention weights of a plurality of input channels, and output channel attention weights describing attention weights of a plurality of output channels; Determine a convolution weight set for the dynamic convolution kernel, each convolution weight in the convolution weight set being determined by applying a corresponding spatial attention weight in a spatial attention weight set, an input channel attention weight in an input channel attention weight set, and an output channel attention weight in an output channel attention weight set; as well as The set of convolution weights is applied to the input feature map to generate an output feature map.
18. The non-transitory computer readable medium of claim 17, wherein: The plurality of attention weight sets are determined by applying the input feature map to an attention computer model.
19. The non-transitory computer readable medium of claim 18, wherein: The attention computer model pools the values of the input feature map within each channel of the input feature map.
20. The non-transitory computer readable medium of claim 19, wherein: After pooling the values, the attention computer model reduces the number of channels of the input feature map.
21. The non-transitory computer readable medium of claim 18, wherein: The attention computer model generates each set of attention weights using one or more parallel neural network layers.
22. The non-transitory computer readable medium of claim 17, wherein: The multiple attention weight sets include core attention weight sets for multiple convolution kernels, and the convolution weight sets are further based on the core attention weight sets applied to the multiple convolution kernels.
23. The non-transitory computer readable medium of claim 18, wherein: The multiple attention weight sets are determined by a computer model, and the parameters of the computer model are jointly trained with the multiple convolution kernels.
24. The non-transitory computer readable medium of claim 17, wherein: The convolution weight set for the dynamic convolution kernel includes multiple output channel filters, and each output channel filter is determined based on the corresponding spatial attention weight, input channel weight and output channel weight in the multiple attention weight sets.
Citation Information
Cited By
Power battery thermal runaway voiceprint early warning method and system
CN120521709A
Monocular depth estimation method and device, storage medium and computer program product
CN120747186A
A monocular depth estimation method, device, storage medium and computer program product
CN120747186B
