Object detection network with spatial uncertainty

Abstract

A hardware accelerator for an object detection network and a method for detecting objects are provided. The present disclosure provides robust object detection that advantageously enhances conventional deterministic bounding box prediction with spatial uncertainty for various computer vision applications such as, for example, autonomous driving, robotic surgery, etc.

Description

BACKGROUND

[0001] The present disclosure relates to computer systems. More specifically, the present disclosure relates to computer systems that include object detection networks.

[0002] Prediction is an essential element of many classification and object detection networks, including machine learning (ML) such as, for example, ANNs with a single hidden layer (i.e., shallow ANNs), deep neural networks (DNNs) such as ANNs with multiple hidden layers (i.e., deep ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc., support vector machines (SVMs), decision trees, Bayesian networks, etc. However, for many reasons such as, for example, insufficient data for a certain class, input data that is out of distribution (OOD) (i.e., data that does not belong to any class), etc., networks can never achieve 100% prediction accuracy. Networks implemented in hardware and software are also susceptible to hard and soft errors that can degrade prediction accuracy or cause fatal events. Typically, classification and object detection networks provide “best” predictions based only on input data and underlying training methods and data.

[0003] Object detection networks such as CNNs typically generate deterministic labels and object bounding boxes during inference. A bounding box is a rectangle that substantially encloses an object and is defined by a location (i.e., a pair of x, y coordinates), a width, and a height. These deterministic outputs are often overconfident, resulting in erroneous predictions of label and bounding box location and size. Various techniques can be used to infer label uncertainty. Unfortunately, there are no known techniques for predicting bounding box spatial uncertainty, which can be important for many industrial applications where objects can not be fully detected due to occlusions, lighting conditions, etc. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 An ANN is depicted in accordance with an embodiment of the present disclosure.

[0005] Figure 2A And Figure 2B A CNN is depicted in accordance with an embodiment of the present disclosure.

[0006] Figure 3 An object detection network is depicted in accordance with an embodiment of the present disclosure.

[0007] Figure 4A An image with bounding boxes is depicted in accordance with an embodiment of the present disclosure.

[0008] Figure 4B An image with a grid and bounding boxes is depicted in accordance with an embodiment of the present disclosure.

[0009] Figure 5 a block diagram depicting a system in accordance with embodiments of the present disclosure is depicted.

[0010] Figure 6 a flowchart representing functions associated with object detection in accordance with embodiments of the present disclosure is depicted. DETAILED DESCRIPTION

[0011] Embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which like reference numerals refer to like elements throughout.

[0012] Embodiments of the present disclosure advantageously provide robust object detection that enhances conventional deterministic bounding box prediction with spatial uncertainty for various computer vision applications, such as, for example, autonomous driving, robotic surgery, etc.

[0013] More specifically, embodiments of the present disclosure advantageously provide an object detection network that includes a backbone feature extraction (BFE) module, a feature detection (FD) module, and a filtering and averaging (FA) module.

[0014] The BFE module is configured to receive image data or a set of image data that includes at least one object, and generate a plurality of feature tensors based on the image data. Each feature tensor has a different scale. In one embodiment, each feature tensor includes feature map data defined by a height, a width, and a plurality of channels, and the scale of a feature tensor is defined as height- width.

[0015] The FD module is configured to receive the feature tensors from the BFE module, perform a plurality of detection passes, and generate an intermediate tensor associated with each feature tensor. Each detection pass includes: applying a random dropout mask to the feature tensors; and for each feature tensor, determining scale-dependent bounding box space data that includes mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-dependent bounding boxes for the object. Each intermediate tensor includes the scale-dependent bounding box space data for each detection pass of the associated feature tensor.

[0016] Applying the random dropout mask to the feature tensors during each detection pass advantageously introduces statistical randomness to the data, which in turn generates bounding box spatial uncertainty.

[0017] In one embodiment, the FD module includes a packet loss mask generation module and a plurality of detector heads. The packet loss mask generation module is configured to generate a random packet loss mask for each feature tensor for each detection pass. Each detector head includes a plurality of convolution modules and at least one packet loss mask application module, and is configured to receive a different feature tensor. During each detection pass, each detector head determines a plurality of scale-dependent bounding boxes for the object based on the feature tensor and the corresponding packet loss mask, determines scale-dependent bounding box spatial data based on the plurality of scale-dependent bounding boxes, and determines an objective score and class probabilities.

[0018] The FA module is configured to receive an intermediate tensor from the FD module, determine scale-independent bounding box spatial data based on the intermediate tensor, and generate an output tensor including the scale-independent bounding box spatial data. The scale-independent bounding box spatial data includes mean angular coordinates for a scale-independent bounding box for the object and a covariance matrix for the angular coordinates.

[0019] An ANN uses a network of interconnected nodes trained through a learning process to model a relationship between input data or signals and output data or signals. The nodes are arranged into various layers, including, for example, an input layer, one or more hidden layers, and an output layer. The input layer receives input data, such as, for example, image data, and the output layer generates output data, such as, for example, a probability that the image data contains a known object. Each hidden layer provides at least a partial transformation of the input data to the output data. A DNN has multiple hidden layers in order to model complex, non-linear relationships between the input data and the output data.

[0020] In a fully connected feed-forward ANN, each node is connected to all nodes in the previous layer, and to all nodes in the subsequent layer. For example, each input layer node is connected to each hidden layer node, each hidden layer node is connected to each input layer node and each output layer node, and each output layer node is connected to each hidden layer node. Additional hidden layers are similarly interconnected. Each connection has a weight value, and each node has an activation function, such as, for example, a linear function, a step function, a sigmoid function, a tanh function, a rectified linear unit (ReLU) function, etc., that determines an output of the node based on a weighted sum of inputs to the node. Input data propagates from the input layer nodes through the respective connection weights to the hidden layer nodes, and then through the respective connection weights to the output layer nodes.

[0021] More specifically, at each input node, input data is provided to an activation function of that node, and then the output of the activation function is provided as an input data value to each hidden layer node. At each hidden layer node, the input data values received from each input layer node are multiplied by respective connection weights, and the resulting products are summed or accumulated into an activation value provided to an activation function of that node. The output of the activation function is then provided as an input data value to each output layer node. At each output layer node, the output data values received from each hidden layer node are multiplied by respective connection weights, and the resulting products are summed or accumulated into an activation value provided to an activation function of that node. The output of the activation function is then provided as output data. Additional hidden layers can be similarly configured to process data.

[0022] A multi-layer perceptron (MLP) is an ANN having an input layer, an output layer, and one or more hidden layers. MLPs can be used in natural language processing applications, such as machine translation, speech recognition, etc. Other ANNs include recurrent neural networks (RNNs), long short-term memory (LSTM), sequence-to-sequence models including encoder RNNs and decoder RNNs, shallow neural networks, etc.

[0023] A CNN is a variant of an MLP that can be used in classification or recognition applications, such as image recognition, speech recognition, etc. A CNN has an input layer, an output layer, and multiple hidden layers including convolutional layers, pooling layers, normalization layers, fully connected layers, etc. Each convolutional layer applies a sliding dot product or cross-correlation to an input volume, applies an activation function to the result, and then provides an activation or output volume to the next layer. The convolutional layers typically use a ReLU function as the activation function. In certain embodiments, the activation function is provided in a separate activation layer, such as a ReLU layer, for example. Pooling layers reduce the size of the output volume received from a preceding convolutional layer, and can compute the average or maximum value of small data clusters, such as 2x2 matrices, for example. In certain embodiments, convolutional layers and pooling layers can form a single layer of a CNN. Fully connected layers follow the convolutional and pooling layers, and include a flattening layer and a classification layer, followed by a normalization layer including a normalization function, such as a SoftMax function. An output layer follows the last fully connected layer; in certain embodiments, the output layer can include a normalization function.

[0024] Figure 1 An ANN 10 according to an embodiment of the disclosure is depicted.

[0025] ANN 10 includes an input layer 20, one or more hidden layers 30, 40, 50, etc., and an output layer 60. Input layer 20 includes one or more input nodes 21, 22, 23, etc. Hidden layer 30 includes one or more hidden nodes 31, 32, 33, 34, 35, etc. Hidden layer 40 includes one or more hidden nodes 41, 42, 43, 44, 45, etc. Hidden layer 50 includes one or more hidden nodes 51, 52, 53, 54, 55, etc. Output layer 60 includes one or more output nodes 61, 62, etc. Generally, ANN 10 includes N hidden layers, input layer 20 includes “i” nodes, hidden layer 30 includes “j” nodes, hidden layer 40 includes “k” nodes, hidden layer 50 includes “m” nodes, and output layer 60 includes “o” nodes. Many variations of input layers, hidden layers, and output layers are obviously possible, including hidden layers that are locally connected to each other rather than fully connected. ANNs with a single hidden layer 30 are shallow ANNs, while ANNs with multiple hidden layers 30, 40, etc. are deep ANNs, i.e., DNNs.

[0026] Training an ANN includes optimizing the connection weights between nodes by minimizing the prediction error of the output data until the ANN achieves a certain level of accuracy. One method is backpropagation or backprop, which iteratively and recursively determines the gradient descent with respect to the connection weights and then adjusts the connection weights to improve the performance of the network.

[0027] Figure 2A A CNN 12 according to embodiments of the disclosure is depicted.

[0028] CNN 12 includes an input layer 20, one or more hidden layers such as convolutional layer 30-1, pooling layer 30-2, hidden (flattened) layer 40, hidden (classification) layer 50, etc., and an output layer 60. Many other variations of input layers, hidden layers, and output layers are contemplated. Because the CNN includes multiple hidden layers, the CNN is a deep neural network, i.e., a DNN.

[0029] Input layer 20 includes one or more input nodes 21, etc., that present input data such as a color image as an input volume to a first convolutional layer, e.g., convolutional layer 30-1. An input volume is a three-dimensional matrix having a width, a height, and a depth. For example, input data representing a color image can be presented as an input volume of 512 pixels x 512 pixels x 3 channels (red, green, blue); other input volume dimensions can also be used, such as 32 x 32 x 3, 64 x 64 x 3, 128 x 128 x 3, etc., 32 x 32 x 1, 64 x 64 x 1, 128 x 128 x 1, 512 x 512 x 1, etc.

[0030] The convolutional layer 30-1 is locally connected to the input layer 20 and includes a plurality of nodes (not shown for clarity) connected to local regions in the input volume. For CNNs using standard convolution, each node computes a dot product between the node’s filter weights and a corresponding local region of the input volume to generate one element of an output volume. An activation function and a bias can be applied to each element of the output volume, and the output volume is then provided as an input volume to the next layer. The activation function and the bias can be applied by each convolutional layer node or by nodes of a subsequent locally connected layer, such as a ReLU layer. Filters or feature detectors identify different features present in an image, such as edges, vertical lines, horizontal lines, curves, etc. Shallower convolutional layers extract higher-level features from an image, while deeper convolutional layers extract lower-level features using a larger number of filters.

[0031] The pooling layer 30-2 is locally connected to the convolutional layer 30-1 and includes a plurality of nodes (not shown for clarity) connected to local regions in the input volume. The pooling layer 30-2 also produces an output volume, which is provided as an input volume to a subsequent layer, such as, for example, another convolutional layer 30-1, a flattening layer 40, etc. In certain embodiments, the convolutional layer 30-1 and the pooling layer 30-2 form a single hidden layer 30. Similarly, in certain embodiments, the convolutional layer 30-1, a ReLU layer, and the pooling layer 30-2 form a single hidden layer 30, while in other embodiments, the convolutional layer 30-1, a batch normalization layer, and a ReLU layer form a single hidden layer 30. In general, the output volume of a convolutional and pooling layer can be described as an output feature map, and one or more single hidden layers 30 form a feature learning portion of the CNN 12.

[0032] The hidden layer 40 is a “flattening” layer that is locally connected to the pooling layer 30-2 and includes one or more hidden (flattening) nodes 41, 42, 43, 44, 45, etc. The hidden (flattening) layer 40 “flattens” the output volume produced by the previous pooling layer 30-2 into a column vector, which is provided to a subsequent fully connected hidden layer 50.

[0033] The hidden layer 50 is a classification layer that is fully connected to the hidden (flattening) layer 40 and includes one or more hidden (classification) nodes 51, 52, 53, 54, 55, etc.

[0034] The output layer 60 includes one or more output nodes 61, 62, etc. and is fully connected to the hidden (classification) layer 50. The fully connected output layer 60 receives the classification results output by the hidden (classification) layer 50 and each node outputs a predicted class score. A normalization function, such as a Softmax function, can be applied to the predicted class scores by the output layer 60 or, alternatively, by an additional layer inserted between the hidden (classification) layer 50 and the output layer 60.

[0035] Similar to ANNs, training a CNN involves optimizing the connection weights between nodes by minimizing the prediction error of the output data until the CNN achieves a certain level of accuracy. As described above, backpropagation can be used to iteratively and recursively determine the gradient descent with respect to the connection weights and then adjust the connection weights to improve the performance of the network. Matrix multiplication operations (and more specifically, MAC operations) are extensively used by CNNs, as well as other ANNs.

[0036] Figure 2B A CNN 14 is depicted, which is a Residual Network (ResNet).

[0037] Essentially, the CNN 14 includes a convolutional layer 30 1 , a max pooling layer 30-2 1 , a convolutional layer 30 2 , a convolutional layer 30 3 , a convolutional layer 30 4 , a convolutional layer 30 5 , a convolutional layer 30 6 , a convolutional layer 30 7 , a convolutional layer 30 8 , a convolutional layer 30 9 , a convolutional layer 30 10 , a convolutional layer 30 11 , a convolutional layer 30 12 , a convolutional layer 30 13 , a convolutional layer 30 14 , a convolutional layer 30 15 , a convolutional layer 30 16 , a convolutional layer 30 17 , an average pooling layer 30-2 2 , a classification (fully connected) layer 50, and a Softmax layer 60. Figure 2B The size of each convolutional kernel and the number of filters in each convolutional layer are depicted in Table 1. Each convolutional layer 30 i includes three layers, namely a convolutional layer 30-1, a batch normalization layer, and an activation (ReLU) layer. In computer vision applications, such as object classification, image data is presented to the convolutional layer 30 1 , and the Softmax layer 60 outputs a predicted label or class.

[0038] The convolutional layer 30 1 has a kernel size of 3x3, 64 filters, and a stride of 2. The max pooling layer 30-2 1 has a size of 3x3 and a stride of 2. The convolutional layer 30 2 , 30 3 , 30 4 , and 30 5 has a kernel size of 3x3, 64 filters, and a stride of 1.6 with kernel size 3x3, 128 filters, and stride 2, while convolutional layer 30 7 , 30 8 , and 30 9 has kernel size 3x3, 256 filters, and stride 1. Convolutional layer 30 10 with kernel size 3x3, 128 filters, and stride 2, while convolutional layer 30 11 , 30 12 , and 30 13 has kernel size 3x3, 256 filters, and stride 1. Convolutional layer 30 14 with kernel size 3x3, 512 filters, and stride 2, while convolutional layer 30 15 , 30 16 , and 30 17 has kernel size 3x3, 512 filters, and stride 1.

[0039] CNN 14 not only sequentially connects layers (indicated by straight arrow connections), but also periodically adds the output of a batch normalization layer of one convolutional layer 30 i to the input of a subsequent convolutional layer 30 i+1 (indicated by curved arrow connections).

[0040] For direct connections, each activation layer generates output feature maps that are provided as input feature maps to the next convolutional layer. Typically, these output feature maps are represented by a three-dimensional matrix or feature tensor that defines the feature map data by height (H), width (W), and a number of channels (C). The dimensionality of the feature tensor is defined as height x width (i.e., H x W).

[0041] For additional connections, CNN 14 groups constituent layers of two consecutive convolutional layers 30 i and 30 i+1 into a“residual block” (as depicted by residual block 70). In addition to the constituent layers of convolutional layers 30 i and 30 i+1 , each residual block 70 also includes an additional element 72 to create a“skip connection” between consecutive convolutional layers 30 i and 30 i+1 . The input to a residual block 70 is denoted by“x”. In the absence of a skip connection, the required underlying mapping to be learned during training is the residual mapping“f(x)”. However, in the presence of a skip connection, the required underlying mapping to be learned during training is the residual mapping“f(x) - x”, which is easy to learn since it identifies the mapping f(x) = x. The skip connection improves the training of CNN 14 by mitigating the vanishing gradient problem, and generally allows information to flow more freely between subsequent convolutional layers.

[0042] In the depicted embodiment in Figure 2B , the image data provided to convolutional layer 30 1 has a size of 224 x 224 x 3 elements. Each element is a pixel having the same bit length (e.g., 8 bits). Convolutional layer 30 1 outputs a feature tensor having a size of 112 x 112 x 64 elements and a scale of 112 x 112. Max pooling layer 30-2 1 outputs a feature tensor having a size of 56 x 56 x 64 elements and a scale of 56 x 56. Convolutional layer 30 2 , 30 3 , 30 4 , and 30 5 outputs a feature tensor having a size of 56 x 56 x 64 elements and a scale of 56 x 56. Convolutional layer 30 6 , 30 7 , 30 8 , and 30 9 outputs a feature tensor having a size of 28 x 28 x 128 elements and a scale of 28 x 28. Convolutional layer 30 10 , 30 11 , 30 12 , and 30 13 outputs a feature tensor having a size of 14 x 14 x 256 elements and a scale of 14 x 14. Convolutional layer 30 14 , 30 15 , 30 16 , and 30 17 outputs a feature tensor having a size of 7 x 7 x 512 elements and a scale of 7 x 7. Average pooling layer 30-2 2 outputs a feature tensor having a size of 1 x 1 x 512 elements, classification (fully connected) layer 50 has 512 x 1000 connections and outputs 1,000 elements, and Softmax layer 60 outputs 1,000 elements. Other configurations of CNN 14 are also contemplated.

[0043] Figure 3 An object detection network 100 according to embodiments of the disclosure is depicted.

[0044] Object detection network 100 includes a BFE module 110, an FD module 120, and a FA module 170. BFE module 110 receives image data and generates feature tensors 112, 114, and 116. FD module 120 receives feature tensors 112, 114, and 116 and generates intermediate tensors 122, 124, and 126. FA module 170 receives intermediate tensors 122, 124, and 126 and generates an output tensor 172.

[0045] Generally, the BFE module 110 includes an ML model, such as, for example, an ANN, a DNN, a CNN, an RNN, a Bayesian network, etc., that performs object classification on image data. In Figure 3 In the depicted embodiment, the BFE module 110 incorporates a CNN 14, namely a ResNet model. In this embodiment, the last three layers of the CNN 14, namely the average pooling layer 30-2 2 , the classification layer 50, and the Softmax layer 60, have been omitted or deactivated, as indicated by the ellipses. In other embodiments, the last three layers of the CNN 14 can be included or activated in order to provide a nominal predicted class for the image data.

[0046] Feature tensors 112, 114, and 116 are extracted from the ResNet model at three different layers having three different scales and are provided to the FD module 120. In this embodiment, the feature tensor 112 has a size of 7 x 7 x 512 and a scale of 7 x 7, the feature tensor 114 has a size of 14 x 14 x 256 and a scale of 14 x 14, and the feature tensor 116 has a size of 28 x 28 x 128 and a scale of 28 x 28.

[0047] After the activation layer of the convolutional layer 30 14 , the convolutional layer 30 15 , the convolutional layer 30 16 , or the convolutional layer 30 17 , the feature tensor 112 can be extracted from the CNN 14. Similarly, after the activation layer of the convolutional layer 30 10 , the convolutional layer 30 11 , the convolutional layer 30 12 , or the convolutional layer 30 13 , the feature tensor 114 can be extracted from the CNN 14, and after the activation layer of the convolutional layer 30 6 , the convolutional layer 30 7 , the convolutional layer 30 8 , or the convolutional layer 30 9 , the feature tensor 116 can be extracted from the CNN 14.

[0048] In many embodiments, each feature tensor is extracted after the activation layer of the last convolutional layer listed above, namely the convolutional layer 30 9 for the feature tensor 116, the convolutional layer 30 13 for the feature tensor 114, and the convolutional layer 30 17 for the feature tensor 112. The present disclosure can also accommodate other feature tensor sizes and scales.

[0049] Generally, the FD module 120 includes an object detector that performs multiple detections per pass. In Figure 3In the depicted embodiment, the FD module 120 includes a CNN-based object detector having three detector heads (i.e., detector head 130, detector head 140, and detector head 150) and a packet dropout mask generation module 160. The detector head 130 receives the feature tensor 112 from the BFE module 110, the detector head 140 receives the feature tensor 114 from the BFE module 110, and the detector head 150 receives the feature tensor 116 from the BFE module 110.

[0050] The detector head 130 includes a convolution module 133, a mask module 134, a convolution module 135, a mask module 136, a convolution module 137, and an intermediate tensor generation module 138. The convolution module 133 receives the feature tensor 112 from the BFE module 110. The mask module 134 is coupled to the convolution module 133 and the packet dropout mask generation module 160 and receives the updated feature tensor 112 from the convolution module 133 and a random packet dropout mask from the packet dropout mask generation module 160. The convolution module 135 is coupled to the mask module 134, includes a convolution layer, and receives the masked feature tensor 112 from the mask module 134.

[0051] The mask module 136 is coupled to the convolution module 135 and the packet dropout mask generation module 160 and receives the updated feature tensor 112 from the convolution module 135 and a random packet dropout mask from the packet dropout mask generation module 160. The convolution module 137 is coupled to the mask module 136, includes a convolution layer, and receives the masked feature tensor 112 from the mask module 136. The convolution module 137 is coupled to and receives the updated feature tensor 112 from the convolution module 137.

[0052] The intermediate tensor generation module 138 generates an intermediate tensor 122 that is associated with the feature tensor 112 and includes, among other things, scale-dependent bounding box spatial data that includes mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-dependent bounding boxes for an object, as described below. In one embodiment, the intermediate tensor 122 can be defined by Equation 1:

[0053] M x H / 4 x W / 4 x A x [K1, K2, K 11 ,p(obj), {C 22 ,p(obj), {C i}] Equation 1

[0054] where M is the number of detections passed, H / 4 x W / 4 is the scale of the feature tensor 112 (e.g., 7 x 7), A is the number of bounding boxes (or anchor bounding boxes), K1is the mean first angular coordinate of the bounding box (i.e., x1, y1), K2is the mean second angular coordinate of the bounding box (i.e., x2, y2), K 11is the bounding box covariance matrix of the first angular coordinate, K 22 is the bounding box covariance matrix of the second angular coordinate, p(obj) is the objective score, and {C i} is the class probability. K 11 and K 22 is given by Equation 2:

[0055]

[0056] where the sigma values are the variance and covariance of the respective coordinates.

[0057] The number of bounding boxes A is typically at least two - one horizontal rectangle (e.g., for objects such as cars, buses, trucks, etc.) and one vertical rectangle (e.g., for objects such as pedestrians, bicycles, etc.).

[0058] The detector head 140 includes an up-sampler module 141, a concatenation module 142, a convolution module 143, a mask module 144, a convolution module 145, a mask module 146, a convolution module 147, and an intermediate tensor generation module 148. The up-sampler module 141 is coupled to the convolution module 133 and receives the updated feature tensor 112 from the convolution module 133. The concatenation module 142 is coupled to the up-sampler module 141 and receives the feature tensor 114 from the BFE module 110 and the up-scaled feature tensor 112 from the up-sampler module 141. The convolution module 143 is coupled to the concatenation module 142 and receives the concatenated feature tensor 114 from the concatenation module 142.

[0059] The mask module 144 is coupled to the convolution module 143 and the dropout mask generation module 160 and receives the updated feature tensor 114 from the convolution module 143 and the random dropout mask from the dropout mask generation module 160. The convolution module 145 is coupled to the mask module 144, includes a convolution layer, and receives the masked feature tensor 114 from the mask module 144. The mask module 146 is coupled to the convolution module 145 and the dropout mask generation module 160 and receives the updated feature tensor 114 from the convolution module 145 and the random dropout mask from the dropout mask generation module 160. The convolution module 147 is coupled to the mask module 146, includes a convolution layer, and receives the masked feature tensor 114 from the mask module 146. The intermediate tensor generation module 148 is coupled to the convolution module 147 and receives the updated feature tensor 114 from the convolution module 147.

[0060] The intermediate tensor generation module 148 generates an intermediate tensor 124 that is associated with the feature tensor 114 and includes, among other things, scale-dependent bounding box spatial data including mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-dependent bounding boxes for an object, as described below. In one embodiment, the intermediate tensor 124 can be defined by Equation 3:

[0061] M x H / 2 x W / 2 x A x [K1, K2, K 11 , 22 , p(obj), {C i} Equation 3

[0062] where H / 2 x W / 2 is the scale of the feature tensor 114 (e.g., 14 x 14), and the remaining elements are given above.

[0063] The detector head 150 includes an up-sampler module 151, a concatenation module 152, a convolution module 153, a mask module 154, a convolution module 155, a mask module 156, a convolution module 157, and an intermediate tensor generation module 158. The up-sampler module 151 is coupled to the convolution module 143 and receives the updated feature tensor 114 from the convolution module 143. The concatenation module 152 is coupled to the up-sampler module 151 and receives the feature tensor 116 from the BFE module 110 and the up-scaled feature tensor 114 from the up-sampler module 151. The convolution module 153 is coupled to the concatenation module 152 and receives the concatenated feature tensor 116 from the concatenation module 152.

[0064] The mask module 154 is coupled to the convolution module 153 and the dropout mask generation module 160 and receives the updated feature tensor 116 from the convolution module 153 and the random dropout mask from the dropout mask generation module 160. The convolution module 155 is coupled to the mask module 154, includes a convolution layer, and receives the masked feature tensor 116 from the mask module 154. The mask module 156 is coupled to the convolution module 155 and the dropout mask generation module 160 and receives the updated feature tensor 116 from the convolution module 155 and the random dropout mask from the dropout mask generation module 160. The convolution module 157 is coupled to the mask module 156, includes a convolution layer, and receives the masked feature tensor 116 from the mask module 156. The intermediate tensor generation module 158 is coupled to the convolution module 157 and receives the updated feature tensor 116 from the convolution module 157.

[0065] An intermediate tensor generation module 158 generates an intermediate tensor 126 that is associated with the feature tensor 116 and that includes, among other things, scale-dependent bounding box spatial data including mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-dependent bounding boxes for an object, as described below. In one embodiment, the intermediate tensor 126 can be defined by Equation 4:

[0066] M x H x W x A x [K1, K2, K 11 , K 22 , p(obj), {C i}] Equation 4

[0067] where H x W is the scale of the feature tensor 116 (e.g., 28 x 28), and the remaining elements are given above.

[0068] In one embodiment, each convolution module includes a group of layers including a first convolution layer with a kernel size of 1 x 1 and 512 filters, a second convolution layer with a kernel size of 3 x 3 and 1,024 filters, a batch normalization layer, and an activation layer. The convolution modules 133, 143, and 153 include a number N of these groups of layers; in one embodiment, N is equal to 2. The present disclosure accommodates other convolution kernel sizes, filter numbers, and group numbers as well.

[0069] The up-sampler module 141 receives the updated feature tensor 112 from the convolution module 133 and increases the scale of the updated feature tensor 112 to match the scale of the feature tensor 114, e.g., from 7 x 7 to 14 x 14. This can be performed, e.g., by an “unpooling” or average pooling routine. Similarly, the up-sampler module 151 receives the updated feature tensor 114 from the convolution module 143 and increases the scale of the updated feature tensor 114 to match the scale of the feature tensor 116, e.g., from 14 x 14 to 28 x 28.

[0070] The concatenation module 142 concatenates the feature tensor 114 and the up-scaled feature tensor 112 from the up-sampler module 141 to generate a concatenated feature tensor 114. In this embodiment, the concatenated feature tensor 114 has a size of 28 x 28 x 768, which is a result of the concatenation of the feature tensor 114 having a size of 28 x 28 x 256 with the up-scaled feature tensor 112 having a size of 28 x 28 x 512. Similarly, the concatenation module 152 concatenates the feature tensor 116 and the up-scaled feature tensor 114 from the up-sampler module 151 to generate a concatenated feature tensor 116. In this embodiment, the concatenated feature tensor 116 has a size of 56 x 56 x 896, which is a result of the concatenation of the feature tensor 116 having a size of 56 x 56 x 128 with the up-scaled and concatenated feature tensor 114 having a size of 56 x 56 x 768.

[0071] Amplifying and concatenating the feature tensors advantageously extracts more features from deeper layers of the CNN 14 and combines them with features extracted from shallower layers.

[0072] In one embodiment, the packet loss mask generation module 160 includes a Bernoulli mask generator. The Bernoulli mask starts with a random seed, and the number of zeros in the mask depends on the packet loss rate. The Bernoulli mask is then fed to a function, such as, for example, an XOR operation, which then emits a new random bit per loop. After n loops, a new random mask of length n is generated. The mask module within each detector head 130, 140, and 150 receives a different size mask due to different feature tensor scales. Detector head 130 receives a packet loss mask with 49 elements (i.e., 7x7), detector head 140 receives a packet loss mask with 196 elements (i.e., 14x14), and detector head 150 receives a packet loss mask with 784 elements (i.e., 28x28).

[0073] In one embodiment, the FA module 170 includes a Monte Carlo simulation model that processes the intermediate tensors 122, 124, and 126 and generates an output tensor 172 that includes, among other things, scale-invariant bounding box spatial data including mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-invariant bounding boxes of an object, as described below. In one embodiment, the output tensor 172 can be defined by Equation 5:

[0074] P x Q x [K1, K2, K 11 ,K 22 ,{C i}] Equation 5

[0075] where P is the number of grid rows, Q is the number of grid columns (i.e., P-Q equals the number of grid cells); the remaining elements are given above.

[0076] As described above, for each image dataset processed by the BFE module 110, each detector head 130, 140, and 150 receives a different feature tensor 112, 114, and 116, respectively. The object detector then performs multiple passes of detection to generate intermediate tensors 122, 124, and 126, such as, for example, 1,000 passes of detection (i.e., M equals 1,000). The packet drop mask generation module 160 generates a random packet drop mask for each feature tensor 112, 114, and 116 for each pass of detection. During each pass of detection, each detector head 130, 140, and 150 determines a plurality of scale-dependent bounding boxes for objects based on the respective feature tensor 112, 114, and 116 and the respective packet drop mask, and determines scale-dependent bounding box spatial data based on the plurality of scale-dependent bounding boxes. Each detector head can also determine an objectness score and class probabilities.

[0077] Generally, the processing performed by the detector head 130 during a pass of detection can be summarized as follows. The convolution module 133 receives the feature tensor 112 from the BFE module 110 and performs a convolution operation on the feature tensor 112 to update the feature tensor 112. The mask module 134 performs a masking operation on the feature tensor 112 to mask the feature tensor 112. The convolution module 135 performs a convolution operation on the feature tensor 112 to update the feature tensor 112. The mask module 136 performs a masking operation on the feature tensor 112 to mask the feature tensor 112. The convolution module 137 performs a convolution operation on the feature tensor 112 to update the feature tensor 112. Also, the intermediate tensor generation module 138 generates the intermediate tensor 122 associated with the feature tensor 112.

[0078] Figure 4A An image 180 with bounding boxes is depicted in accordance with an embodiment of the present disclosure. The image 180 includes at least one object of interest, namely a bus. Scale-dependent bounding boxes 181, 182, and 183 are depicted, as well as the general locations of K1 and K2 of the intermediate tensor 122. In this example, the intermediate tensor 122 has three anchor bounding boxes (i.e., A equals 3). For clarity, the scale-dependent bounding boxes of the intermediate tensors 124 and 126 generated by the detector heads 140 and 150, respectively, are not depicted.

[0079] The processing performed by the detector head 140 during the detection pass can be summarized as follows. The upsampler module 141 receives the feature tensor 112 from the convolution module 133 and increases the scale of the feature tensor 112 to match the scale of the feature tensor 114. The concatenation module 142 receives the feature tensor 114 from the BFE module 110 and concatenates the scaled feature tensor 112 and the feature tensor 114. The convolution module 143 performs a convolution operation on the feature tensor 114 to update the feature tensor 114. The masking module 144 performs a masking operation on the feature tensor 114 to mask the feature tensor 114. The convolution module 145 performs a convolution operation on the feature tensor 114 to update the feature tensor 114. The masking module 146 performs a masking operation on the feature tensor 114 to mask the feature tensor 114. The convolution module 147 performs a convolution operation on the feature tensor 114 to update the feature tensor 114. Also, the intermediate tensor generation module 148 generates the intermediate tensor 124 associated with the feature tensor 114.

[0080] The processing performed by the detector head 150 during the detection pass can be summarized as follows. The upsampler module 151 receives the feature tensor 114 from the convolution module 143 and increases the scale of the feature tensor 114 to match the scale of the feature tensor 116. The concatenation module 152 receives the feature tensor 116 from the BFE module 110 and concatenates the scaled feature tensor 114 and the feature tensor 116. The convolution module 153 performs a convolution operation on the feature tensor 116 to update the feature tensor 116. The masking module 154 performs a masking operation on the feature tensor 116 to mask the feature tensor 116. The convolution module 155 performs a convolution operation on the feature tensor 116 to update the feature tensor 116. The masking module 156 performs a masking operation on the feature tensor 116 to mask the feature tensor 116. The convolution module 157 performs a convolution operation on the feature tensor 116 to update the feature tensor 116. Also, the intermediate tensor generation module 158 generates the intermediate tensor 126 associated with the feature tensor 116.

[0081] The FA module 170 then processes the intermediate tensors 122, 124, and 126 and generates the output tensor 172 that includes, among other things, scale-invariant bounding box spatial data that includes mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-invariant bounding boxes for the object.

[0082] Figure 4B An image 190 with a grid and bounding boxes according to an embodiment of the disclosure is depicted. The image 190 includes the same object of interest, i.e., a bus. A scale-invariant bounding box 192 is depicted, as well as the general locations of K1 and K2 of the output tensor 172. In this example, the grid contains nine cells, i.e., P equals 3 and Q equals 3.

[0083] Figure 5 A block diagram depicting system 200, in accordance with embodiments of the present disclosure, is depicted.

[0084] System 200 includes computer 202, I / O device 242, and display 252. Computer 202 includes communication bus 210 coupled to one or more processors 220, memory 230, I / O interface 240, display interface 250, one or more communication interfaces 260, and one or more HAs 280. Generally, I / O interface 240 is coupled to I / O device 242 using a wired or wireless connection, display interface 250 is coupled to display 252, and communication interfaces 260 connect to network 262 using wired or wireless connections. In some embodiments, certain components of computer 202 are implemented as a system on a chip (SoC); in other embodiments, computer 202 can be hosted on a traditional printed circuit board, motherboard, or the like.

[0085] In some embodiments, system 200 is an embedded system, in which Figure 5 one or more of the components depicted in FIG. 1 are not present, such as, for example, I / O interface 240, I / O device 242, display interface 250, display 252, and the like. Additionally, certain components, when present, can be optimized based on various design constraints, such as, for example, power, area, and the like, for example, HA 280.

[0086] Communication bus 210 is a communication system that transfers data among processor 220, memory 230, I / O interface 240, display interface 250, communication interfaces 260, HA 280, and other components not depicted in FIG. 1. Power connector 212 is coupled to communication bus 210 and a power supply (not shown). In some embodiments, communication bus 210 is a network on a chip (NoC). Figure 5 Processor 220 includes one or more general- or special-purpose microprocessors that execute instructions to perform control, computing, input / output, and the like, functions of system 200. Processor 220 can include a single integrated circuit, such as a microprocessing device, or multiple integrated circuit devices and / or circuit boards that cooperate to implement the functions of processor 220. Additionally, processor 220 can include multiple processing cores, as depicted in FIG. 1. Generally, system 200 can include one or more processors 220, each containing one or more processing cores and various other modules.

[0087] Figure 5

[0088] ​​In some embodiments, system 200 can include 2 processors 220, each containing multiple processing cores. For example, one processor 220 can be a high-performance processor containing 4“big” processing cores, such as, for example, Arm Cortex-A73, Cortex-A75, Cortex-A76, etc., while the other processor 220 can be a high-efficiency processor containing 4“small” processing cores, such as, for example, Arm Cortex-53, Arm Cortex-55, etc. In this example, the“big” processing cores include a memory management unit (MMU). In other embodiments, system 200 can be an embedded system that includes a single processor 220 with one or more processing cores, such as, for example, Arm Cortex-M cores. In these embodiments, processor 220 typically includes a memory protection unit (MPU).

[0089] In many embodiments, processor 220 can also be configured to execute classification-based machine learning (ML) models, such as, for example, ANNs, DNNs, CNNs, RNNs, SVMs, Naive Bayes, etc., and more specifically, object detection network 100. In these embodiments, processor 220 can provide the same functionality as a hardware accelerator, such as HA 280. For example, system 200 can be an embedded system that does not include HA 280.

[0090] Additionally, processor 220 can execute computer programs or modules stored within memory 230, such as operating system 232, software modules 234, etc. For example, software modules 234 can include autonomous vehicle applications, robotic applications, such as, for example, robots that perform surgical procedures, work with people in a collaborative environment, etc., that can include classification networks, such as, for example, ANNs, CNNs, RNNs, BNNs, SVMs, decision trees, Bayesian networks, Naive Bayes, etc., and more specifically, object detection network 100.

[0091] Generally, storage elements or memory 230 store instructions and data for execution by processor 220. Memory 230 can include various non-transitory computer-readable media that are accessible by processor 220. In various embodiments, memory 230 can include volatile and non-volatile media, non-removable media, and / or removable media. For example, memory 230 can include any combination of random access memory (RAM), DRAM, SRAM, ROM, flash memory, cache memory, and / or any other type of non-transitory computer-readable media.

[0092] Memory 230 includes various means for retrieving, presenting, modifying, and storing data. For example, memory 230 stores software modules that, when executed by processor 220, provide functionality. The software modules include operating system 232, which provides operating system functionality for system 200. Software modules 234 provide various functionality, such as image classification using CNNs, etc. Data 236 can include data associated with operating system 232, software modules 234, etc.

[0093] I / O interface 240 is configured to transmit and / or receive data from I / O device 242. I / O interface 240 enables a connection between processor 220 and I / O device 242 by encoding data to be transmitted from processor 220 to I / O device 242 and decoding data received from I / O device 242 for processor 220. Generally, data can be transmitted through wired and / or wireless connections. For example, I / O interface 240 can include one or more wired communication interfaces (such as USB, Ethernet, etc.) and / or one or more wireless communication interfaces (coupled to one or more antennas, such as WiFi, Bluetooth, cellular, etc.).

[0094] Generally, I / O device 242 provides input to and / or output from system 200. As described above, I / O device 242 is operatively connected to system 200 using wired and / or wireless connections. I / O device 242 can include a local processor coupled to a communication interface configured to communicate with system 200 using wired and / or wireless connections. For example, I / O device 242 can include a keyboard, mouse, touchpad, joystick, etc., sensors, actuators, etc.

[0095] Display interface 250 is configured to transmit image data from system 200 to monitor or display 252.

[0096] Communication interface 260 is configured to transmit data to and from network 262 using one or more wired and / or wireless connections. Network 262 can include one or more local area networks, wide area networks, the Internet, etc., which can execute various network protocols, such as, for example, wired and / or wireless Ethernet, Bluetooth, etc. Network 262 can also include various combinations of wired and / or wireless physical layers, such as, for example, copper or coaxial cable networks, fiber optic networks, Bluetooth wireless networks, WiFi wireless networks, CDMA, FDMA, and TDMA cellular wireless networks, etc.

[0097] The HA 280 is configured to, among other things, execute a classification network, such as, for example, an ANN, a CNN, an object detection network 100, etc., to support various applications embodied by the software modules 234. Generally, the HA 280 includes one or more processors, co-processors, processing engines (PEs), computing engines (CEs), etc., such as, for example, CPUs, GPUs, NPUs (e.g., ARM ML processors), DSPs, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), controllers, microcontrollers, matrix multiplier circuits, MAC arrays, etc. The HA 280 also includes a communication bus interface, as well as non-volatile and / or volatile memory, such as, for example, ROM, flash memory, SRAM, DRAM, etc.

[0098] In many embodiments, the HA 280 receives the ANN model and weights from the memory 230 over the communication bus 210 for storage in local volatile memory (e.g., SRAM, DRAM, etc.). In other embodiments, the HA 280 receives a portion of the ANN model and weights from the memory 230 over the communication bus 210. In these embodiments, the HA 280 determines the instructions required to execute the ANN model or the ANN model portion. In other embodiments, the ANN model (or ANN model portion) includes only the instructions required to execute the ANN model (or ANN model portion). In these embodiments, the processor 220 determines the instructions required to execute the ANN model, or the processor 220 divides the ANN model into ANN model portions and then determines the instructions required to execute each ANN model portion. The instructions are then provided to the HA 280 as the ANN model or ANN model portions.

[0099] In other embodiments, the HA 280 can store the ANN model, instructions, and weights in non-volatile memory. In some embodiments, the ANN model can be implemented directly in hardware using DSPs, FPGAs, ASICs, controllers, microcontrollers, adder circuits, multiplier circuits, MAC circuits, etc. Generally, the HA 280 receives input data from the memory 230 over the communication bus 210 and transmits output data to the memory 230 over the communication bus 210. In some embodiments, the input data can be associated with a layer (or a portion of a layer) of the ANN model, and the output data from that layer (or that portion of a layer) can be transmitted to the memory 230 over the communication bus 210.

[0100] For example, ARM ML processors support various ANNs, CNNs, RNNs, etc. for classification, object detection, image enhancement, speech recognition, and natural language understanding. ARM ML processors include a control unit, a direct memory access (DMA) engine, local memory, and 16 CEs. Each CE includes, among other things, a MAC engine that performs convolution operations, a programmable layer engine (PLE), local SRAM, a weight decoder, a control unit, a direct memory access (DMA) engine, etc. Each MAC engine accumulates up to eight 16-wide dot products. Generally, the PLE performs non-convolutional operations such as, for example, pooling operations, ReLU activations, etc. Each CE receives input feature maps (IFMs) and weight sets through the NoC and stores them in local SRAM. The MAC engine and the PLE process the IFMs to generate output feature maps (OFMs) that are also stored in local SRAM before being transmitted through the NoC.

[0101] Figure 6 A flowchart 400 representing functionality associated with detecting objects is depicted in accordance with an embodiment of the present disclosure.

[0102] In this embodiment, the functionality at 410 and 420 is performed at the BFE module 110, the functionality at 430, 440, and 450 is performed at the FD module 120, and the functionality at 460, 470, and 480 is performed at the FA module 170.

[0103] At 410, image data is received. The image data includes at least one object.

[0104] At 420, a plurality of feature tensors is generated based on the image data. Each feature tensor has a different scale.

[0105] At 430, the feature tensors from the BFE module are received.

[0106] At 440, a plurality of detection passes is performed. Each detection pass includes: applying a random dropout mask to the feature tensors; and for each feature tensor, determining scale-dependent bounding box spatial data including mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-dependent bounding boxes for the object.

[0107] At 450, an intermediate tensor is generated associated with each feature tensor. Each intermediate tensor includes the scale-dependent bounding box spatial data for each detection pass for the associated feature tensor.

[0108] At 460, the intermediate tensors from the FD module are received.

[0109] At 470, scale-independent bounding box space data is determined based on the intermediate tensors. The scale-independent bounding box space data includes mean angular coordinates of a scale-independent bounding box for the object and a covariance matrix of the angular coordinates.

[0110] At 480, an output tensor is generated. The output tensor includes the scale-independent bounding box space data.

[0111] The implementations described herein can be combined.

[0112] In one implementation, a hardware accelerator for object detection includes a backbone feature extraction (BFE) module, a feature detection (FD) module, and a filtering and averaging (FA) module. The BFE module is configured to receive image data including at least one object, and generate a plurality of feature tensors based on the image data, each feature tensor having a different scale. The FD module is configured to: receive the feature tensors from the BFE module; perform a plurality of detection passes, each detection pass including: applying a random dropout mask to the feature tensors, and for each feature tensor, determining scale-dependent bounding box space data including mean angular coordinates of a plurality of scale-dependent bounding boxes for the object and a covariance matrix of the angular coordinates; and generate intermediate tensors associated with each feature tensor, each intermediate tensor including the scale-dependent bounding box space data for each detection pass of the associated feature tensor. The FA module is configured to: receive the intermediate tensors from the FD module; determine scale-independent bounding box space data based on the intermediate tensors, the scale-independent bounding box space data including mean angular coordinates of a scale-independent bounding box for the object and a covariance matrix of the angular coordinates; and generate an output tensor including the scale-independent bounding box space data.

[0113] In another implementation of the hardware accelerator, each feature tensor includes feature map data defined by a height, a width, and a plurality of channels, and the scale of the feature tensor is defined as the height • the width.

[0114] In another implementation of the hardware accelerator, the FD module includes: a dropout mask generation module configured to generate, for each detection pass, a random dropout mask for each feature tensor; and a plurality of detector heads, each detector head including a plurality of convolution modules and at least one dropout mask application module, each detector head configured to receive a different feature tensor, and for each detection pass, determine a plurality of scale-dependent bounding boxes for the object based on the feature tensor and the corresponding dropout mask, and determine the scale-dependent bounding box space data based on the plurality of scale-dependent bounding boxes.

[0115] In another implementation of the hardware accelerator, each detector head is configured to determine an objectness score and a class probability.

[0116] In another implementation of the hardware accelerator, the FD module includes a first detector head and a second detector head, the first detector head configured to receive a first feature tensor having a first scale, and the second detector head configured to receive a second feature tensor having a second scale greater than the first scale; the second detector head includes an up-sampler module configured to receive the first feature tensor from a first convolutional module of the first detector head and to increase the scale of the first feature tensor to match the scale of the second feature tensor, and a concatenation module configured to concatenate the up-scaled first feature tensor and the second feature tensor and to provide the concatenated second feature tensor to the first convolutional module of the second detector head; and the concatenated second feature tensor has a total number of channels equal to the number of channels of the up-scaled first feature tensor plus the number of channels of the second feature tensor.

[0117] In another implementation of the hardware accelerator, the FD module includes a third detector head configured to receive a third feature tensor having a third scale greater than the second scale; the third detector head includes an up-sampler module configured to receive the second feature tensor from a first convolutional module of the second detector head and to increase the scale of the second feature tensor to match the scale of the third feature tensor, and a concatenation module configured to concatenate the up-scaled second feature tensor and the third feature tensor and to provide the concatenated third feature tensor to the first convolutional module of the third detector head; and the concatenated third feature tensor has a total number of channels equal to the number of channels of the up-scaled second feature tensor plus the number of channels of the third feature tensor.

[0118] In another implementation of the hardware accelerator, the first detector head further includes a first mask module coupled to the first convolutional module and the packet loss mask generation module, a second convolutional module coupled to the first mask module, a second mask module coupled to the second convolutional module and the packet loss mask generation module, and a third convolutional module coupled to the second mask module; the second detector head further includes a first mask module coupled to the first convolutional module and the packet loss mask generation module, a second convolutional module coupled to the first mask module, a second mask module coupled to the second convolutional module and the packet loss mask generation module, and a third convolutional module coupled to the second mask module; and the third detector head further includes a first mask module coupled to the first convolutional module and the packet loss mask generation module, a second convolutional module coupled to the first mask module, a second mask module coupled to the second convolutional module and the packet loss mask generation module, and a third convolutional module coupled to the second mask module.

[0119] In another implementation of the hardware accelerator, the BFE module includes a convolutional neural network (CNN) model.

[0120] In another implementation of the hardware accelerator, the CNN model is a Residual Network (ResNet) model.

[0121] In another implementation of the hardware accelerator, the packet drop mask generation module includes a Bernoulli mask generator and the FA module includes a Monte Carlo simulation model.

[0122] In one implementation, a method for detecting objects includes: at a backbone feature extraction (BFE) module, receiving image data including at least one object; and generating a plurality of feature tensors based on the image data, each feature tensor having a different scale; at a feature detection (FD) module, receiving the feature tensors from the BFE module; performing a plurality of detection passes, each detection pass including: applying a random packet drop mask to the feature tensors, determining, for each feature tensor, scale-dependent bounding box spatial data including mean angular coordinates and a covariance matrix of the angular coordinates for a plurality of scale-dependent bounding boxes for the object, and generating an intermediate tensor associated with each feature tensor, each intermediate tensor including the scale-dependent bounding box spatial data for each detection pass of the associated feature tensor; at a filtering and averaging (FA) module, receiving the intermediate tensors from the FD module; determining scale-independent bounding box spatial data based on the intermediate tensors, the scale-independent bounding box spatial data including mean angular coordinates and a covariance matrix of the angular coordinates for a scale-independent bounding box for the object, and generating an output tensor including the scale-independent bounding box spatial data.

[0123] In another implementation of the method, each feature tensor includes feature map data defined by a height, a width, and a plurality of channels, and the scale of the feature tensor is defined as the height • width.

[0124] In another implementation of the method, the method further includes: at the FD module, generating, by a packet drop mask generation module, a random packet drop mask for each feature tensor for each detection pass; and receiving, at a plurality of detector heads, different feature tensors, each detector head including a plurality of convolution modules and at least one packet drop mask application module, the performing each detection pass including: at each detector head, determining, based on the feature tensor and the respective packet drop mask, a plurality of scale-dependent bounding boxes for the object, and at each detector head, determining the scale-dependent bounding box spatial data based on the plurality of scale-dependent bounding boxes.

[0125] In another implementation of the method, the performing each detection pass includes, at each detector head, determining an objectness score and a class probability.

[0126] In another embodiment of the method, the FD module includes a first detector head and a second detector head, the second detector head including an up-sampler module and a concatenation module, and the method further includes: at the first detector head, receiving a first feature tensor having a first scale; at the second detector head, receiving a second feature tensor having a second scale greater than the first scale, receiving the first feature tensor from the first convolutional module of the first detector head, increasing the scale of the first feature tensor at the up-sampler module to match the scale of the second feature tensor, concatenating the up-scaled first feature tensor and the second feature tensor at the concatenation module, and providing the concatenated second feature tensor to the first convolutional module of the second detector head, the concatenated second feature tensor having a total number of channels equal to the number of channels of the up-scaled first feature tensor plus the number of channels of the second feature tensor.

[0127] In another embodiment of the method, the FD module includes a third detector head, the third detector head including an up-sampler module and a concatenation module, and the method further includes: at the third detector head, receiving a third feature tensor having a third scale greater than the second scale, receiving the second feature tensor from the first convolutional module of the second detector head, increasing the scale of the second feature tensor at the up-sampler module to match the scale of the third feature tensor, concatenating the up-scaled second feature tensor and the third feature tensor at the concatenation module, and providing the concatenated third feature tensor to the first convolutional module of the third detector head, the concatenated third feature tensor having a total number of channels equal to the number of channels of the up-scaled second feature tensor plus the number of channels of the third feature tensor.

[0128] In another embodiment of the method, the method further comprises: at the first detector head, performing a convolution operation on the first feature tensor by a first convolution module to update the first feature tensor, performing a masking operation on the first feature tensor by a first mask module to mask the first feature tensor, performing a convolution operation on the first feature tensor by a second convolution module to update the first feature tensor, performing a masking operation on the first feature tensor by a second mask module to mask the first feature tensor, performing a convolution operation on the first feature tensor by a third convolution module to update the first feature tensor, and generating a first intermediate tensor based on the first feature tensor by an intermediate tensor generation module; at the second detector head, performing a convolution operation on the second feature tensor by the first convolution module to update the second feature tensor, performing a masking operation on the second feature tensor by the first mask module to mask the second feature tensor, performing a convolution operation on the second feature tensor by the second convolution module to update the second feature tensor, performing a masking operation on the second feature tensor by the second mask module to mask the second feature tensor, performing a convolution operation on the second feature tensor by the third convolution module to update the second feature tensor, and generating a second intermediate tensor based on the second feature tensor by the intermediate tensor generation module; and at the third detector head, performing a convolution operation on the third feature tensor by the first convolution module to update the third feature tensor, performing a masking operation on the third feature tensor by the first mask module to mask the third feature tensor, performing a convolution operation on the third feature tensor by the second convolution module to update the third feature tensor, performing a masking operation on the third feature tensor by the second mask module to mask the third feature tensor, performing a convolution operation on the third feature tensor by the third convolution module to update the third feature tensor, and generating a third intermediate tensor based on the third feature tensor by the intermediate tensor generation module.

[0129] In another embodiment of the method, the BFE module comprises a convolutional neural network (CNN) model.

[0130] In another embodiment of the method, the CNN model is a residual network (ResNet) model.

[0131] In another embodiment of the method, the random packet loss mask is a Bernoulli mask, and the FA module comprises a Monte Carlo simulation model.

[0132] While specific embodiments of the present disclosure can have numerous adaptations, the specific embodiments illustrated in the drawings, and described in the specification are presented by way of example only and are not intended to limit the scope of the disclosure to the specific embodiments presented. In the above description, similar reference characters have been used to describe similar features in the several views of the drawings.

[0133] In this document, relational terms such as first and second, top and bottom, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "includes," "including," "contains," "containing," "has," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, includes, contains, has, or the like a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0134] References in this document to "one embodiment," "certain embodiments," "an embodiment," "the embodiments," "the implementation," "aspects," or similar terms, indicate that the alternative named

[0135] As used herein, the term "or" is to be interpreted as inclusive or meaning any one or any combination of the listed items. Thus, "A, B, or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." Only when a combination of elements, functions, steps, or acts is inherently mutually exclusive is an exception to this definition. Additionally, use of the term "set" means "one or more," unless otherwise indicated. Further, the terms "set," "group," "composition," "assembly," or the like, are intended to mean one or more, unless otherwise indicated. Additionally, grammar variants of the term "include," such as "comprise" and "comprising," are intended to be interpreted the same as the term "include." Unless specifically stated otherwise, the term "about" preceding a noun or noun phrase refers to a range of values including the value itself. The term "coupled" is used herein to express a relationship between or among two or more elements, components, or steps, in which the (a) element or elements are directly or indirectly connected or linked to the other (a) element or elements, and / or (b) where the element or elements are mutually coordinated so as to work together. The term "coupled" does not necessarily denote a direct connection or linkage between elements.

[0136] Unless otherwise indicated, expressions of values of ranges of values herein are not intended to be limiting, but rather to refer to any and all values falling within the range, and each individual value within such range is incorporated herein as if it were individually stated. The words "about," "approximately," and the like, when used with respect to a numerical value, should be understood to indicate a deviation of plus or minus 10% from the value, as would be understood by one of ordinary skill in the art, which operates satisfactorily for intended purposes. Values and / or ranges of values are provided herein only as examples, and do not constitute a limitation on the scope of the described embodiments. The use of any and all examples, or exemplary language (e.g., "such as," "for instance," "like," etc.) provided herein, is intended merely to better illuminate embodiments, and does not pose a limitation on the scope of the embodiments. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the embodiments.

[0137] To simplify and clarify the description, reference numerals can be repeated in the drawings to identify corresponding or analogous elements among the several embodiments. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. The embodiments can be practiced without the specific details (e.g., quantities, materials, etc.) set forth in this description. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the described embodiments. The description is not to be considered as limited to the scope of the embodiments described herein.

[0138] In the following description, it is to be understood that terms such as "first," "second," "top," "bottom," "front," "back," "position," "lower," "upper," and the like are used only to facilitate discussion, and do not limit the terms to the relative position of the items described. Also, the terms "set," "component," "system," and the like are used herein to refer to one or more items, unless otherwise indicated by the context.

[0139] Many features and advantages of the present disclosure are apparent from the detailed specification, and thus, it is intended that the appended claims encompass within their scope all such features and advantages of the present disclosure. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the present disclosure to the exact construction and operation described and illustrated, and accordingly, all suitable modifications and equivalents can be resorted to, falling within the scope of the present disclosure.

Claims

1. A hardware accelerator for object detection, the hardware accelerator comprising: The backbone feature extraction (BFE) module is configured as follows: Receive image data including at least one object, and Multiple feature tensors are generated based on the image data, each feature tensor having a different scale; Feature detection FD module, the FD module is configured as follows: Receive the feature tensor from the BFE module. The test process is repeated multiple times, and each test process includes: Apply a random packet loss mask to the feature tensor, and For each feature tensor, scale-dependent bounding box spatial data is determined, which includes the average angular coordinates of multiple scale-dependent bounding boxes for the object and the covariance matrix of the angular coordinates, and Generate intermediate tensors associated with each feature tensor, each intermediate tensor including the scale-dependent bounding box spatial data passed through for each detection of the associated feature tensor; and The filtering and averaging FA module is configured as follows: Receive the intermediate tensor from the FD module. The scale-independent bounding box spatial data is determined based on the intermediate tensor. This scale-independent bounding box spatial data includes the mean angular coordinates of the scale-independent bounding boxes for the object and the covariance matrix of the angular coordinates. Generate an output tensor that includes the scale-independent bounding box spatial data.

2. The hardware accelerator of claim 1, wherein each feature tensor includes feature map data defined by height, width and multiple channels, and the scale of the feature tensor is defined as height × width.

3. The hardware accelerator according to claim 2, wherein the FD module comprises: A packet loss mask generation module is configured to generate a random packet loss mask for each feature tensor for each detection pass. and Multiple detector heads, each including multiple convolutional modules and at least one packet loss masking application module, each detector head is configured as follows: Receives tensors with different features, and For each detection pass, the multiple scale-dependent bounding boxes for the object are determined based on the feature tensor and the corresponding packet loss mask, and the scale-dependent bounding box spatial data is determined based on the multiple scale-dependent bounding boxes.

4. The hardware accelerator of claim 3, wherein each detector head is configured to determine an objective score and a class probability.

5. The hardware accelerator according to claim 3, wherein: The FD module includes a first detector head and a second detector head. The first detector head is configured to receive a first feature tensor having a first scale, and the second detector head is configured to receive a second feature tensor having a second scale greater than the first scale. The second detector head includes an upsampler module and a cascade module. The upsampler module is configured to receive a first feature tensor from a first convolutional module of the first detector head and to increase the scale of the first feature tensor to match the scale of the second feature tensor. The cascade module is configured to cascade the first feature tensor and the second feature tensor with the scale increased and to provide the cascaded second feature tensor to the first convolutional module of the second detector head. The total number of channels in the cascaded second feature tensor is equal to the number of channels in the first feature tensor increased by the scale plus the number of channels in the second feature tensor.

6. The hardware accelerator according to claim 5, wherein: The FD module includes a third detector head configured to receive a third feature tensor having a third scale greater than the second scale. The third detector head includes an upsampler module and a cascade module. The upsampler module is configured to receive a second feature tensor from a first convolutional module of the second detector head and to increase the scale of the second feature tensor to match the scale of the third feature tensor. The cascade module is configured to cascade the second feature tensor with the increased scale and the third feature tensor, and to provide the cascaded third feature tensor to the first convolutional module of the third detector head. The total number of channels in the cascaded third feature tensor is equal to the number of channels in the second feature tensor plus the number of channels in the third feature tensor.

7. The hardware accelerator according to claim 6, wherein: The first detector head further includes a first mask module coupled to the first convolution module and the packet loss mask generation module, a second convolution module coupled to the first mask module, a second mask module coupled to the second convolution module and the packet loss mask generation module, and a third convolution module coupled to the second mask module; The second detector head further includes a first mask module coupled to the first convolutional module and the packet loss mask generation module, a second convolutional module coupled to the first mask module, a second mask module coupled to the second convolutional module and the packet loss mask generation module, and a third convolutional module coupled to the second mask module; and The third detector head further includes a first mask module coupled to the first convolution module and the packet loss mask generation module, a second convolution module coupled to the first mask module, a second mask module coupled to the second convolution module and the packet loss mask generation module, and a third convolution module coupled to the second mask module.

8. The hardware accelerator of claim 1, wherein the BFE module comprises a convolutional neural network (CNN) model.

9. The hardware accelerator according to claim 8, wherein the CNN model is a ResNet residual network model.

10. The hardware accelerator of claim 1, wherein the packet loss mask generation module includes a Bernoulli mask generator, and the FA module includes a Monte Carlo simulation model.

11. A method for detecting an object, the method comprising: At the BFE (Branch Feature Extraction) module: Receive image data including at least one object; Multiple feature tensors are generated based on the image data, each feature tensor having a different scale; At the feature detection FD module: Receive the feature tensor from the BFE module; The test process is repeated multiple times, and each test process includes: Apply a random packet loss mask to the feature tensor; For each feature tensor, determine scale-dependent bounding box spatial data, which includes the average angular coordinates of multiple scale-dependent bounding boxes for the object and the covariance matrix of the angular coordinates; Generate intermediate tensors associated with each feature tensor, each intermediate tensor including the scale-dependent bounding box spatial data passed through for each detection of the associated feature tensor; At the filtering and averaging FA module: Receive the intermediate tensor from the FD module; The scale-independent bounding box spatial data is determined based on the intermediate tensor, wherein the scale-independent bounding box spatial data includes the mean angular coordinates of the scale-independent bounding boxes for the object and the covariance matrix of the angular coordinates; and Generate an output tensor that includes the scale-independent bounding box spatial data.

12. The method of claim 11, wherein each feature tensor includes feature map data defined by height, width and multiple channels, and the scale of the feature tensor is defined as height × width.

13. The method of claim 12, further comprising, at the FD module: The packet loss mask generation module generates a random packet loss mask for each feature tensor for each detection; and Different feature tensors are received at multiple detector heads, each detector head including multiple convolutional modules and at least one packet loss masking application module. Each detection process includes: At each detector head, the plurality of scale-dependent bounding boxes for the object are determined based on the feature tensor and the corresponding packet loss mask. At each detector head, the scale-dependent bounding box spatial data is determined based on the plurality of scale-dependent bounding boxes.

14. The method of claim 13, wherein performing each detection pass includes determining an objective score and a class probability at each detector head.

15. The method of claim 13, wherein the FD module includes a first detector head and a second detector head, the second detector head including an upsampler module and a cascade module, and the method further includes: At the first detector head: Receive a first feature tensor with a first scale; At the second detector head: Receive a second feature tensor with a second scale that is larger than the first scale; Receive the first feature tensor from the first convolution module of the first detector head; The scale of the first feature tensor is increased at the upsampler module to match the scale of the second feature tensor; The first feature tensor and the second feature tensor of the scale are cascaded at the cascaded module; as well as The cascaded second feature tensor is provided to the first convolution module of the second detector head. The total number of channels in the cascaded second feature tensor is equal to the number of channels in the first feature tensor increased by the scale plus the number of channels in the second feature tensor.

16. The method of claim 15, wherein the FD module includes a third detector head, the third detector head including an upsampler module and a cascade module, and the method further includes: At the third detector head: Receive a third feature tensor with a third scale that is larger than the second scale; Receive the second feature tensor from the first convolutional module of the second detector head; The scale of the second feature tensor is increased at the upsampler module to match the scale of the third feature tensor; The second feature tensor and the third feature tensor of the aforementioned scale are cascaded at the cascaded module. as well as The cascaded third feature tensor is provided to the first convolution module of the third detector head. The total number of channels in the cascaded third feature tensor is equal to the number of channels in the second feature tensor plus the number of channels in the third feature tensor.

17. The method according to claim 16, further comprising: At the first detector head: The first convolution module performs a convolution operation on the first feature tensor to update the first feature tensor; The first masking module performs a masking operation on the first feature tensor to mask the first feature tensor; The second convolution module performs a convolution operation on the first feature tensor to update the first feature tensor; The second masking module performs a masking operation on the first feature tensor to mask the first feature tensor. The third convolution module performs a convolution operation on the first feature tensor to update the first feature tensor; The intermediate tensor generation module generates a first intermediate tensor based on the first feature tensor; At the second detector head: The first convolution module performs a convolution operation on the second feature tensor to update the second feature tensor; The first masking module performs a masking operation on the second feature tensor to mask the second feature tensor; The second convolution module performs a convolution operation on the second feature tensor to update the second feature tensor; The second masking module performs a masking operation on the second feature tensor to mask the second feature tensor. The second feature tensor is updated by performing a convolution operation on it by the third convolution module; The intermediate tensor generation module generates a second intermediate tensor based on the second feature tensor; At the third detector head: The first convolution module performs a convolution operation on the third feature tensor to update the third feature tensor; The first masking module performs a masking operation on the third feature tensor to mask the third feature tensor; The third feature tensor is updated by performing a convolution operation on the third feature tensor by the second convolution module. The second masking module performs a masking operation on the third feature tensor to mask the third feature tensor; The third convolution module performs a convolution operation on the third feature tensor to update the third feature tensor; as well as The intermediate tensor generation module generates a third intermediate tensor based on the third feature tensor.

18. The method of claim 11, wherein the BFE module comprises a convolutional neural network (CNN) model.

19. The method of claim 18, wherein the CNN model is a ResNet residual network model.

20. The method of claim 11, wherein the random packet loss mask is a Bernoulli mask, and the FA module comprises a Monte Carlo simulation model.

Images