Deep learning system for cuboid detection

Abstract

Systems and methods for cuboid detection and key point localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and key point localization in a monocular image. The deep cuboid detector may include multiple convolutional layers and non-convolutional layers of a trained convolutional neural network to determine a convolutional feature map from an input image. A region generation network of the deep cuboid detector can use the convolutional feature map to determine a bounding box around a cuboid in an image. The pooling layer and regression layer of the deep cuboid detector can implement iterative feature pooling to determine a refined bounding box and a parameterized representation of the cuboid.

Description

[0001] This application is a divisional application of the Chinese patent application with application number 201780082830.5. The application date of the original application is November 14, 2017, the priority date is November 15, 2016, the date of entering the Chinese national phase is July 9, 2019, and the name of the invention is "Deep learning system for rectangular block detection".

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Patent Application No. 62 / 422,547, filed on November 15, 2016, entitled “DEEP CUBOID DETECTION: BEYOND 2D BOUNDING BOXES,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0004] The present disclosure relates generally to systems and methods for three-dimensional object detection in images, and more particularly to a deep machine learning system for detecting cuboids in images. Background Art

[0005] Deep Neural Network (DNN) is a computational machine learning method. DNN belongs to a class of artificial neural networks (NN). Using NN, a computational graph that simulates the characteristics of biological neural networks is constructed. Biological neural networks include features that are prominent for computing and responsible for many capabilities of biological systems, which otherwise may be difficult to capture by other methods. In some embodiments, such a network is arranged into a sequential hierarchical structure with unidirectional connections. For example, the output of an artificial neuron in a particular layer can be connected to the input of an artificial neuron in a subsequent layer. DNN can be a NN with a large number of layers (e.g., 10, 100 or more layers).

[0006] Different NNs differ from each other in different perspectives. For example, the topology or architecture (e.g., the number of layers and how the layers are interconnected) and weights of different NNs can be different. The weights can be approximately analogous to the synaptic strength of neural connections in biological systems. The weights affect the strength of the effect propagated from one layer to another. The output of an artificial neuron can be a nonlinear function of the weighted sum of its inputs. The weights of a NN can be the weights that appear in these sums. Summary of the invention

[0007] Constructing a three-dimensional (3D) representation of the world from a single monocular image is an important challenge in computer vision. The present disclosure provides examples of systems and methods for 3D cuboid (e.g., box-shaped objects) detection and key point localization in an image. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and key point localization in an image. The deep cuboid detector may include multiple convolutional layers and non-convolutional layers of a trained convolutional neural network for determining a convolutional feature map from an input image. The region generation network of the deep cuboid detector can use the convolutional feature map to determine a bounding box around a cuboid in an image. The pooling layer and regression layer of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and parameterized representation of the cuboid.

[0008] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. Neither this summary nor the following detailed description is intended to define or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A is an example monocular image showing two-dimensional (2D) object detection, with a bounding box overlaid around the detected object.

[0010] Figure 1B is an example monocular image showing three-dimensional (3D) cuboid detection, where a representation of the cuboid is overlaid on the detected object. Figure 1B Detecting a cuboid within a monocular image and locating its vertices (shown as eight connected black circles) is shown.

[0011] Figure 2 Depicts an example architecture for a cuboid detector.

[0012] Figure 3 is an example image showing the normalized coordinates of the region of interest (RoI).

[0013] Figures 4A-4G Images illustrating example cuboid detection and keypoint localization are shown. In each image, one or more cuboids are detected and the keypoints of each cuboid are located, shown as white connected circles.

[0014] Figures 5A-5C Example images are shown, which illustrate the improved performance with keypoint refinement via iterative feature pooling.

[0015] Figure 6 is a schematic diagram showing an example rectangular parallelepiped vanishing point.

[0016] Figures 7A-7Fis a graph showing example performance of a cuboid detector.

[0017] Figure 8 is a flowchart of an example process for training a cuboid detector.

[0018] Fig. 9 is a flowchart of an example process for cuboid detection and keypoint localization using a cuboid detector.

[0019] Fig.10 An example of a wearable display system is schematically shown, which can implement an embodiment of a depth cuboid detector.

[0020] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION

[0021] Overview

[0022] A model (such as a function, algorithm, system, etc.) that represents data relationships and patterns can accept inputs and produce outputs corresponding to the inputs in some way. For example, the model can be implemented as a machine learning method, such as a convolutional neural network (CNN) or a deep neural network (DNN). Deep learning is part of a broader family of machine learning methods based on the idea of ​​learning data representations rather than task-specific algorithms, and shows great promise in solving audio-visual computing problems useful for augmented reality, mixed reality, virtual reality, and machine intelligence. In machine learning, a convolutional neural network (CNN or ConvNet) can include a class of deep feedforward artificial neural networks, and CNNs have been successfully applied to analyze visual images. Machine learning methods include a range of methods that can achieve robust and accurate solutions to a wide variety of problems, including eye image segmentation and eye tracking.

[0023] Disclosed herein is an example of a cuboid detector that processes an input image of a scene and locates at least one cuboid in the image. For example, a cuboid detector (such as a deep cuboid detector) can process a consumer-quality red-green-blue (RGB) image of a cluttered scene in an image and locate some or all three-dimensional (3D) cuboids. A cuboid may include a box-shaped or box-like object, and may include a polyhedron (which may be convex) having, for example, 4, 5, 6, 7, 8, 10, 12 or more faces. For example, a cuboid may include a pyramid, a cube, a prism, a parallelepiped, and the like. A cuboid is not limited to such a polyhedral shape of a geometric shape, and may include a box-like structure, such as, for example, a household appliance (e.g., a television, a computer monitor, a toaster, a washing machine, a refrigerator), furniture (e.g., a sofa, a chair, a bed, a crib, a table, a bookcase, a cabinet), a vehicle (e.g., a car, a bus), and the like. As further described below, cuboids may be identified based on the presence of their faces, vertices, edges, or bounding boxes.

[0024] In some embodiments, a cuboid may comprise a geometric shape characterized as a tuple of N parameters. The parameters may be geometric in nature, such as the radius of a sphere or the length, width, and height of a cuboid. A more general approach to parameterizing any geometric primitive may be to represent it as a set of points on the surface of the primitive. If random points on the surface of the primitive are chosen, the random points may be impossible to locate from a computer vision perspective. It may be advantageous that the set of parameterized points is geometrically informative and visually distinguishable. For example, in the case of a cuboid, the set of parameterized points may be the vertices of the cuboid (which may sometimes be referred to herein as corners or key points).

[0025] In some embodiments, a cuboid is represented as a tuple of eight vertices, where each vertex can be represented by its coordinates in the image (e.g., Cartesian x, y coordinates). In this representation, the cuboid is represented by 16 parameters: two coordinates for each of the eight vertices. In some cases, all 16 parameters may not be needed, for example, as described below, an alternative cuboid representation may not include some vertices (e.g., using only six vertices) and use vanishing points to determine other vertices.

[0026] In contrast to other methods that fit 3D models from low-level clues such as corners, edges, and vanishing points, the cuboid detector disclosed herein can be an end-to-end deep learning system that detects cuboids across many semantic categories (e.g., ovens, shipping boxes, and furniture). In some embodiments, the cuboid detector can use a two-dimensional (2D) bounding box to locate the cuboid and simultaneously locate the key points of the cuboid (e.g., vertices or corners), effectively producing a 3D interpretation or representation of the box-shaped object. The cuboid detector can improve the key points by iteratively pooling convolutional features, thereby improving the accuracy of the detected key points. Based on an end-to-end deep learning framework, an advantage of some embodiments of the cuboid detector is that there is little or no need to design customized low-level detectors for line segments, vanishing points, intersections, etc.

[0027] The cuboid detector may include multiple convolutional and non-convolutional layers of a convolutional neural network, a region generation network (RPN), and multiple pooling and regression layers. The RPN can generate object recommendations in an image. Multiple convolutional and non-convolutional layers can generate a convolutional feature map of an input image. The convolutional layer of a CNN may include a kernel stack of kernels. The kernel of a convolutional layer, when applied to its input, may produce a resulting output activation map showing the response to the particular learning kernel. The resulting output activation map may then be processed by another layer of the CNN. The non-convolutional layer of a CNN may include, for example, a normalization layer, a rectified linear layer, or a pooling layer.

[0028] A region generation network (RPN), which may be a convolutional neural network or a deep neural network, may determine a 2D bounding box around a cuboid in an image from a convolutional feature map. The 2D bounding box may represent a region of interest (RoI) on the image, which includes the cuboid at the image location. Multiple pooling and regression layers may include, for example, a pooling layer and two or more fully connected layers (such as 3, 5, 10 or more layers). Based on the initial 2D bounding box, multiple cuboid pooling and regression layers may iteratively determine a refined 2D bounding box and key points of the cuboid.

[0029] The cuboid detector can be trained in an end-to-end manner and in some implementations can be suitable for real-time applications in augmented reality (AR), mixed reality (MR), or robotics. As described below, a wearable mixed reality display device (e.g., reference Fig.10The wearable display system 1000 described may include a processor programmed to perform cuboid detection on an image acquired by an outward-facing camera of the display device. Some or all parameters of the cuboid detector may be learned in a process called training. For example, a machine learning model may be trained using training data that includes input data and a correct or preferred output of the model corresponding to the input data. The machine learning model may repeatedly process the input data, and the parameters (e.g., weight values) of the machine learning model may be modified in an amount for a trial and error process until the model produces (or "converges") the correct or preferred output. For example, the modification of weight values ​​may be performed by a process called "back propagation." Back propagation includes determining the difference between the expected model output and the obtained model output, and then determining how to modify the values ​​of some or all parameters of the model to reduce the difference between the expected model output and the obtained model output.

[0030] Example comparison of object detection and cuboid detection

[0031] Constructing a 3D representation of the world from a single monocular image is an important problem in computer vision. In some applications, objects with explicit 3D models are localized using their estimated poses. However, in the absence of such a 3D model, it is difficult for humans or computer systems (e.g., ref. Fig.10 The wearable display system 1000 described above may still need to infer its surroundings based on simple combinations of geometric shapes such as cuboids, cylinders, and spheres. Such primitives (sometimes called geons) may be easy for humans to infer. Humans can make rough estimates of the poses of these simple geometric primitives without difficulty, and even compare geometric parameters such as length, radius, or area between completely different instances. Although many objects are composed of multiple geometric primitives, a large number of real objects can be well approximated by only one primitive.

[0032] For example, a common shape is a box. Many everyday objects can be geometrically classified as boxes (e.g., shipping boxes, cabinets, washing machines, dice, microwave ovens, desktop computers). Boxes (which are examples of cuboids) span different sets of everyday object instances, and humans can easily fit imaginary cuboids to these objects and locate their vertices and faces. People can also compare the sizes of different box-like objects, even if they don't know the exact size of the box-like object, or even if the object is not a perfect cuboid. Disclosed herein are systems and methods for implementing a cuboid detector for detecting class agnostic geometric entities (such as cuboids). Class agnostic means not distinguishing between different classes of geometric entities. For example, a cuboid detector may not be able to distinguish between different classes of cuboids, such as shipping boxes, microwave ovens, or cabinets. All of these box-like objects can be represented by the same simplified concept (cuboid).

[0033] Embodiments of the cuboid detector can be used for 3D object detection as follows: fitting a 3D bounding box to an object in an image (e.g., an RGB image or an RGB depth (RGB-D) image), detecting 3D key points in an image, or performing 3D model to 2D image alignment. Because an image may contain multiple cuboids as well as a large number of cluttered objects (e.g., non-cuboid objects), the cuboid detector can first determine a candidate list of regions of interest (RoIs) corresponding to the cuboids. In addition to the 2D bounding box surrounding each cuboid, the cuboid detector can also determine the locations of all eight vertices.

[0034] Deep learning has revolutionized image recognition in the past few years. Many of today's state-of-the-art methods in object detection are built on top of deep networks that have been trained for image classification tasks. A cuboid detector can be a deep cuboid detector that implements one or more deep learning methods.

[0035] The cuboid detector can have high accuracy and use the hardware of the mobile device (e.g., reference Fig.10 The wearable display system 1000 described herein operates in real time.

[0036] Figure 1A is an example monocular image 100a illustrating two-dimensional (2D) object detection with a bounding box 104 overlaid around the detected object. Figure 1B is an example monocular image 100b showing three-dimensional (3D) cuboid detection, where a representation 108 of the cuboid is overlaid on the detected object. Figure 1B It is shown that a cuboid 108 within the monocular image 100 is detected and its vertices are located. Eight vertices are shown as four black circles 112a-112d connected by four edges 120a-120d (represented as dotted lines) and four other black circles 116a-116d connected by four edges 124a-124d (represented as solid lines). Four vertices in the vertices 112a-112d represent one face 128a of the cuboid, and the other four vertices in the vertices 116a-116d represent another face 128b of the cuboid. The two faces 128a, 128b of the cuboid 108 are connected by four edges 132a-132d (represented as dotted lines) through the vertices 112a-112d, 116a-116d. The cuboid detector can detect box-shaped objects in the scene. Unlike object detection, the cuboid detector can determine more than the bounding box of the object. In addition, the cuboid detector can locate the vertices of the cuboid (for example, compare Figure 1A and Figure 1B). In some embodiments, the cuboid detector can be category agnostic. For example, the cuboid detector does not care about the category of the cuboid being detected. For example, the cuboid detector can distinguish between two types of objects: cuboids and non-cubic objects. The cuboid detector can perform 3D cuboid detection by determining all cuboids within a monocular image and locating their vertices. The cuboid detector can be trained in an end-to-end manner. The cuboid detector can run in real time and use RGB images of cluttered scenes captured using consumer-grade cameras as input to perform cuboid detection. Wearable display devices (e.g., reference Fig.10 The wearable display system 1000 described herein can implement a cuboid detector and use information about the detected cuboids to generate or update a world map indicating the environment surrounding the user of the wearable display device.

[0037] A cuboid is a geometric object that can be parameterized, and a cuboid detector (e.g., a deep cuboid detector) can determine the parameters of cuboids in a scene. One way to detect cuboids is to detect edges and try to fit a model of the cuboid to those edges. Therefore, robust edge selection can be a useful aspect of a system. However, this becomes challenging when there are misleading textures on the surface of the cuboid, for example, if the edges and corners are occluded or the scene contains significant background clutter. It can be challenging to classify a given line as belonging to a given cuboid with purely local features. A cuboid detector can learn to detect cuboids in an image using a data-driven approach. A cuboid detector can assign a single label (e.g., "cuboid") to a box-shaped object in a scene, even if the labels are distributed over many categories, such as a house, a washing machine, a voting box, a table, a car, a television, etc. A cuboid detector can include systems that can successfully learn to help implement it (e.g., reference Fig.10 The wearable display system 1000 described herein is a CNN that recognizes features of rectangular blocks in different scenes.

[0038] In some embodiments, the cuboid detector may implement a deep learning model that jointly performs cuboid detection and key point positioning. For example, the cuboid detector may include a deep neural network that jointly performs cuboid detection and key point positioning. The cuboid detector may exceed the detection accuracy and positioning accuracy performed by other methods. In some implementations, the cuboid detector may first detect the object of interest, and then make a rough or initial prediction about the location of its vertices. The cuboid may utilize a rough or initial prediction as an attention mechanism to perform vertex refinement by only looking at areas with a high probability of being a cube. In some embodiments, the cuboid detector may implement an iterative feature pooling mechanism to improve accuracy. The cuboid detector may combine losses associated with the cuboid, and / or implement alternative parameterizations to improve accuracy.

[0039] Example Cuboid network architecture and loss function

[0040] Figure 2 An example architecture of a cuboid detector is depicted. The cuboid detector 200 may include one or more of the following components: a convolutional layer 204 (also referred to herein as a CNN tower), a region generation network (RPN) 208, at least one pooling layer 212, or one or more fully connected layers 216 (e.g., a region CNN (R-CNN) regressor (or classifier)). The pooling layer 212 and the fully connected layer 216 may implement iterative feature pooling, which refines the cuboid keypoint locations. The R-CNN may be a Faster R-CNN.

[0041] The cuboid detector 200 can implement a deep cuboid detection pipeline. The first action of the deep cuboid detection pipeline can determine the region of interest (RoI) 220a1, 220b in the image 202a where the cuboid may exist. The region generation network (RPN) 200 can be trained to output such RoI 220a1, 220b as shown in the image 202b. Then, one or more pooling layers 212 can be used to pool the region 224a with features corresponding to each RoI 220a1, 220b from the convolutional feature map 228 (e.g., the fifth convolutional feature map conv5 in VGG-M from the Visual Geometry Group of the University of Oxford). These pooled features can pass through two fully connected layers 216. In some implementations, instead of only generating a 2D bounding box, the cuboid detector 200 can output the normalized offset of the vertex from the center of the RoI 220a1, 220b. The cuboid detector 200 can refine the prediction by performing iterative feature pooling. Figure 2 The dashed lines in 202b show regions 224a, 224b of the convolutional feature map 228 corresponding to the RoI 220a1 in the image 202b and the refined RoI 220a2 in the image 202c, from which features can be pooled. The two fully connected layers 216 can process the region 224b of the convolutional feature map 228 corresponding to the refined RoI 220a2 to determine a representation of the further refined RoI and / or cuboid 232 in the image 202d.

[0042] CNN tower 204 may be a pre-trained fully convolutional portion of ConvNets, such as VGG and ResNets. Convolutional feature map 228 refers to the output of the last layer of CNN tower 204. For example, convolutional feature map 228 may be the output of the fifth convolutional layer, such as conv5 in VGG16 from the Visual Geometry Group of the University of Oxford, with a size of m×n×512.

[0043] RPN 208 can be a fully convolutional network that maps each unit in convolutional feature map 228 to a distribution over K multi-scale anchor boxes, bounding box offsets, and objectness scores. RPN can have two associated loss functions: a logarithmic loss function for objects and a smooth L1 loss function for bounding box regression. For example, RPN 208 can use 512 3×3 filters, then 18 1×1 filters for objects, and 36 1×1 filters for bounding box offsets.

[0044] The RoI pooling layer 212 can use, for example, max pooling to convert features within any valid region of interest 220a1, 220a2, 220b into a small fixed-size feature map (or a sub-map of the convolutional feature map 228). For example, for conv5 of size m×n×512, the pooling layer 212 can produce an output of size 7×7×512, regardless of the aspect ratio and scale of the input region. In some embodiments, spatial pyramid matching can be implemented.

[0045] A fully connected layer 216 (e.g., an R-CNN regressor) can then be applied to each fixed-size feature vector, outputting a cuboid score, a bounding box offset (four numbers), and eight cuboid keypoint locations (16 numbers). The bounding box regression values ​​(Δx, Δy, Δw, Δh) can be used to fit an initial object proposal that closely surrounds the object. The keypoint locations can be encoded as offsets relative to the RoI center and can be normalized by the proposal width / height, as Figure 3 as shown in . Figure 3 The RoI normalized coordinates of the vertices are shown, expressed as offsets from the center of the RoI 304 in the image 300 and normalized by the width w and height h of the region, where (x v ,y v ) is a key point 308, and (x c ,y c ) is the center of the RoI 312. Example ground truth targets for each keypoint are shown in equations [1] and [2]:

[0046]

[0047]

[0048] refer to Figure 2 , an R-CNN may include two fully connected layers 216 (e.g., each with 4096 neurons) and may have three associated loss functions: a logarithmic loss function for cuboids and a smooth L1 loss function for both bounding box and vertex regression.

[0049] When viewed jointly, the RoI pooling layer 212 and the R-CNN layer act as a refinement mechanism, mapping the input box to a refined box given the feature map. The cuboid detector 200 can apply the last part of the network multiple times (e.g., 2, 3, 4, or more times), referred to herein as iterative feature pooling.

[0050] The loss function used in RPN 208 may include L anchor-cls , the logarithmic loss on two categories (e.g., cuboids and non-cubic blocks), and L anchor-reg , a smooth L1 loss for the bounding box regression value of each anchor box. The loss function of R-CNN can include L ROI-cls , the logarithmic loss on two categories (e.g., cuboids and non-cubic blocks), L ROI-reg , a smooth L1 loss for bounding box regression of the RoI, and L ROI-corner , a smooth L1 loss on the predicted keypoint locations of the RoI. The last term can be called the corner or vertex regression loss. The full loss function can be a weighted sum of the above losses and can be written as shown in equation [3]. The loss weight λ i It may be different in different implementations, such as 0.1, 0.5, 1, 2, 5, 10 or more.

[0051] L=λ1L anchor-cls +λ2L anchor-reg +λ3L ROI-cls +λ4L ROI-reg +λ5L ROI-corner Equation [3]

[0052] Example Performance

[0053] To determine its performance, an embodiment of the cuboid detector 200 was implemented using Caffe and built on top of an implementation of Faster R-CNN. To determine performance, a VGG-M or VGG16 network that had been pre-trained on ImageNet for image classification tasks was used. VGG-M is a smaller model with 7 layers, while VGG16 contains 16 layers. All models were fine-tuned for 50K iterations using stochastic gradient descent (SGD) with a learning rate of 0.001, which was reduced by a factor of 10 after 30K iterations. Additional parameters used included a momentum of 0.9, a weight decay of 0.0005, and a loss of 0.5. Rather than training stage by stage, the components of the cuboid detector 200 were trained with all loss weights set to 1 (e.g., λ in Equation [3] i =1) Joint optimization.

[0054] data. The SUN primitive dataset (a comprehensive collection of annotated images covering various environmental scenes, places, and interior objects; available from https: / / groups.csail.mit.edu / vision / SUN / ) is used to train the deep cuboid detector 200. The dataset consists of 3516 images and is a mixture of indoor scenes with multiple cluttered objects, Internet images containing only single cuboids, and outdoor images of buildings that look like cuboids. Both the cuboid bounding boxes and the cuboid keypoints have ground truth annotations. The dataset includes 1269 annotated cuboids in 785 images. The remaining images are negative, for example, they do not contain any cuboids. The dataset is split to create a training set of 3000 images and their horizontally flipped versions and a test set with 516 test images.

[0055] The cuboid detector 200 is evaluated on two tasks: cuboid bounding box detection and cuboid keypoint localization. For detection, the bounding box is correct if the intersection over union (IoU) overlap is greater than 0.5.2. Detections are classified by having an average average precision (AP) and a confidence of the entire precision-recall curve reported (e.g., the classifier softmax output of the network). For keypoint localization, the performance of the cuboid detector is determined using the correct keypoint probability (PCK) and keypoint average precision (APK) metrics. PCK and APK are used in the human pose estimation literature to measure the performance of systems that predict the positions of human parts (such as head, wrist, etc.). When all ground truth boxes are given as input to the system, PCK measures the fraction of correctly annotated instances. A predicted keypoint is considered correct if its normalized distance from the annotation is less than a threshold (α). APK, on ​​the other hand, takes into account both detection confidence and keypoint localization. A normalized distance α of 0.1 is used, which means that a predicted keypoint is considered correct if it is within a number of pixels of the ground truth annotation of the keypoint as shown in equation [4]. The normalized distance α may be different in different implementations, such as 0.01, 0.2, 0.3, 0.5, 0.9 or more.

[0056] 1.1*max(height, width) Equation [4]

[0057] These metrics are reported on the SUN graph test set and on Figures 4A-4G For a sample of cuboid detection and vertex localization in monocular images 400a-400y, 404a-404e shown in FIG. Figures 7A-7F .For example, Figure 4A A monocular image 400a is shown with example representations 108a-108d of four cuboids, each represented as eight vertices. As another example, Figure 4AAnother monocular image 400b is shown having an example representation 108a of a cuboid having four vertices representing one face of the cuboid connected by four edges (shown as solid lines) and four vertices representing another face of the cuboid connected by the other four edges (shown as dashed lines). The eight vertices on the two faces of the representation 108a of the cuboid are connected by four edges (shown as dashed lines).

[0058] Figures 7A-7F is a graph showing example deep cuboid detector evaluation metrics. APK: Average Precision of keypoints, PCK: Probability of a correct keypoint: normalized distance from ground truth corner, keypoint order: front top left, back top left, front bottom left, front top right, back bottom left, front bottom right, back top right, back bottom right. B: Bounding box loss, C: Corner loss, and I: Iteration. Figures 4A-4F An image illustrating example cuboid detection and keypoint locations using VGG16 as a CNN tower and iterative feature pooling is shown. The cuboid detector 200 is able to locate the vertices of a cuboid in a consumer-grade RGB image. The cuboid detector 200 is able to handle objects such as boxes (which are completely modeled by a cuboid) as well as objects such as sinks (which are only approximately cuboids). Figure 4G Example images 404a - 404e are shown illustrating incorrect cuboid detection and keypoint localization which may be reduced or eliminated as further described below.

[0059] In one implementation, Cuboid Detector 2 achieves an mAP of 75.47 for bounding box detection, which significantly outperforms a HOG-based system with an mAP of 24.0.

[0060] Multi-task learning . Train multiple networks that each perform different multiple tasks. Train a base network that outputs only a bounding box around a cuboid. The base network performs general object detection using a rectangle that encloses the cuboid. The base network outputs the category of the box and the bounding box regression value. Next, train a different network with additional supervision about the location of the corners. This network does not output bounding box regression coordinates. Then, train a network that outputs both bounding box regression values ​​and vertex coordinates (e.g., cuboid detector 200). For each additional task, a corresponding term is added to the loss function. Starting from testing, adding more tasks (bounding box detection, keypoint localization, or both bounding box detection and keypoint localization) affects the performance of the cuboid detector (see Table 1).

[0061] Table 1. Multi-task learning results. The network is trained using only the bounding box loss, and then the cuboid corner loss.

[0062] Additional loss function AP APK PCK Bounding Box Loss 66.33 - - Corner loss 58.39 28.68 27.64 Bounding Box + Corner Loss 67.11 34.62 29.38

[0063] Iterative feature pooling. In R-CNN, the final output is a classification score and a bounding box regression value for each region proposal. Bounding box regression allows the region proposal to be moved and scaled so that the final bounding box only locates the object. This means that the initial region where the features are pooled to make this prediction is not completely correct. In some embodiments, the cuboid detector 200 returns and pools features from the refined bounding boxes. This can be implemented in the network itself, meaning that the cuboid detector 200 performs iterative bounding box regression while training and testing in exactly the same way. The input to the fully connected layer 216 of the regressor is a fixed-size feature map, a sub-map of the convolutional feature map 228, which includes pooled features from different region proposals from the conv5 layer. The R-CNN output can be used for bounding box regression on the input object proposal to produce new recommendations. Features can then be pooled from these new recommendations and passed through the fully connected layer 216 of the regressor again. In some embodiments, the cuboid detector 200 is an "any-time prediction system" where the bounding box regression can be performed more than once for applications that are not limited by latency. The performance results (see Table 2) show that iterative feature pooling can significantly improve both bounding box detection and vertex localization (see Figures 5A-5C ). When the features are iteratively pooled two or more times (e.g., 2, 3, 4, 5, 6 or more times), there is no significant change in performance. In some implementations, two iterations are used. Figures 5A-5C Example images 500a1-500l1, 500a2-500l2 are shown, which show improved performance using keypoint refinement via iterative feature pooling (e.g., compared to the representations 108b1, 108b2 of the cuboids in images 500a1, 500a2 and the shape of the bookshelf 504 in these images 504). The cuboid detection region is refined by re-pooling the features from conv5 using the predicted bounding box.

[0064] Table 2. Results of iterative feature pooling. Iterative feature pooling improves box detection AP by more than 4% and PCK by more than 7%.

[0065] method AP APK PCK Corner loss 58.39 28.68 27.64 Corner loss + iteration 62.89 33.98 35.56 BB+Corner Loss 67.11 34.62 29.38 BB+corner loss+iteration 71.72 37.61 36.53

[0066] Network Depth. Two base models, VGG16 and VGG-M, were tested. While VGG16 has a very deep architecture of 16 layers, VGG-M is a smaller model with 7 layers. Table 3 shows the test results. Interestingly, for this dataset and task, two iterations through the shallower network outperform one iteration through the deeper network. Combined with the fact that the shallower network with iterations runs twice as fast, the cuboid detector 200 can advantageously include a shallower CNN tower with less than 10 layers (e.g., 5, 7, or 9 layers). In some embodiments, the cuboid detector 200 may include deeper CNN towers (e.g., 12, 15, 20, or more layers). The four models tested each had an average precision (AP) that was higher than the AP (24.0) of the HOG-based system.

[0067] Table 3. VGG-M (7 layers) vs. VGG16 (16 layers) base network. I: Perform iterative feature pooling. Deeper cuboid detectors outperform the outperformed cuboid detectors.

[0068]

[0069]

[0070] The impact of training set size The effect of increasing the training data size was measured. Three datasets of different sizes, 1K, 2K and 3K images, were created and used to train a common network (VGG-M+ iterations). The results (see Table 4) show significantly improved performance when using larger training set sizes.

[0071] Table 4. Performance vs. number of training images. Deep cuboid detection can benefit from more training images.

[0072] Number of images AP APK PCK 1000 40.47 20.83 26.60 2000 52.17 27.51 29.31 3000 71.72 37.61 26.53

[0073] Memory and runtime complexity The cuboid detector 200 is able to run at interactive rates on a Titan Z GPU, whereas HOG-based methods would take several minutes to process a single image. The real-time nature of the system may be a result of Faster R-CNN being used as a regressor. In some embodiments, the cuboid detector 200 may implement a single display multi-box detector (SSD) to further improve its speed performance. Table 3 shows the model size, which may be relatively large on a mobile device (e.g., reference Fig.10 The wearable display system 1000 described herein is reduced.

[0074] Example Keypoint Parameterization

[0075] Embodiments of the cuboid detector 200 may directly output the vertices of the cuboid. Many convex cuboids have eight vertices, six faces, and twelve edges (not all of these vertices, faces, and edges may be visible in the image). However, some viewpoints may have inherent ambiguity, which may result in Figure 4G Inappropriate cuboid recognition as shown in . For example, Figure 4G Which face of the cube in should be marked as the front? Since the cuboid detector 200 detector may need to handle this configuration, alternative cuboid parameterizations are explored. If the world origin is considered to coincide with the camera center coordinates, the parameterization of the cuboid can be expressed with 12 numbers. The following parameterization is likely to be minimal; in other parameterizations, additional or different parameters may be used.

[0076] (X, Y, Z) - Coordinates of the center of the cuboid in 3D

[0077] (L, W, H) - dimensions of the cuboid

[0078] (θ, ψ, φ) - 3 rotation angles of the cuboid (e.g., Euler angles)

[0079] (f,c x , c y ) - intrinsic camera parameters (e.g. focal length and coordinates of the optical center)

[0080] For many modern cameras, it can be assumed that there is no skew in the camera and equal focal length (in orthogonal directions). Over-parameterization of the cuboid (e.g., parameterization of the cuboid by sixteen parameters) may allow the cuboid detector 200 to produce output that does not represent a cuboid (see, e.g., Figure 4G ). Several different reparameterizations of the cuboid were tested to better exploit the geometric constraints. In general, the test results show that the network is able to learn features for tasks with more visual evidence in the image and predict parameters that can be appropriately scaled for stable optimization. Proper parameterization is advantageous when dealing with 3D geometry and deep learning. Even image-to-image transformations such as homographies (e.g., isomorphisms of projective spaces) can benefit from reparameterization (e.g., four-point parameterization). These techniques can reduce or eliminate incorrect identification of cuboids in images.

[0081] Hexagonal parameterization. Another parameterization in which the detector predicts only six coordinates of the eight cuboid vertices. The relationship that there may be parallel edges in the cuboid is used to infer the positions of the remaining two coordinates. For example, parallel edges in 3D meet at a vanishing point in the image. There may be two pairs of parallel lines on the top base surface of the cuboid 600, and there may be two pairs of parallel lines on the bottom surface of the cuboid. A pair of parallel lines 604a, 604b on the top surface of the cuboid 600 and a pair of parallel lines 606a, 606b on the bottom surface of the cuboid should meet at the same vanishing point 608a, as shown in FIG. Figure 6 . A pair of parallel lines 604c, 604d on the top surface of the cuboid 600 and a pair of parallel lines 606c, 606d on the bottom surface of the cuboid should meet at the same vanishing point 608b. Therefore, the positions of the remaining two points 612a, 612b can be inferred. In some implementations, this allows the cuboid detector 200 to parameterize the output of 12 numbers. Figure 6 Example cuboid vanishing points 608a, 608b are schematically shown. Vanishing points 608a, 608b generated by extrapolating the edges of the cube form vanishing lines 616 and can be used to reduce the number of parameters. Front top left (FTL) keypoints 612a and back bottom right (BBR) keypoints 612b can be excluded from parameterization and inferred using estimated vanishing point (VP) techniques.

[0082] Octagonal parameterization is compared to hexagonal parameterization. Ground truth data for two vertices is not used during training. One vertex from each back and front is discarded (those with the worst detection rate (PCK)). The network is trained to predict the positions of the remaining six corners. These six corners are used to infer the positions of the two descending vertices. The cuboid detector 200 first determines the vanishing points corresponding to the predicted six points. This reparameterization may result in a degradation in performance (see Table 5). The degradation may be due to the discarding of visual evidence corresponding to the two inferred corners present in the image. In addition, any error in the prediction of one vertex due to occlusion or any other reason will propagate directly to the inferred corners. However, it is left to the cuboid detector 200, which has learned multiple models to detect cuboids. The network of the cuboid detector 200 is free to use all visual evidence to locate the corners of the cuboid. The cuboid detector 200 is able to perform pure geometric reasoning because in many cases, due to self-occlusion, the corners on the back face have no visual evidence in the image.

[0083] Table 5. Octagonal and hexagonal parameterization. The octagonal parameterization uses all cuboid corners, while in the hexagonal parameterization, the BBR and FTL corners are discarded (see Figure 6 ) and infer from the vanishing point. This shows how the example network is able to do geometric reasoning and that over-parameterization can increase the robustness of the system. BBR: Back-Down-Right and FTL: Front-Up-Left.

[0084]

[0085] Vanishing Point Parameterization : Another reparameterization uses the positions of the two vanishing points and the slopes of the six lines that will form the sides of the cuboid (see Figure 6 ). Note that these vanishing points correspond to specific cuboids and may be different from the vanishing points for the entire image. In this example, the intersections of these six lines will give the vertices of the cuboid. However, many of the locations of the vanishing points are outside the region of interest and have little or confusing visual evidence in the region of interest or in the entire image itself. It may also be difficult to normalize the target to directly predict the vanishing points. The slopes of the six lines can vary between -∞ and +∞. Instead of predicting the slopes directly, the slopes can be regressed to sin(tan -1 (θ)). There may be a set of hyperparameters (e.g., loss weights, learning rates, solvers, etc.) in which embodiments of the network may be trained.

[0086] Example process of training a cuboid detector

[0087] Figure 8 800 is a flow chart of an example process 800 for training a cuboid detector. Process 800 begins at box 804, where a plurality of training images are received, each training image including at least one cuboid. Some training images may each include one or more cuboids. Process 800 may include executing a cuboid-specific (e.g., cuboid-specific) data augmentation strategy to improve the performance of the trained cuboid detector. At box 808, a convolutional neural network is received. The convolutional neural network may be trained for object detection. For example, the convolutional neural network may be a VGG16 or a VGG-M. In some implementations, the convolutional neural network may be a deep neural network.

[0088] At block 812, a cuboid detector is generated. The cuboid detector may include a CNN tower. The CNN tower may include a plurality of convolutional layers and non-convolutional layers of the convolutional neural network received at block 808. For example, the CNN tower may include some or all of the convolutional layers of the received convolutional neural network. The non-convolutional layers may include normalization layers, brightness normalization layers, batch normalization layers, rectified linear layers, upsampling layers, cascade layers, pooling layers, softening layers, or any combination thereof. The CNN tower may generate a convolutional feature map from an input image, such as a monocular image.

[0089] The cuboid detector may include a region generation network (RPN), such as a CNN or a DNN. The region generation network may be connected to a layer of a CNN tower. The region generation network may determine a region of interest (RoI) including a cuboid in an image using a convolutional feature map. For example, the region of interest may be represented as a two-dimensional (2D) bounding box surrounding the cuboid at the cuboid image location. The cuboid may include a cuboid, a cylinder, a sphere, or any combination thereof. The RPN may be associated with at least two loss functions, such as a logarithmic loss function and a smooth L1 loss function during training.

[0090] The cuboid detector may include a pooling layer and at least one regression layer. The pooling layer may be connected to a layer of a CNN tower. The pooling layer may use the cuboid image position to determine a sub-map of a convolutional feature map corresponding to a region of interest including the cuboid. The pooling layer and the region generation network may be connected to the same layer of the CNN tower.

[0091] The cuboid detector may include two regression layers, such as two fully connected layers of a regional CNN (R-CNN) or a Fast R-CNN. As another example, the regression layer is not fully connected. During training, the regression layer may be associated with at least three loss functions. For example, the at least three loss functions include a logarithmic loss function and a smooth L1 loss function.

[0092] The cuboid detector can be trained. At box 816, the cuboid detector can determine a region of interest at an image position of a cuboid in the training image received at box 804. In some embodiments, a representation of the cuboid in the image can be determined. In order to determine the RoI and the representation of the cuboid at the cuboid image position, the cuboid detector can use the convolutional layer and the non-convolutional layer of the CNN tower to generate a convolutional feature map of the training image. Based on the convolutional feature map, the region generation network can determine the RoI of the cuboid at the initial image position in the training image. Based on the initial image position of the cuboid in the training image, the pooling layer of the cuboid detector can determine a sub-map of the convolutional feature map corresponding to the RoI of the cuboid at the initial image position. At least one regression layer can determine the RoI and the representation of the cuboid at the cuboid image position. The initial cuboid image position or the cuboid image position can be represented as a two-dimensional (2D) bounding box. In some embodiments, method 800 may include iteratively determining a representation of the RoI and the cuboid at the cuboid image location using a pooling layer, at least one regression layer, and a sub-map of a convolutional feature map corresponding to the RoI including the cuboid.

[0093] In different implementations, the representation of the cuboid may be different. The representation may include a parameterized representation of the cuboid. For example, the parameterized representation of the cuboid may include the locations of multiple key points of a cuboid (e.g., a cuboid) in an image, such as six or eight vertices of the cuboid in the image. As another example, the parameterized representation may include a normalized offset of multiple key points of the cuboid from the center of the image. As another example, the parameterized representation includes N tuples, such as 6 tuples. As an example, the parameterized representation of the cuboid includes a vanishing point parameterization.

[0094] At block 820, a first difference between the reference image location and the determined image location and a second difference between the reference representation of the cuboid and the determined representation of the cuboid may be determined. The reference representation of the cuboid may include a ground truth object for each keypoint, as shown in equations [1] and [2] above. The reference image location may include a bounding box represented by the ground truth object.

[0095] At box 824, the weights of the cuboid detector can be updated based on the first difference and the second difference. The difference can be expressed as a loss function (or its components) shown in equation [3]. Some or all weights of the cuboid detector can be updated based on the determined difference. For example, the weights of the region generation network and the weights of at least one regression layer can be updated based on the difference. As another example, the weights of the RPN and the weights of at least one regression layer can be updated without updating the weights of the first CNN based on the difference. As another example, the weights of the CNN tower, the weights of the region generation network, and the weights of at least one regression layer can be updated based on the difference. Process 800 can optionally include training the cuboid detector from larger data sets and synthetic data, network optimization, and regularization techniques to improve generalization.

[0096] Example process of cuboid detection and keypoint localization using cuboid detector

[0097] Fig. 9 900 is a flow chart of an example process 900 for cuboid detection and key point localization using a cuboid detector. The process 900 begins at block 904, where a system (e.g., reference Fig.10 The wearable display system 1000 described herein receives an input image including possible cuboids. The image may include one or more cuboids. The image may include a color image (e.g., RGB or RGB-D), and the image may be monocular. The image may be a frame of a video, and may be imaged using a reference image. Fig.10 The described wearable display system 1000 is obtained by the outward-facing imaging system 1044.

[0098] At block 908, the wearable display system 1000 may access a cuboid detector (such as a Figure 8The cuboid detector may include a CNN tower including a plurality of convolutional layers and non-convolutional layers. The cuboid detector may include a region generation network connected to the CNN tower. The cuboid detector may include a pooling layer and at least one regression layer. The pooling layer may be connected to the CNN tower.

[0099] At block 912, the wearable display system 1000 may generate a convolutional feature map (e.g., Figure 2 At block 916, the wearable display system 1000 may use a region generation network to determine at least one RoI (e.g., Figure 2 The cuboid image location may be represented as a two-dimensional (2D) bounding box. At block 920, the wearable display system 1000 may use a pooling layer (e.g., Figure 2 The sub-image of the convolution feature map corresponding to the region of interest including the cuboid can be determined by using the pooling layer 212 in FIG. 220 and the position of the cuboid image. For example, the sub-image can be determined from the region 224a of the convolution feature map 228, which can be Figure 2 In block 924, the wearable display system 1000 can use a regression layer (e.g., an R-CNN regressor) and a sub-graph to determine a representation of the refined RoI and the cuboid at the refined cuboid image location. The refined cuboid image location can be represented as a two-dimensional (2D) bounding box.

[0100] In some embodiments, method 900 includes iterative feature pooling. For example, wearable display system 1000 may use the refined cuboid image position to determine a refined sub-graph of the convolutional feature map corresponding to the refined region of interest including the cuboid. For example, a sub-graph may be determined from region 224b of convolutional feature map 228, which may be Figure 2 The wearable display system 1000 can use a pooling layer, at least one regression layer, and a refined sub-map of the convolutional feature map corresponding to the refined RoI to determine a representation of a further refined RoI and a further defined cuboid at a further refined cuboid image location.

[0101] The wearable display system 1000 can interact with the user of the system based on the refined region of interest and the representation of the cuboid at the refined cuboid image position. For example, the cuboid may correspond to a fixed box, and the wearable display system 1000 may generate a character animation associated with the fixed box based on the refined image position of the cuboid and the representation of the cuboid. As another example, the cuboid may correspond to a handheld cuboid. The wearable display system 1000 may use the representation of the cuboid to determine the posture of the cuboid and interact with the user of the system based on the posture of the cuboid. As another example, the cuboid may correspond to a rare object that CNN cannot recognize. The wearable display system 1000 may provide a notification to the user that a rare object that CNN cannot recognize has been detected. As an example, the cuboid corresponds to a man-made structure (e.g., a building). The wearable display system 1000 may assist the user of the system during unmanned flight based on the refined RoI and the representation of the cuboid at the refined cuboid image position. As another example, the cuboid may be a cuboid corresponding to a marker. The wearable display system 1000 can perform simultaneous localization and mapping (SLAM) based on the posture of the cuboid.

[0102] Sample Application

[0103] Detecting box-like objects in images and extracting 3D information such as pose can help overall scene understanding. Many high-level semantic problems can be solved by first detecting boxes in the scene (e.g., extracting free space in a room by reducing objects in the scene to boxes, estimating supporting surfaces in the scene, and estimating the scene layout).

[0104] The cuboid detector disclosed herein may generally open up one or more possibilities for augmented reality (AR), human-computer interaction (HCI), autonomous vehicles, drones, or robots. For example, the cuboid detector may be used as follows.

[0105] For augmented reality, cuboid vertex localization is followed by 6-DOF pose estimation, which allows content creators to drive character animation using a cuboid-centered coordinate system defined by a fixed box. Because the volume of space occupied by the fixed cuboid is known based on the cuboid vertex location followed by 6-DOF pose estimation, the animated character can jump onto the box, hide behind it, and even start drawing on one of the faces of the box. Therefore, content creators can use the cuboid detector to build a dynamic world around the cuboid.

[0106] For human-computer interaction, users can interact with the scene using boxy objects that surround them. Content creators can create games or user environments where the world is made of cuboids. As another example, a handheld cuboid can be used as a lightweight game controller. A system such as reference Fig.10 The wearable display system 1000 described may include a camera that captures images of a handheld cube over time. And the system can use the captured images to estimate the pose of the cube, effectively tracking the cube in 3D space. In some embodiments, the cuboid can be used as a way to improve interaction in an AR system (e.g., a desktop AR demonstration using a cuboid).

[0107] For autonomous vehicles, 3D cuboid detection allows the vehicle to reason about the spatial extent of rare objects that may be lost in the supervised training set. By reasoning about the pose of objects in a class-agnostic way, autonomous vehicles can be safer drivers.

[0108] For drones, man-made structures such as buildings, houses or cars can be well approximated by cuboids to aid navigation during unmanned flight. For general robotics, detecting box-like objects in images and extracting 3D information such as pose helps with overall scene understanding. For example, placing a small number of cuboids (instead of Aruco markers) in the scene can make pose tracking more robust for simultaneous localization and mapping (SLAM) applications.

[0109] Additional Embodiments

[0110] In some embodiments, the cuboid detector does not rely on bottom-up image processing and works satisfactorily on real-time images. The cuboid detector can be trained using a large training database of 3D models and some kind of learning for 2D to 3D alignment. In some embodiments, the cuboid detector can implement a geometry-based approach, a deformable component model, a histogram orientation gradient (HOG)-based model (e.g., a HOG classifier). The cuboid detector can detect cuboid vertices in different views and determine the final cuboid configuration based on the HOG classifier score, 2D vertex displacement, edge alignment score, and 3D shape score, which takes into account the proximity of the predicted vertex to the 3D cube. The cuboid detector can jointly optimize the visual evidence (corners and edges) found in the image while penalizing predictions that deviate too far from the actual 3D cuboid.

[0111] Without being limited by theory, the cuboid detector can be attributed to the performance of a convolutional neural network. CNN can outperform existing methods for image classification tasks. To locate a cuboid in an image, the image is decomposed into regions and these regions are classified, for example, in real time. The cuboid detector can perform detection in a single step. For example, in reference Fig.10The cuboid detector running on the described wearable display system 1000 can process 50-60 frames per second, thereby performing real-time cuboid detection and keypoint localization. The iterative keypoint refinement implemented by the cuboid detector can be based on an iterative error feedback method (network cascade) of iterative bounding box regression of a multi-region CNN and an internal and external network. Alternatively or additionally, the iterative keypoint refinement implemented by the cuboid detector can be based on a recursive neural network.

[0112] Example NN layer

[0113] A layer of a neural network (NN) such as a deep neural network (DNN) may apply a linear or nonlinear transformation to its input to generate its output. A deep neural network layer may be a normalization layer, a convolutional layer, a softsign layer, a rectified linear layer, a cascade layer, a pooling layer, a recursive layer, a class initial layer, or any combination thereof. A normalization layer may normalize the brightness of its input to generate its output by, for example, L2 normalization. For example, a normalization layer may normalize the brightness of multiple images relative to each other at once to generate multiple normalized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization may nonlinearly normalize the contrast of an image by normalizing a local region of the image on a per-pixel basis to have a mean of zero and a variance of one (or a mean and variance of other values). Local response normalization may normalize an image on a local input region to have a mean of zero and a variance of one (or a mean and variance of other values). Normalization layers can speed up the training process.

[0114] A convolutional layer may apply a set of kernels that convolve its input to generate its output. A softsign layer may apply a softsign function to its input. The softsign function (softsign(x)) may be, for example, (x / (1+|x|)). The softsign layer may ignore the effects of outliers for each element. A rectified linear layer may be a rectified linear layer unit (ReLU) or a parameterized rectified linear layer unit (PReLU). A ReLU layer may apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) may be, for example, max(0, x). A PReLU layer may apply a PReLU function to its input to generate its output. For example, if x≥0, the PReLU function PReLU(x) may be x, and if x<0, it may be ax, where a is a positive number. A cascade layer may cascade its input to generate its output. For example, a cascade layer may cascade four 5×5 images to generate a 20×20 image. A pooling layer may apply a pooling function that downsamples its input to generate its output. For example, a pooling layer may downsample a 20×20 image to a 10×10 image. Non-limiting examples of pooling functions include max pooling, average pooling, or min pooling.

[0115] At time point t, the recursive layer can calculate a hidden state s(t), and the recursive connection can provide the hidden state s(t) to the recursive layer at time t as an input at a subsequent time point t+1. The recursive layer can calculate its output at time t+1 based on the hidden state s(t) at time t. For example, the recursive layer can apply a softsign function to the hidden state s(t) at time t to calculate its output at time t+1. The hidden state of the recursive layer at time t+1 has the hidden state s(t) of the recursive layer at time t as its input. The recursive layer can calculate the hidden state s(t+1) by applying, for example, a ReLU function to its input. The class initiation layer may include one or more of a normalization layer, a convolutional layer, a softsign layer, a rectified linear layer (such as a ReLU layer and a PReLU layer), a cascade layer, a pooling layer, or any combination thereof.

[0116] In different implementations, the number of layers in the NN can be different. For example, the number of layers in the DNN can be 50, 100, 200, or more. The input types of the deep neural network layers can be different in different implementations. For example, a layer can receive the outputs of multiple layers as its input. The input of a layer can include the outputs of five layers. As another example, the input of a layer can include 1% of the layers of the NN. The output of a layer can be the input of multiple layers. For example, the output of a layer can be used as the input of five layers. As another example, the output of a layer can be used as the input of 1% of the layers of the NN.

[0117] The input size or output size of a layer can be very large. The input size or output size of a layer can be n×m, where n represents the width of the input or output, and m represents the height of the input or output. For example, n or m can be 11, 21, 31, or more. In different implementations, the channel size of the input or output of a layer can be different. For example, the channel size of the input or output of a layer can be 4, 16, 32, 64, 128, or more. In different implementations, the kernel size of a layer can be different. For example, the kernel size can be n×m, where n represents the width of the kernel, and m represents the height of the kernel. For example, n or m can be 5, 7, 9, or more. In different implementations, the stride size of a layer can be different. For example, the stride size of a deep neural network layer can be 3, 5, 7, or more.

[0118] In some embodiments, NN may refer to multiple NNs that together compute the output of the NN. Different NNs in the multiple NNs may be trained for different tasks. A processor (e.g., referring to Fig.10 The processor of the local data processing module 1024 described in the foregoing description may calculate the output of the NN of the plurality of NNs to determine the output of the NN. For example, the output of the NN of the plurality of NNs may include a likelihood score. The processor may determine the output of the NN including the plurality of NNs based on the likelihood scores of the outputs of different NNs of the plurality of NNs.

[0119] Example Wearable Display System

[0120] In some embodiments, the user device may be or may be included in a wearable display device that may advantageously provide a more immersive virtual reality (VR), augmented reality (AR), or mixed reality (MR) experience in which digitally reproduced images, or portions thereof, are presented to the wearer in a manner that they appear or may be perceived as real.

[0121] Without being limited by theory, it is believed that the human eye can generally interpret a finite number of depth planes to provide depth perception. Therefore, by providing the eye with a different presentation of an image corresponding to each of these finite number of depth planes, a highly convincing simulation of perceived depth can be achieved. For example, a display including a waveguide stack can be configured to be worn and positioned in front of the eyes of a user or viewer. A waveguide stack can be utilized to provide a three-dimensional perception to the eye / brain by using multiple waveguides to direct light from an image injection device (e.g., a discrete display or output of a multiplexed display that transmits image information via one or more optical fibers) to the viewer's eyes at specific angles (and divergences) corresponding to the depth planes associated with a particular waveguide.

[0122] In some embodiments, two waveguide stacks may be utilized, one for each eye of the viewer, to provide a different image to each eye. As an example, an augmented reality scene may cause the wearer of the AR technology to see a real-world park-like setting featuring people, trees, buildings in the background, and a specific platform. In addition to these items, the wearer of the AR technology may also perceive that he "sees" a robot statue standing on the real-world platform, and a cartoon-like avatar character flying by that appears to be an anthropomorphic representation of a bumblebee, even though the robot statue and the bumblebee do not exist in the real world. (Multiple) waveguide stacks may be used to generate a light field corresponding to an input image, and in some implementations, the wearable display includes a wearable light field display. Examples of wearable display devices and waveguide stacks for providing light field images are described in U.S. Patent Publication No. 2015 / 0016777, the entire contents of which are incorporated herein by reference in their entirety.

[0123] Fig.10 An example of a wearable display system 1000 is shown that can be used to present a VR, AR, or MR experience to a display system wearer or viewer 1004. The wearable display system 1000 can be programmed to perform any application or embodiment described herein (e.g., perform a CNN, reorder the values ​​of an input activation map or kernel, eye image segmentation, or eye tracking). The display system 1000 includes a display 1008, and various mechanical and electronic modules and systems that support the functionality of the display 1008. The display 1008 can be coupled to a frame 1012 that can be worn by the display system wearer or viewer 1004 and is configured to position the display 1008 in front of the eyes of the wearer 1004. The display 1008 can be a light field display. In some embodiments, a speaker 1016 is coupled to the frame 1012 and in some embodiments is positioned adjacent to the user's ear canal, and another speaker (not shown) is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control. The display system 1000 may include an outward-facing imaging system 1044 (e.g., one or more cameras) that may obtain images (e.g., still images or video) of the environment surrounding the wearer 1004. The images obtained by the outward-facing imaging system 1044 may be analyzed by an embodiment of a depth cuboid detector to detect and locate cuboids in the environment surrounding the wearer 1004.

[0124] The display 1008 is operably coupled 1020 to a local data processing module 1024, such as by a wired lead or a wireless connection, which can be mounted in various configurations, such as fixedly attached to the frame 1012, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 1004 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0125] The local processing and data module 1024 may include a hardware processor, as well as non-transitory digital memory, such as non-volatile memory, e.g., flash memory, both of which may be used to assist in the processing, caching, and storage of data. The data may include data that is: (a) captured from a sensor (which may, for example, be operably coupled to the frame 1012 or otherwise attached to the wearer 1004), such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope; and / or (b) acquired and / or processed using a remote processing module 1028 and / or a remote data repository 1032, possibly for use through the display 1008 after such processing or acquisition. The local processing and data module 1024 may be operably coupled to the remote processing module 1028 and the remote data repository 1032 by communication links 1036, 1040, such as via a wired or wireless communication link, such that these remote modules 1028, 1032 are operably coupled to each other and may serve as resources for the local processing and data module 1024. Image capture device(s) may be used to capture eye images used in eye image segmentation or eye tracking processes.

[0126] In some embodiments, the remote processing module 1028 may include one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device. Video data may be stored locally in the local processing and data module 1024 and / or in a remote data repository 1032. In some embodiments, the remote data repository 1032 may include a digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 1024, allowing for full autonomous use from the remote module.

[0127] In some embodiments, the local processing and data module 1024 and / or the remote processing module 1028 are programmed to perform embodiments of reordering the values ​​of the input activation map or kernel, eye image segmentation, or eye tracking disclosed herein. For example, the local processing and data module 1024 and / or the remote processing module 1028 can be programmed to perform reference Fig. 9Embodiments of the process 900 described herein. The local processing and data module 1024 and / or the remote processing module 1028 can be programmed to perform cuboid detection and key point positioning disclosed herein. The image capture device can capture videos for specific applications (generally, for example, augmented reality (AR), human-computer interaction (HCI), autonomous vehicles, drones, or robots). The video can be analyzed using a CNN by one or both of the processing modules 1024, 1028. In some cases, offloading at least some of the reordered values ​​of the input activation map or kernel, eye image segmentation, or eye tracking to a remote processing module (e.g., in the "cloud") can improve the efficiency or speed of the calculation. The parameters of the CNN (e.g., weights, bias terms, subsampling factors for pooling layers, the number and size of kernels in different layers, the number of feature maps, etc.) can be stored in the data module 1024 and / or 1032.

[0128] The results of cuboid detection and key point localization (e.g., the output of the cuboid detector 200) can be used by one or both of the processing modules 1024, 1028 for additional operations or processing. For example, the processing modules 1024, 1028 of the wearable display system 1000 can be programmed to perform additional applications described herein (typically such as augmented reality, human-computer interaction (HCI), autonomous vehicles, drones, or robots) based on the output of the cuboid detector 200.

[0129] Additional aspects

[0130] In the first aspect, a system for cuboid detection and key point localization is disclosed. The system includes: a non-transitory memory configured to store: executable instructions, an image for cuboid detection, and a cuboid detector, the cuboid detector including: a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN) for generating a convolutional feature map from an image, a region generation network (RPN) of a second CNN for determining at least one region of interest (RoI) of a cuboid at a cuboid image position of the image using the convolutional feature map, and a method for determining a refined RoI at a refined cuboid image position and a representation of the cuboid using the convolutional feature map and the RoI including the cuboid. a pooling layer and at least one regression layer; a hardware processor in communication with a non-transitory memory, the hardware processor being programmed with executable instructions to: receive an image; generate a convolutional feature map using multiple convolutional and non-convolutional layers of a first CNN and the image; determine at least one RoI of a cuboid at a cuboid image position of the image using an RPN; determine a sub-map of the convolutional feature map corresponding to the RoI including the cuboid using the pooling layer and the cuboid image position; and determine a refined RoI at the refined cuboid image position and a representation of the cuboid using at least one regression layer and the sub-map of the convolutional feature map corresponding to the RoI including the cuboid.

[0131] In a second aspect, according to the system of aspect 1, the hardware processor is further programmed to: determine a refined sub-map of the convolutional feature map corresponding to the refined RoI including the cuboid using the refined cuboid image position; determine a further refined RoI at the further refined cuboid image position and a representation of the further defined cuboid using a pooling layer, at least one regression layer, and the refined sub-map of the convolutional feature map corresponding to the refined RoI including the cuboid.

[0132] In a third aspect, the system according to any one of aspects 1-2, wherein the cuboid image location is represented as a two-dimensional (2D) bounding box.

[0133] In a 4th aspect, according to any one of aspects 1-3, the thinned cuboid image position is represented as a two-dimensional (2D) bounding box.

[0134] In aspect 5, according to the system of any one of aspects 1-4, the non-convolutional layer of the first CNN includes a normalization layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a cascade layer, a pooling layer, a softsign layer or any combination thereof.

[0135] In a sixth aspect, the system according to any one of aspects 1-5, wherein the at least one regression layer comprises two or more layers.

[0136] In a 7th aspect, according to the system of aspect 6, the two or more layers include fully connected layers, non-fully connected layers, or any combination thereof.

[0137] In aspect 8, the system according to any one of aspects 1-7, wherein at least one regression layer is associated with at least three loss functions during training.

[0138] In a 9th aspect, according to the system of aspect 8, the at least three loss functions include a logarithmic loss function and a smooth L1 loss function.

[0139] In aspect 10, the system according to any one of aspects 1-9, wherein the RPN comprises a deep neural network (DNN).

[0140] In an 11th aspect, the system according to any one of aspects 1-10, wherein the RPN is associated with at least two loss functions during training.

[0141] In a 12th aspect, according to the system of aspect 11, the at least two loss functions include a logarithmic loss function and a smooth L1 loss function.

[0142] In a 13th aspect, the system of any of aspects 1-12, wherein the representation of the cuboid comprises a parameterized representation of the cuboid.

[0143] In a 14th aspect, the system of aspect 13, wherein the parameterized representation of the cuboid comprises locations of a plurality of key points of the cuboid in the image.

[0144] In a 15th aspect, the system of aspect 14, wherein the plurality of key points comprises eight vertices of a cuboid in the image.

[0145] In a 16th aspect, the system of aspect 13, wherein the parameterized representation comprises a normalized offset of a plurality of key points of a cuboid from a center of the image.

[0146] In aspect 17, the system of aspect 13, wherein the parameterized representation comprises N tuples.

[0147] In an 18th aspect, the system of aspect 13, wherein the parametric representation of the cuboid comprises 12 parameters.

[0148] In a 19th aspect, the system of aspect 13, wherein the parameterized representation of the cuboid comprises a vanishing point parameterization.

[0149] In aspect 20, the system of any of aspects 1-19, wherein the hardware processor is further programmed to: interact with a user of the system based on the representation of the refined RoI and the cuboid at the refined cuboid image location.

[0150] In a 21st aspect, the system of aspect 20, wherein the cuboid corresponds to a fixed box, and wherein, for user interaction with the system, the hardware processor is further programmed to: generate a character animation associated with the fixed box based on the refined image position of the cuboid and the representation of the cuboid.

[0151] In a 22nd aspect, the system of aspect 20, wherein the cuboid corresponds to a handheld cuboid, and wherein, in order to interact with a user of the system, the hardware processor is further programmed to: determine a posture of the cuboid using a representation of the cuboid; and interact with the user of the system based on the posture of the cuboid.

[0152] In a 23rd aspect, the system of aspect 20, wherein the cuboid corresponds to a rare object that is not recognizable by the third CNN, and wherein, in order to interact with a user of the system, the hardware processor is further programmed to: provide a notification to the user that the rare object that is not recognizable by the third CNN has been detected.

[0153] In a 24th aspect, the system of any of aspects 1-23, wherein the cuboid corresponds to a man-made structure, and wherein the hardware processor is further programmed to: assist a user of the system during unmanned flight based on the refined RoI and the representation of the cuboid at the location of the refined cuboid image.

[0154] In aspect 25, the system of any of aspects 1-24, wherein the cuboid corresponds to a marker, and wherein the hardware processor is further programmed to: perform simultaneous localization and mapping (SLAM) based on the refined RoI and the representation of the cuboid at the refined cuboid image location.

[0155] In a 26th aspect, a wearable display system is disclosed. The wearable display comprises: an outward-facing imaging system configured to obtain images for cuboid detection; and the system for cuboid detection and key point localization of any one of aspects 1-25.

[0156] In aspect 27, a system for training a cuboid detector is disclosed. The system includes: a non-volatile memory configured to store executable instructions; and a hardware processor in communication with the non-volatile memory, the hardware processor being programmed by the executable instructions to: receive a plurality of training images, each training image including at least one cuboid; generate a cuboid detector, wherein the cuboid detector includes: a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN), a region generation network (RPN) connected to a first layer of the plurality of convolutional layers and non-convolutional layers, and a pooling layer and at least one regression layer, the pooling layer and the at least one regression layer being connected to a second layer of the plurality of convolutional layers and non-convolutional layers; and train the cuboid detector, wherein in order to train the cuboid detector, the hardware processor is configured to: determine, using the cuboid detector, an RoI at a cuboid image position and a representation of the cuboid in a training image of a plurality of training images; determine a first difference between a reference cuboid image position and the cuboid image position, and a second difference between a reference representation of the cuboid and a determined representation of the cuboid; and update a weight of the cuboid detector based on the first difference and the second difference.

[0157] In aspect 28, the system of aspect 27, wherein the cuboid comprises a cuboid, a cylinder, a sphere, or any combination thereof.

[0158] In aspect 29, the system of any one of aspects 27-28, wherein a first layer of the plurality of convolutional and non-convolutional layers and a second layer of the plurality of convolutional and non-convolutional layers are identical.

[0159] In aspect 30, the system of any one of aspects 27-29, wherein, to determine the RoI and the representation of the cuboid at the cuboid image position, the hardware processor is further programmed to: generate a convolutional feature map for at least one training image of a plurality of training images using a plurality of convolutional layers and non-convolutional layers; determine at least one RoI including the cuboid at an initial cuboid image position in the training image using the RPN; determine a sub-map of the convolutional feature map corresponding to the at least one RoI including the cuboid using the initial cuboid image position; and determine the RoI and the representation of the cuboid at the cuboid image position using a pooling layer, at least one regression layer, and a sub-map of the convolutional feature map corresponding to the at least one RoI including the cuboid.

[0160] In aspect 31, the system of any of aspects 27-30, wherein the initial cuboid image position is represented as a two-dimensional (2D) bounding box.

[0161] In aspect 32, according to the system of any one of aspects 27-31, in order to determine the RoI and the representation of the cuboid at the cuboid image position, the hardware processor is further programmed to: use a pooling layer, at least one regression layer, and a sub-map of the convolutional feature map corresponding to the RoI including the cuboid to iteratively determine the RoI and the representation of the cuboid at the cuboid image position.

[0162] In a 33rd aspect, the system of any of aspects 27-32, wherein the initial cuboid image position is represented as a two-dimensional (2D) bounding box.

[0163] In aspect 34, the system of any of aspects 27-33, wherein to update the weights of the cuboid detector, the hardware-based processor is programmed to: update the weights of the RPN and the weights of the at least one regression layer.

[0164] In aspect 35, the system of any of aspects 27-33, wherein, to update the weights of the cuboid detector, the hardware-based processor is programmed to: update the weights of the RPN and the weights of at least one regression layer without updating the weights of the first CNN.

[0165] In aspect 36, the system of any one of aspects 27-33, wherein, to update the weights of the cuboid detector, the hardware-based processor is programmed to: update the weights of the first CNN, the weights of the RPN, and the weights of at least one regression layer.

[0166] In aspect 37, the system of any of aspects 27-36, wherein to generate the cuboid detector, the hardware-based processor is programmed to: receive a first CNN.

[0167] In aspect 38, the system of any of aspects 27-37, wherein at least one regression layer comprises two or more layers.

[0168] In a 39th aspect, the system of aspect 38, wherein the two or more layers comprise fully connected layers, non-fully connected layers, or any combination thereof.

[0169] In aspect 40, the system of any of aspects 27-38, wherein at least one regression layer is associated with at least three loss functions during training of the cuboid detector.

[0170] In aspect 41, the system of aspect 40, wherein the at least three loss functions include a logarithmic loss function and a smoothed L1 loss function.

[0171] In aspect 42, the system of any one of aspects 27-41, wherein the RPN comprises a deep neural network (DNN).

[0172] In aspect 43, the system of any of aspects 27-42, wherein the RPN is associated with at least two loss functions during training of the cuboid detector.

[0173] In aspect 44, the system of aspect 43, wherein the at least two loss functions include a logarithmic loss function and a smoothed L1 loss function.

[0174] In aspect 45, the system of any of aspects 27-44, wherein the representation of the cuboid comprises a parameterized representation of the cuboid.

[0175] In aspect 46, the system of aspect 45, wherein the parameterized representation comprises N tuples.

[0176] In a 47th aspect, a wearable display system is disclosed. The wearable display system includes: an outward-facing imaging system configured to obtain an image of an environment of a wearer of the wearable display system; a non-volatile memory configured to store the image; and a hardware processor in communication with the non-volatile memory, the processor being programmed to: access the image of the environment; analyze the image to detect a cuboid in the image, wherein to analyze the image, the processor is programmed to: generate a convolutional feature map including features using a layer of a convolutional neural network (CNN); map the convolutional feature map to a region of interest (RoI) using a region generation network (RPN); generate a first pooled feature by a pooling function in the RoI; pass the first pooled feature through a regressor to generate a first bounding box estimate and a first cuboid vertex estimate; generate a second pooled feature based on the first bounding box estimate; and pass the second pooled feature through a regressor to generate a second bounding box estimate and a second cuboid vertex estimate.

[0177] In a 48th aspect, the wearable display system of aspect 47, wherein the image comprises a monocular color image.

[0178] In aspect 49, the wearable display system of aspect 47 or aspect 48, wherein the RPN comprises a CNN that maps the convolutional feature map to the RoI.

[0179] In aspect 50, a wearable display system according to any one of aspects 47 to 49, wherein the first bounding box estimate or the second bounding box estimate includes an offset from a center of the bounding box.

[0180] In aspect 51, a system for detecting a cuboid in an image is disclosed. The system includes: a non-transitory memory configured to store an image of a region; a hardware processor in communication with the non-transitory memory, the processor being programmed to: evaluate a convolutional neural network to generate a feature map; analyze the feature map to obtain a region of interest (RoI); determine that the RoI contains a cuboid; analyze a first pooled feature in the RoI of the feature map to generate a first estimate for the vertices of the cuboid; generate an improved RoI based at least in part on the first estimate of the vertices of the cuboid; analyze a second pooled feature in the improved RoI of the feature map to generate a second estimate of the vertices of the cuboid; and output the second estimate of the vertices of the cuboid.

[0181] In a 52nd aspect, the system of aspect 51, wherein to analyze the feature map to obtain a region of interest (RoI), the processor is programmed to evaluate a region proposal network (RPN).

[0182] In a 53rd aspect, the system of aspect 51 or 52, wherein the first estimate for the vertices of the cuboid comprises an offset from the center of the RoI, or the second estimate for the vertices of the cuboid comprises an offset from the center of the refined RoI.

[0183] In aspect 54, a method for cuboid detection and keypoint localization is disclosed. The method is controlled by a hardware processor and includes: receiving an image; generating a convolutional feature map using a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN) of a cuboid detector and the image; determining at least one RoI of a cuboid at a cuboid image position of the image using a region generation network (RPN) of a second CNN including a cuboid detector; determining a sub-map of the convolutional feature map corresponding to the RoI including the cuboid using a pooling layer of the cuboid detector and the cuboid image position; and determining a representation of the refined RoI and the cuboid at the refined cuboid image position using at least one regression layer of the cuboid detector and a sub-map of the convolutional feature map corresponding to the RoI including the cuboid.

[0184] In aspect 55, the method according to aspect 54 further includes: using the refined cuboid image position, determining a refined sub-map of the convolutional feature map corresponding to the refined RoI including the cuboid; using a pooling layer, at least one regression layer, and the refined sub-map of the convolutional feature map corresponding to the refined RoI including the cuboid to determine a further refined RoI at the further refined cuboid image position and a representation of the further defined cuboid.

[0185] In aspect 56, a method according to any one of aspects 54-55, wherein the cuboid image location is represented as a two-dimensional (2D) bounding box.

[0186] In aspect 57, a method according to any one of aspects 54-56, wherein the thinned cuboid image location is represented as a two-dimensional (2D) bounding box.

[0187] In aspect 58, a method according to any one of aspects 54-57, wherein the non-convolutional layer of the first CNN includes a normalization layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a cascade layer, a pooling layer, a softsign layer, or any combination thereof.

[0188] In aspect 59, the method of any one of aspects 54-58, wherein at least one regression layer comprises two or more layers.

[0189] In aspect 60, the method of aspect 59, wherein the two or more layers comprise fully connected layers, non-fully connected layers, or any combination thereof.

[0190] In aspect 61, the method of any one of aspects 54-60, wherein the RPN comprises a deep neural network (DNN).

[0191] In aspect 62, a method according to any one of aspects 54-61, wherein the representation of the cuboid comprises a parameterized representation of the cuboid.

[0192] In aspect 63, the method of aspect 62, wherein the parameterized representation of the cuboid comprises locations of a plurality of key points of the cuboid in the image.

[0193] In aspect 64, the method of aspect 63, wherein the plurality of key points comprises eight vertices of a cuboid in the image.

[0194] In aspect 65, the method of aspect 62, wherein the parameterized representation comprises a normalized offset of a plurality of key points of a cuboid from a center of the image.

[0195] In aspect 66, the method of aspect 62, wherein the parameterized representation comprises N tuples.

[0196] In aspect 67, the method of aspect 62, wherein the parameterized representation of the cuboid comprises 12 parameters.

[0197] In aspect 68, the method of aspect 62, wherein the parameterized representation of the cuboid comprises a vanishing point parameterization.

[0198] In aspect 69, the method according to any one of aspects 54-58 further comprises: interacting with a user based on the representation of the refined RoI and the cuboid at the refined cuboid image position.

[0199] In aspect 70, the method of aspect 69, wherein the cuboid corresponds to a fixed box, and interacting with the user comprises: generating a character animation relative to the fixed box based on the refined image position of the cuboid and the representation of the cuboid.

[0200] In aspect 71, the method of aspect 69, wherein the cuboid corresponds to a hand-held cuboid, and wherein interacting with the user comprises: determining a posture of the cuboid using a representation of the cuboid; and interacting with the user based on the posture of the cuboid.

[0201] In aspect 72, the method of aspect 69, wherein the cuboid corresponds to a rare object that is not recognizable by the third CNN, and wherein interacting with the user comprises: providing a notification to the user that the rare object that is not recognizable by the third CNN has been detected.

[0202] In aspect 73, the method according to any one of aspects 54-72 further comprises: assisting a user of the system during unmanned flight based on the refined RoI and the representation of the cuboid at the location of the refined cuboid image, wherein the cuboid corresponds to a man-made structure.

[0203] In aspect 74, the method according to any one of aspects 54-73 further comprises: performing simultaneous localization and mapping (SLAM) based on the thinned RoI and the representation of the cuboid at the thinned cuboid image location, wherein the cuboid corresponds to the marker.

[0204] In aspect 75, the method according to any one of aspects 54-74 further includes: receiving multiple training images, each training image including at least one training cuboid; generating a cuboid detector and the training cuboid detector includes: using the cuboid detector to determine a training RoI at a training cuboid image position and a representation of the training cuboid in a training image of multiple training images; determining a first difference between a reference cuboid image position and a training cuboid image position, and a second difference between a reference representation of the training cuboid and a determined representation of the training cuboid; and updating a weight of the cuboid detector based on the first difference and the second difference.

[0205] In aspect 76, a method according to aspect 75, wherein determining the representation of the training RoI and the training cuboid at the training cuboid image position includes: using multiple convolutional layers and non-convolutional layers to generate a training convolutional feature map of at least one training image of multiple training images; using RPN to determine at least one training RoI of the training cuboid at an initial training cuboid image position included in the training image; using the initial training cuboid image position to determine a sub-map of the convolutional feature map corresponding to the at least one RoI including the cuboid; and using a pooling layer, at least one regression layer, and a sub-map of the training convolutional feature map corresponding to the at least one training RoI including the training cuboid to determine the representation of the training RoI and the training cuboid at the training cuboid image position.

[0206] In aspect 77, the method of aspect 76, wherein the initial training cuboid image locations are represented as two-dimensional (2D) bounding boxes.

[0207] In aspect 78, a method according to aspect 75, wherein determining the training RoI and the representation of the training cuboid at the training cuboid image position includes: using a pooling layer, at least one regression layer, and a sub-graph of the training convolutional feature map corresponding to the training RoI including the training cuboid to iteratively determine the RoI and the representation of the training cuboid at the training cuboid image position.

[0208] In aspect 79, the method of aspect 78, wherein the initial training cuboid image locations are represented as two-dimensional (2D) bounding boxes.

[0209] In aspect 80, the method according to any one of aspects 75-79, wherein updating the weights of the cuboid detector comprises: updating the weights of the RPN and the weights of at least one regression layer.

[0210] In aspect 81, the method according to any one of aspects 75-79, wherein updating the weights of the cuboid detector comprises: updating the weights of the RPN and the weights of at least one regression layer without updating the weights of the first CNN.

[0211] In aspect 82, the method according to any one of aspects 75-79, wherein updating the weights of the cuboid detector comprises: updating the weights of the first CNN, the weights of the RPN, and the weights of at least one regression layer.

[0212] In aspect 83, the method according to any one of aspects 54-82, wherein generating a cuboid detector comprises: receiving a first CNN.

[0213] In aspect 84, a method according to any one of aspects 75-83, wherein during training of the cuboid detector, at least one regression layer is associated with at least three loss functions.

[0214] In aspect 85, the method of claim 84, wherein the at least three loss functions include a logarithmic loss function and a smooth L1 loss function.

[0215] In aspect 86, a method according to any one of aspects 75-85, wherein during training of the cuboid detector, the RPN is associated with at least two loss functions.

[0216] In aspect 87, the method of claim 86, wherein the at least two loss functions include a logarithmic loss function and a smoothed L1 loss function.

[0217] In aspect 88, a method is disclosed. The method is controlled by a hardware processor and includes: accessing an image of an environment; analyzing the image to detect a cuboid in the image, including: using a layer of a convolutional neural network (CNN) to generate a convolutional feature map including features; using a region generation network (RPN) to map the convolutional feature map to a region of interest (RoI); pooling the features in the RoI to generate a first pooled feature; passing the first pooled feature through a regressor to generate a first bounding box estimate and a first cuboid vertex estimate; generating a second pooled feature based on the first bounding box estimate; and passing the second pooled feature through a regressor to generate a second bounding box estimate and a second cuboid vertex estimate.

[0218] In aspect 89, the method of aspect 88, wherein the image comprises a monocular color image.

[0219] In aspect 90, the method of aspect 88 or aspect 89, wherein the RPN comprises a CNN that maps the convolutional feature map to the RoI.

[0220] In aspect 91, a method according to any one of aspects 88 to 89, wherein the first bounding box estimate or the second bounding box estimate includes an offset from a center of the bounding box.

[0221] In aspect 92, a method for detecting a cuboid in an image is disclosed. The method is controlled by a hardware processor and includes: evaluating a convolutional neural network to generate a feature map; analyzing the feature map to obtain a region of interest (RoI); determining that the RoI contains a cuboid; analyzing first pooled features in the RoI of the feature map to generate a first estimate of the vertices of the cuboid; generating an improved RoI based at least in part on the first estimate of the vertices of the cuboid; analyzing second pooled features in the improved RoI of the feature map to generate a second estimate of the vertices of the cuboid; and outputting the second estimate of the vertices of the cuboid.

[0222] In a 93rd aspect, the method of aspect 92, wherein analyzing the feature map to obtain a region of interest (RoI) comprises evaluating a region proposal network (RPN).

[0223] In aspect 94, the method of aspect 92 or 93, wherein the first estimate of the vertices of the cuboid comprises an offset from a center of the RoI, or the second estimate of the vertices of the cuboid comprises an offset from a center of the refined RoI.

[0224] in conclusion

[0225] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings can be embodied in a code module executed by one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute dedicated and specific computer instructions, and fully or partially automated by the code module. For example, a computing system may include a general-purpose computer (e.g., a server) programmed with specific computer instructions or a dedicated computer, dedicated circuit, etc. The code module can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some embodiments, specific operations and methods can be performed by circuits specific to a given function.

[0226] Furthermore, certain implementations of the functionality of the present disclosure are mathematically, computationally or technically complex enough that dedicated hardware or one or more physical computing devices (with appropriate dedicated executable instructions) may be necessary, e.g., due to the amount or complexity of the calculations involved to perform the functionality or provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0227] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, a combination of the same or similar thereof, and the like. Methods and modules (or data) may also be sent as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media), and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

[0228] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood to represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (e.g., logic or arithmetic) or a step in a process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith can be executed in other appropriate sequences, for example, in serial, parallel or in some other manner. Tasks or events can be added to or deleted from the disclosed example embodiments. In addition, the separation of various system components in the implementation described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0229] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0230] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired attributes disclosed herein. The various features and processes described herein can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications to the implementations described in the present disclosure will be apparent, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are consistent with the widest scope of the present disclosure, principles, and novel features disclosed herein.

[0231] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, although the features above may be described as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a subcombination or variations of the subcombination. It is not necessary or essential for each and all embodiments to require a single feature or group of features.

[0232] Unless otherwise specified, or understood in the context of use in other ways, conditional language used herein, such as "can", "may", "possibly", "may", "for example", etc., is generally intended to convey that certain embodiments include certain features, elements and / or steps, while other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included or to be performed in any particular embodiment (regardless of whether there is author input or prompt). The terms "comprise", "include", "have", etc. are synonymous and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so when used, for example, to connect a list of elements, the term "or" means one, some or all of the elements in the list. In addition, unless otherwise specified, the articles "a", "an" and "the" used in this application and the appended claims should be understood to mean "one or more" or "at least one".

[0233] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, and Z" is otherwise understood by context as being generally used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0234] Similarly, although operations may be depicted in the accompanying drawings in a particular order, it should be appreciated that it is not necessary to perform these operations in the particular order shown or in sequence, or to perform all the operations shown, to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be included in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, operations may be rearranged or reordered in other embodiments. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. In addition, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in different orders and still achieve the desired results.

Claims

1. A system for training a cuboid detector, the system comprising: a non-transitory memory configured to store executable instructions; as well as one or more hardware processors in communication with the non-transitory memory, the one or more hardware processors being programmed by the executable instructions to: accessing a plurality of training images, wherein the plurality of training images includes a first training image; Generate a cuboid detector, wherein the cuboid detector includes: First, multiple convolutional and non-convolutional layers of a convolutional neural network (CNN); a region proposing network (RPN) connected to a first layer of the plurality of convolutional and non-convolutional layers; Pooling layers; and At least one regression layer; wherein both the pooling layer and the at least one regression layer are connected to a second layer of the plurality of convolutional and non-convolutional layers; and Training the cuboid detector, wherein training the cuboid detector comprises: Determining a region of interest (RoI) at a cuboid image location by applying the cuboid detector to the first training image; determining a representation of a cuboid in the training image by applying the cuboid detector to the first training image; determining a first difference between a reference cuboid image position and the cuboid image position; determining a second difference between the reference representation of the cuboid and the determined representation of the cuboid; and A weight of the cuboid detector is updated based on the first difference and the second difference.

2. The system according to claim 1, wherein: The cuboid includes a cuboid, a cylinder, a sphere or any combination thereof.

3. The system according to any one of claims 1 to 2, wherein: The first layer and the second layer are identical.

4. The system according to any one of claims 1 to 2, wherein: The one or more hardware processors are further programmed to: generating a convolutional feature map for the first training image using the plurality of convolutional layers and non-convolutional layers; Determine at least one RoI including the cuboid at an initial cuboid image position in the training image using the RPN; Determining a sub-graph of the convolutional feature map corresponding to the at least one RoI including the cuboid using the initial cuboid image position; as well as Representations of the RoI and the cuboid at the cuboid image location are determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the at least one RoI including the cuboid.

5. The system according to claim 4, wherein: The initial cuboid image position is represented as a two-dimensional (2D) bounding box.

6. The system according to claim 4, wherein: The one or more hardware processors are further programmed to: Representations of the RoI and the cuboid at the cuboid image location are iteratively determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the RoI including the cuboid.

7. The system according to claim 1 or 2, wherein: To update the weights of the cuboid detector, the one or more processors are further programmed to: Updating the weight of the RPN; and The weights of the at least one regression layer are updated.

8. The system according to claim 1 or 2, wherein: To update the weights of the cuboid detector, the one or more processors are further programmed to: Updating the weight of the RPN; and The weights of the at least one regression layer are updated without updating the weights of the first CNN.

9. The system according to claim 1 or 2, wherein: To update the weights of the cuboid detector, the one or more processors are further programmed to: Update the weights of the first CNN; Updating the weight of the RPN; as well as The weights of the at least one regression layer are updated.

10. The system according to claim 1 or 2, wherein: The one or more hardware processors are further programmed to: Receive the first CNN.

11. The system according to claim 1 or 2, wherein: The at least one regression layer includes two or more layers.

12. The system according to claim 11, wherein: The two or more layers include fully connected layers, non-fully connected layers, or any combination thereof.

13. The system according to claim 1 or 2, wherein: The at least one regression layer is associated with at least three loss functions during training of the cuboid detector.

14. The system according to claim 13, wherein: The at least three loss functions include a logarithmic loss function and a smoothed L1 loss function.

15. The system according to claim 1 or 2, wherein: RPN includes a deep neural network (DNN).

16. The system according to claim 1 or 2, wherein: The RPN is associated with at least two loss functions during training of the cuboid detector.

17. The system of claim 16, wherein: The at least two loss functions include a logarithmic loss function and a smoothed L1 loss function.

18. The system according to claim 1 or 2, wherein: The representation of the cuboid includes a parameterized representation of the cuboid.

19. The system of claim 18, wherein: The parameterized representation includes N tuples.

20. A method for training a cuboid detector, the method comprising: accessing a plurality of training images, wherein the plurality of training images includes a first training image; Generate a cuboid detector, wherein the cuboid detector includes: First, multiple convolutional and non-convolutional layers of a convolutional neural network (CNN); a region proposing network (RPN) connected to a first layer of the plurality of convolutional and non-convolutional layers; Pooling layers; and at least one regression layer, wherein both the pooling layer and the at least one regression layer are connected to a second layer of the plurality of convolutional and non-convolutional layers; and Training the cuboid detector, wherein training the cuboid detector comprises: Determining a region of interest (RoI) at a cuboid image location by applying the cuboid detector to the first training image; determining a representation of a cuboid in the training image by applying the cuboid detector to the first training image; determining a first difference between a reference cuboid image position and the cuboid image position; determining a second difference between the reference representation of the cuboid and the determined representation of the cuboid; and A weight of the cuboid detector is updated based on the first difference and the second difference.

21. The method according to claim 20, wherein: The first layer and the second layer are identical.

22. The method according to claim 20, further comprising: generating a convolutional feature map for the first training image using the plurality of convolutional layers and non-convolutional layers; Determine at least one RoI including the cuboid at an initial cuboid image position in the training image using the RPN; Determining a sub-graph of the convolutional feature map corresponding to the at least one RoI including the cuboid using the initial cuboid image position; as well as Representations of the RoI and the cuboid at the cuboid image location are determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the at least one RoI including the cuboid.

23. The method according to claim 22, further comprising: Representations of the RoI and the cuboid at the cuboid image location are iteratively determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the RoI including the cuboid.

24. A system for training a cuboid detector, the system comprising: a non-transitory memory configured to store executable instructions; as well as one or more hardware processors in communication with the non-transitory memory, the one or more hardware processors being programmed by the executable instructions to: Training a cuboid detector, wherein training the cuboid detector comprises: Determining by applying the cuboid detector to a first training image of a plurality of training images: A region of interest (RoI) at the cuboid image location; and A representation of a cuboid in the training image; determining a first difference between a reference cuboid image position and the cuboid image position; determining a second difference between the reference representation of the cuboid and the determined representation of the cuboid; and A weight of the cuboid detector is updated based on the first difference and the second difference.

25. The system of claim 24, wherein: The one or more hardware processors are further programmed to: A cuboid detector is generated, wherein the cuboid detector includes a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN).

26. The system of claim 25, wherein the one or more hardware processors are further programmed to: A cuboid detector is generated, wherein the cuboid detector includes a region proposing network (RPN) connected to a first layer of a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN).

27. The system of claim 26, wherein the cuboid detector further comprises: A pooling layer and at least one regression layer, wherein both the pooling layer and the at least one regression layer are connected to a second layer of the plurality of convolutional layers and non-convolutional layers of the first convolutional neural network (CNN).

28. The system of claim 27, wherein: The first layer and the second layer are identical.

29. The system of claim 27, wherein: The one or more hardware processors are further programmed to: generating a convolutional feature map for the first training image using the plurality of convolutional layers and non-convolutional layers; determining, using the RPN, at least one RoI including a cuboid associated with a representation of the cuboid at an initial cuboid image location in the first training image; Determining a sub-graph of the convolutional feature map corresponding to the at least one RoI including the cuboid using the initial cuboid image position; as well as Representations of the RoI and the cuboid at the cuboid image location are determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the at least one RoI including the cuboid.

30. The system of claim 29, wherein: The one or more hardware processors are further programmed to: Representations of the RoI and the cuboid at the cuboid image location are iteratively determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the RoI including the cuboid.

31. The system of claim 27, wherein: The one or more hardware processors are further programmed to: Update the weights of the first CNN; updating the weight of the RPN; or The weights of the at least one regression layer are updated.

32. The system of claim 27, wherein: The at least one regression layer includes two or more layers.

33. The system of claim 32, wherein: The two or more layers include fully connected layers, non-fully connected layers, or any combination thereof.

34. The system of claim 26, wherein: The at least one regression layer is associated with at least three loss functions during training of the cuboid detector.

35. The system of claim 26, wherein: The RPN: Includes a deep neural network (DNN); or At least two loss functions are associated during training of the cuboid detector.

36. The system of claim 24, wherein: The representation of the cuboid includes a parameterized representation of the cuboid.

37. A method for training a cuboid detector, the method comprising: Training a cuboid detector, wherein training the cuboid detector comprises: Determining by applying the cuboid detector to a first training image of a plurality of training images: A region of interest (RoI) at the cuboid image location; and A representation of a cuboid in the training image; determining a first difference between a reference cuboid image position and the cuboid image position; determining a second difference between the reference representation of the cuboid and the determined representation of the cuboid; and A weight of the cuboid detector is updated based on the first difference and the second difference.

38. The method of claim 37, further comprising: A cuboid detector is generated, wherein the cuboid detector includes a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN).

39. The method of claim 37, further comprising: A cuboid detector is generated, wherein the cuboid detector includes a region proposing network (RPN) connected to a first layer of a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN).

40. The method of claim 39, wherein the cuboid detector further comprises: A pooling layer and at least one regression layer, wherein both the pooling layer and the at least one regression layer are connected to a second layer of the plurality of convolutional layers and non-convolutional layers of the first convolutional neural network (CNN).

41. The method of claim 40, wherein: The first layer and the second layer are identical.

42. The method of claim 40, wherein: The one or more hardware processors are further programmed to: generating a convolutional feature map for the first training image using the plurality of convolutional layers and non-convolutional layers; determining, using the RPN, at least one RoI including a cuboid associated with a representation of the cuboid at an initial cuboid image location in the first training image; Determining a sub-graph of the convolutional feature map corresponding to the at least one RoI including the cuboid using the initial cuboid image position; as well as Representations of the RoI and the cuboid at the cuboid image location are determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the at least one RoI including the cuboid.

43. The method of claim 42, further comprising: Representations of the RoI and the cuboid at the cuboid image location are iteratively determined using the pooling layer, the at least one regression layer, and the sub-map of the convolutional feature map corresponding to the RoI including the cuboid.

Images